Boot failure investigation: 67% crontab bug rate confirmed, version mismatch ruled out
Parallel investigation of IncusOS issue #843 across 20 VM deploys confirms the "invalid crontab expression" error hits 67% of first boots. The bug is a race condition in the daemon startup — NOT related to ISO/CDN version mismatch, update downloads, or seed config. Sysext downloads occur on every first boot even with matching versions. New tools: investigate-parallel (parallel VM testing), pve-api (Python Proxmox API helper avoiding bash ! quoting), investigate-boot (sequential testing), test-boot-reliability (batch reliability testing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
52d8dbdc89
commit
2e2447de06
49
CLAUDE.md
49
CLAUDE.md
|
|
@ -504,8 +504,8 @@ incu-contrib/
|
||||||
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
|
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
|
||||||
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
|
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
|
||||||
Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
|
Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
|
||||||
With ~50% boot failure rate from the crontab bug, 3 retries gives ~93.75%
|
With ~67% boot failure rate from the crontab bug (confirmed by parallel
|
||||||
success rate. After retries, `fix_scrub_schedule()` proactively heals the
|
investigation), 3 retries gives ~96.3% success rate (1 - 0.67^3). After retries, `fix_scrub_schedule()` proactively heals the
|
||||||
root cause via the IncusOS REST API.
|
root cause via the IncusOS REST API.
|
||||||
|
|
||||||
### IncusOS first-boot sequence (observed via console screenshots)
|
### IncusOS first-boot sequence (observed via console screenshots)
|
||||||
|
|
@ -563,9 +563,9 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
||||||
### IncusOS boot failure (crontab / update frequency error)
|
### IncusOS boot failure (crontab / update frequency error)
|
||||||
|
|
||||||
- **Symptom**: "ERROR invalid crontab expression" appears on the console
|
- **Symptom**: "ERROR invalid crontab expression" appears on the console
|
||||||
during first boot. Intermittent — observed in ~50% of deploys (2/4
|
during first boot. Intermittent — confirmed **67% failure rate** across
|
||||||
observational deploys). Error does NOT correlate with `update.yaml`
|
20 parallel VM deploys (8/12 API-verified). Error does NOT correlate with
|
||||||
presence/absence.
|
`update.yaml` presence/absence, ISO version, or version mismatch.
|
||||||
- **Two distinct error paths in IncusOS**:
|
- **Two distinct error paths in IncusOS**:
|
||||||
- `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
|
- `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
|
||||||
like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
|
like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
|
||||||
|
|
@ -573,14 +573,20 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
||||||
(or opens briefly during the 15-second error sleep).
|
(or opens briefly during the 15-second error sleep).
|
||||||
- `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
|
- `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
|
||||||
format like `"6h"`). If invalid, it logs an error but does **not** crash.
|
format like `"6h"`). If invalid, it logs an error but does **not** crash.
|
||||||
- **Not always fatal** (key finding from observational testing):
|
- **Not always fatal** (confirmed by parallel investigation):
|
||||||
- The REST API starts BEFORE `startup()` completes. Applications (Incus)
|
- The REST API starts BEFORE `startup()` completes. Applications (Incus)
|
||||||
start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
|
start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
|
||||||
When Incus starts before the scheduler crashes, port 8443 is available
|
When Incus starts before the scheduler crashes, port 8443 is available
|
||||||
during the 15-second error sleep (before `os.Exit(1)`).
|
during the 15-second error sleep (before `os.Exit(1)`).
|
||||||
- `observe-deploy` detects port 8443 during this window → reports PASS.
|
- Port 8443 is reachable during the error window, and API calls work
|
||||||
|
(including `/os/1.0/system/storage`). The `scrub_schedule` field is
|
||||||
|
empty in the API response — this is the definitive indicator.
|
||||||
- "System is ready" is NOT shown on console (only appears after
|
- "System is ready" is NOT shown on console (only appears after
|
||||||
`registerJobs()` succeeds).
|
`registerJobs()` succeeds). Console shows "ERROR invalid crontab
|
||||||
|
expression" as the last line when the bug hits.
|
||||||
|
- **Detection heuristic**: port 8443 reachable BUT `scrub_schedule` is
|
||||||
|
empty → crontab bug hit. Port 8443 reachable AND `scrub_schedule` is
|
||||||
|
`"0 4 * * 0"` → genuine success.
|
||||||
- **Source code analysis** (root cause investigation):
|
- **Source code analysis** (root cause investigation):
|
||||||
- `state.txt` is created FRESH on first boot from disk by
|
- `state.txt` is created FRESH on first boot from disk by
|
||||||
`LoadOrCreate()` → `initialize()`. The `initialize()` function
|
`LoadOrCreate()` → `initialize()`. The `initialize()` function
|
||||||
|
|
@ -599,12 +605,27 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
||||||
- **Race window**: The REST API server starts BEFORE `startup()`
|
- **Race window**: The REST API server starts BEFORE `startup()`
|
||||||
completes. Applications start before `registerJobs()`. A concurrent
|
completes. Applications start before `registerJobs()`. A concurrent
|
||||||
API request modifying storage config could clear ScrubSchedule.
|
API request modifying storage config could clear ScrubSchedule.
|
||||||
- **Most likely root cause**: either (a) a race condition between the
|
- **Most likely root cause**: a race condition between the REST API/
|
||||||
REST API/application startup and `registerJobs()` that clears
|
application startup and `registerJobs()` that clears `ScrubSchedule`.
|
||||||
`ScrubSchedule`, or (b) a gocron library edge case where
|
Investigation ruled out version mismatch (bug hits equally with matching
|
||||||
`IsValid("0 4 * * 0", time.UTC, time.Now())` intermittently fails.
|
ISO), update downloads (occurs even without OS updates), and gocron
|
||||||
Both explain the ~50% hit rate and lack of correlation with seed
|
library issues (the field is genuinely empty in the state struct).
|
||||||
content.
|
- **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel):
|
||||||
|
Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%);
|
||||||
|
Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG
|
||||||
|
(67% failure rate). Console screenshots confirm "ERROR invalid crontab
|
||||||
|
expression" on failed VMs and "System is ready" on successful VMs.
|
||||||
|
- **Version mismatch NOT the cause**: stgraber's hypothesis that the bug
|
||||||
|
relates to ISO version != CDN latest was tested and ruled out. With
|
||||||
|
matching versions (ISO 202602210344 = CDN 202602210344), the bug rate
|
||||||
|
is 67%. Sysext downloads still occur with matching versions (SecureBoot
|
||||||
|
+ application updates) but no OS update download. The bug occurs
|
||||||
|
regardless of whether sysext downloads happen.
|
||||||
|
- **Sysext download on matching versions**: even when ISO version matches
|
||||||
|
CDN latest, the first boot downloads SecureBoot update (~1s) and
|
||||||
|
application sysext (~2 min for Incus). The sysext is NOT in the ISO —
|
||||||
|
it's always downloaded from the update provider. An OS update download
|
||||||
|
(and reboot) only occurs when versions differ.
|
||||||
- **Proposed upstream fix** (for `incus-os` project):
|
- **Proposed upstream fix** (for `incus-os` project):
|
||||||
```go
|
```go
|
||||||
// In registerJobs(): validate and fall back to default
|
// In registerJobs(): validate and fall back to default
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,992 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# investigate-boot - Systematic IncusOS boot failure investigation
|
||||||
|
#
|
||||||
|
# Runs repeated deploy/destroy cycles with observe-deploy-style screenshot
|
||||||
|
# capture, PLUS probes the IncusOS REST API after each deploy to capture
|
||||||
|
# state.txt contents. Records structured results for analysis.
|
||||||
|
#
|
||||||
|
# Supports different ISO sources to test the version mismatch hypothesis:
|
||||||
|
# --iso-source cdn Use latest CDN ISO (downloads if not on Proxmox)
|
||||||
|
# --iso-source old Use IncusOS_202602200553.iso (already on Proxmox)
|
||||||
|
# --iso-source FILE Use a specific ISO file path
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./investigate-boot --runs 5 --iso-source old --label old-iso
|
||||||
|
# ./investigate-boot --runs 5 --iso-source cdn --label new-iso
|
||||||
|
# ./investigate-boot --runs 5 --iso-source /tmp/IncusOS-oc.iso --label oc-iso
|
||||||
|
# ./investigate-boot --dry-run
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
readonly VERSION="0.1.0"
|
||||||
|
readonly SCRIPT_NAME="investigate-boot"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Defaults
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
VMID=850
|
||||||
|
VM_NAME="investigate-boot"
|
||||||
|
RUNS=5
|
||||||
|
ISO_SOURCE="old" # "cdn", "old", or a file path
|
||||||
|
LABEL=""
|
||||||
|
DRY_RUN=false
|
||||||
|
SCREENSHOT_INTERVAL=2
|
||||||
|
INSTALL_TIMEOUT=600
|
||||||
|
BOOT_TIMEOUT=180
|
||||||
|
INSTALL_POLL=5
|
||||||
|
STABLE_THRESHOLD=3
|
||||||
|
APPLY_DEFAULTS=true # true for standalone, false for OC-destined nodes
|
||||||
|
|
||||||
|
# Proxmox connection (from proxmox.yaml)
|
||||||
|
PVE_HOST=""
|
||||||
|
PVE_NODE=""
|
||||||
|
PVE_STORAGE=""
|
||||||
|
PVE_ISO_STORAGE=""
|
||||||
|
PVE_BRIDGE=""
|
||||||
|
PVE_POOL=""
|
||||||
|
PVE_API_TOKEN_ID=""
|
||||||
|
|
||||||
|
# ISO tracking
|
||||||
|
ISO_FILENAME=""
|
||||||
|
ISO_VERSION=""
|
||||||
|
ISO_UPLOADED_BY_US=false
|
||||||
|
|
||||||
|
# Run output
|
||||||
|
RESULTS_DIR=""
|
||||||
|
RUN_DIR=""
|
||||||
|
LOG_FILE=""
|
||||||
|
FRAME_NUM=0
|
||||||
|
|
||||||
|
# State tracking
|
||||||
|
PHASE="init"
|
||||||
|
PREV_WR_BYTES=0
|
||||||
|
WRITES_STARTED=false
|
||||||
|
STABLE_COUNT=0
|
||||||
|
|
||||||
|
# Results accumulator
|
||||||
|
declare -a RUN_OUTCOMES=()
|
||||||
|
declare -a RUN_TIMES=()
|
||||||
|
declare -a RUN_SCRUB_SCHEDULES=()
|
||||||
|
declare -a RUN_CHECK_FREQUENCIES=()
|
||||||
|
declare -a RUN_OS_VERSIONS=()
|
||||||
|
declare -a RUN_DIRS=()
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Colors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
setup_colors() {
|
||||||
|
if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
|
||||||
|
RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[0;33m'
|
||||||
|
BLUE='\033[0;34m' CYAN='\033[0;36m' BOLD='\033[1m'
|
||||||
|
DIM='\033[2m' RESET='\033[0m'
|
||||||
|
else
|
||||||
|
RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET=''
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
setup_colors
|
||||||
|
|
||||||
|
step() { echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
||||||
|
success() { echo -e "${GREEN}[ok]${RESET} $*"; }
|
||||||
|
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
||||||
|
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
||||||
|
info() { echo -e "${BLUE}[info]${RESET} $*"; }
|
||||||
|
detail() { echo -e "${DIM} $*${RESET}"; }
|
||||||
|
|
||||||
|
log() {
|
||||||
|
local ts
|
||||||
|
ts=$(date '+%H:%M:%S.%3N')
|
||||||
|
local msg="[${ts}] $*"
|
||||||
|
echo -e "${DIM}${msg}${RESET}"
|
||||||
|
[[ -n "${LOG_FILE:-}" ]] && echo "$msg" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_quiet() {
|
||||||
|
local ts
|
||||||
|
ts=$(date '+%H:%M:%S.%3N')
|
||||||
|
[[ -n "${LOG_FILE:-}" ]] && echo "[${ts}] $*" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Argument parsing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--vmid) VMID="${2:?'--vmid requires a number'}"; shift 2 ;;
|
||||||
|
--runs) RUNS="${2:?'--runs requires a number'}"; shift 2 ;;
|
||||||
|
--iso-source) ISO_SOURCE="${2:?'--iso-source requires cdn|old|path'}"; shift 2 ;;
|
||||||
|
--label) LABEL="${2:?'--label requires a string'}"; shift 2 ;;
|
||||||
|
--no-defaults) APPLY_DEFAULTS=false; shift ;;
|
||||||
|
--interval) SCREENSHOT_INTERVAL="${2:?'--interval requires seconds'}"; shift 2 ;;
|
||||||
|
--dry-run) DRY_RUN=true; shift ;;
|
||||||
|
-h|--help)
|
||||||
|
cat <<EOF
|
||||||
|
${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} - Systematic IncusOS boot failure investigation
|
||||||
|
|
||||||
|
${BOLD}USAGE${RESET}
|
||||||
|
${SCRIPT_NAME} [OPTIONS]
|
||||||
|
|
||||||
|
${BOLD}OPTIONS${RESET}
|
||||||
|
--runs N Number of deploy/destroy cycles (default: 5)
|
||||||
|
--iso-source SRC ISO source: "cdn" (latest), "old" (202602200553), or file path
|
||||||
|
--vmid N VMID to use (default: 850, must be 850-869)
|
||||||
|
--label NAME Label for results directory
|
||||||
|
--no-defaults Use apply_defaults: false in seed
|
||||||
|
--interval N Screenshot interval in seconds (default: 2)
|
||||||
|
--dry-run Preview without deploying
|
||||||
|
-h, --help Show this help
|
||||||
|
|
||||||
|
${BOLD}EXAMPLES${RESET}
|
||||||
|
${SCRIPT_NAME} --runs 5 --iso-source old --label baseline-old
|
||||||
|
${SCRIPT_NAME} --runs 5 --iso-source cdn --label baseline-new
|
||||||
|
${SCRIPT_NAME} --runs 5 --iso-source /tmp/IncusOS-oc.iso --label oc-iso
|
||||||
|
|
||||||
|
${BOLD}OUTPUT${RESET}
|
||||||
|
Results saved to: observe-runs/investigation_<timestamp>_<label>/
|
||||||
|
run-01/frame-*.png, deploy.log
|
||||||
|
run-02/...
|
||||||
|
results.txt (structured results matrix)
|
||||||
|
summary.txt (human-readable summary)
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*) error "Unknown option: $1"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load environment
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
load_env() {
|
||||||
|
local dir="$SCRIPT_DIR"
|
||||||
|
while [[ "$dir" != "/" ]]; do
|
||||||
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "${dir}/env"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
dir="$(dirname "$dir")"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
load_env
|
||||||
|
|
||||||
|
if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then
|
||||||
|
error "PROXMOX_TOKEN_SECRET not set. Create an env file or export it."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]]; then
|
||||||
|
error "PROXMOX_ROOT_PASSWORD not set. Add it to the env file."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load proxmox.yaml
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
yaml_get() {
|
||||||
|
local file="$1" key="$2" default="${3:-}"
|
||||||
|
local val
|
||||||
|
val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val=""
|
||||||
|
val="${val#\"}" ; val="${val%\"}"
|
||||||
|
val="${val#\'}" ; val="${val%\'}"
|
||||||
|
echo "${val:-$default}"
|
||||||
|
}
|
||||||
|
|
||||||
|
load_proxmox_yaml() {
|
||||||
|
local yaml_file=""
|
||||||
|
if [[ -f "${SCRIPT_DIR}/proxmox.yaml" ]]; then
|
||||||
|
yaml_file="${SCRIPT_DIR}/proxmox.yaml"
|
||||||
|
elif [[ -f "proxmox.yaml" ]]; then
|
||||||
|
yaml_file="proxmox.yaml"
|
||||||
|
else
|
||||||
|
error "proxmox.yaml not found"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
PVE_HOST=$(yaml_get "$yaml_file" "host")
|
||||||
|
PVE_NODE=$(yaml_get "$yaml_file" "node" "pve")
|
||||||
|
PVE_STORAGE=$(yaml_get "$yaml_file" "storage" "local-lvm")
|
||||||
|
PVE_ISO_STORAGE=$(yaml_get "$yaml_file" "iso_storage" "local")
|
||||||
|
PVE_BRIDGE=$(yaml_get "$yaml_file" "bridge" "vmbr0")
|
||||||
|
PVE_POOL=$(yaml_get "$yaml_file" "pool")
|
||||||
|
PVE_API_TOKEN_ID=$(yaml_get "$yaml_file" "api_token_id")
|
||||||
|
|
||||||
|
if [[ -z "$PVE_HOST" ]]; then
|
||||||
|
error "proxmox.yaml: 'host' is required"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Proxmox API helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
pve_api() {
|
||||||
|
local method="$1" endpoint="$2" data="${3:-}" content_type="${4:-form}"
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json${endpoint}"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
local curl_args=(-fsSk -H "$auth")
|
||||||
|
[[ "$content_type" == "json" ]] && curl_args+=(-H "Content-Type: application/json")
|
||||||
|
case "$method" in
|
||||||
|
GET) curl "${curl_args[@]}" "$url" ;;
|
||||||
|
POST) curl "${curl_args[@]}" -X POST ${data:+-d "$data"} "$url" ;;
|
||||||
|
PUT) curl "${curl_args[@]}" -X PUT ${data:+-d "$data"} "$url" ;;
|
||||||
|
DELETE) curl "${curl_args[@]}" -X DELETE "$url" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_exists() { pve_api GET "/nodes/${PVE_NODE}/qemu/$1/status/current" &>/dev/null; }
|
||||||
|
pve_vm_status() { pve_api GET "/nodes/${PVE_NODE}/qemu/$1/status/current" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['status'])" 2>/dev/null; }
|
||||||
|
pve_start_vm() { pve_api POST "/nodes/${PVE_NODE}/qemu/$1/status/start" &>/dev/null; }
|
||||||
|
pve_stop_vm() { pve_api POST "/nodes/${PVE_NODE}/qemu/$1/status/stop" &>/dev/null; }
|
||||||
|
|
||||||
|
pve_destroy_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
local resp
|
||||||
|
resp=$(pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" 2>/dev/null) || return 1
|
||||||
|
local upid
|
||||||
|
upid=$(echo "$resp" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid=""
|
||||||
|
if [[ -n "$upid" ]]; then
|
||||||
|
local enc
|
||||||
|
enc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null)
|
||||||
|
local w=0
|
||||||
|
while [[ $w -lt 30 ]]; do
|
||||||
|
sleep 2; w=$((w + 2))
|
||||||
|
local st
|
||||||
|
st=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${enc}/status" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || st=""
|
||||||
|
[[ "$st" == "stopped" ]] && break
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_disk_writes() {
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/$1/status/current" 2>/dev/null | python3 -c "
|
||||||
|
import json,sys
|
||||||
|
d = json.load(sys.stdin).get('data',{})
|
||||||
|
print(d.get('blockstat',{}).get('scsi0',{}).get('wr_bytes',0))
|
||||||
|
" 2>/dev/null || echo "0"
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_mac() {
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/$1/config" 2>/dev/null | python3 -c "
|
||||||
|
import json, sys, re
|
||||||
|
net0 = json.load(sys.stdin).get('data',{}).get('net0','')
|
||||||
|
m = re.search(r'([0-9A-Fa-f:]{17})', net0)
|
||||||
|
print(m.group(1).lower() if m else '')
|
||||||
|
" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_ip() {
|
||||||
|
local vmid="$1"
|
||||||
|
local mac
|
||||||
|
mac=$(pve_vm_mac "$vmid") || return 1
|
||||||
|
[[ -z "$mac" ]] && return 1
|
||||||
|
ip neigh flush nud stale 2>/dev/null || true
|
||||||
|
local bridge_subnet
|
||||||
|
bridge_subnet=$(ip -4 addr show | awk '/inet / && !/127\.0\.0/ {print $2; exit}')
|
||||||
|
if [[ -n "$bridge_subnet" ]]; then
|
||||||
|
local broadcast
|
||||||
|
broadcast=$(python3 -c "import ipaddress; print(ipaddress.ip_interface('${bridge_subnet}').network.broadcast_address)" 2>/dev/null) || true
|
||||||
|
[[ -n "${broadcast:-}" ]] && { ping -c 2 -W 1 -b "$broadcast" &>/dev/null || true; sleep 1; }
|
||||||
|
fi
|
||||||
|
local found_ip
|
||||||
|
found_ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/ || /STALE/) {print $1; exit}') || true
|
||||||
|
if [[ -n "$found_ip" ]] && ping -c 1 -W 1 "$found_ip" &>/dev/null; then
|
||||||
|
echo "$found_ip"; return 0
|
||||||
|
fi
|
||||||
|
if [[ -n "$bridge_subnet" ]]; then
|
||||||
|
local hosts
|
||||||
|
hosts=$(python3 -c "
|
||||||
|
import ipaddress
|
||||||
|
for h in ipaddress.ip_interface('${bridge_subnet}').network.hosts():
|
||||||
|
print(h)
|
||||||
|
" 2>/dev/null)
|
||||||
|
if [[ -n "$hosts" ]]; then
|
||||||
|
while IFS= read -r hip; do ping -c 1 -W 1 "$hip" &>/dev/null & done <<< "$hosts"
|
||||||
|
wait; sleep 1
|
||||||
|
found_ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/) {print $1; exit}') || true
|
||||||
|
if [[ -n "$found_ip" ]] && ping -c 1 -W 1 "$found_ip" &>/dev/null; then
|
||||||
|
echo "$found_ip"; return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Screenshot capture
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -o LogLevel=ERROR)
|
||||||
|
|
||||||
|
capture_screenshot() {
|
||||||
|
local vmid="$1" output_png="$2"
|
||||||
|
local rppm="/tmp/vm-${vmid}-screen.ppm" lppm="/tmp/vm-${vmid}-screen.ppm"
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" "root@${PVE_HOST}" \
|
||||||
|
"echo 'screendump ${rppm}' | qm monitor ${vmid}" &>/dev/null || return 1
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}:${rppm}" "$lppm" &>/dev/null || return 1
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}" "rm -f ${rppm}" &>/dev/null || true
|
||||||
|
python3 -c "from PIL import Image; Image.open('${lppm}').save('${output_png}')" 2>/dev/null || {
|
||||||
|
mv "$lppm" "${output_png%.png}.ppm" 2>/dev/null || true; return 0
|
||||||
|
}
|
||||||
|
rm -f "$lppm"; return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
take_frame() {
|
||||||
|
FRAME_NUM=$((FRAME_NUM + 1))
|
||||||
|
local fname
|
||||||
|
fname=$(printf "frame-%03d.png" "$FRAME_NUM")
|
||||||
|
if capture_screenshot "$VMID" "${RUN_DIR}/${fname}"; then
|
||||||
|
log_quiet "Frame ${FRAME_NUM}: ${fname} (phase=${PHASE})"
|
||||||
|
else
|
||||||
|
log_quiet "Frame ${FRAME_NUM}: FAILED (phase=${PHASE})"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ISO management
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
resolve_iso() {
|
||||||
|
step "Resolving ISO source: ${ISO_SOURCE}"
|
||||||
|
|
||||||
|
case "$ISO_SOURCE" in
|
||||||
|
old)
|
||||||
|
ISO_FILENAME="IncusOS_202602200553.iso"
|
||||||
|
ISO_VERSION="202602200553"
|
||||||
|
# Check it exists on Proxmox
|
||||||
|
local iso_list
|
||||||
|
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null) || iso_list=""
|
||||||
|
if ! echo "$iso_list" | python3 -c "import json,sys; data=json.load(sys.stdin).get('data',[]); assert any('${ISO_FILENAME}' in d.get('volid','') for d in data)" 2>/dev/null; then
|
||||||
|
error "ISO ${ISO_FILENAME} not found on Proxmox. Upload it first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
success "Using existing ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
;;
|
||||||
|
cdn)
|
||||||
|
# Get latest version from CDN
|
||||||
|
local index_json
|
||||||
|
index_json=$(curl -fsSk "https://images.linuxcontainers.org/os/index.json" 2>/dev/null) || {
|
||||||
|
error "Failed to fetch CDN index"; exit 1
|
||||||
|
}
|
||||||
|
ISO_VERSION=$(echo "$index_json" | python3 -c "
|
||||||
|
import json, sys
|
||||||
|
data = json.load(sys.stdin)
|
||||||
|
versions = sorted(data.get('versions', {}).keys(), reverse=True)
|
||||||
|
print(versions[0] if versions else '')
|
||||||
|
" 2>/dev/null)
|
||||||
|
[[ -z "$ISO_VERSION" ]] && { error "Could not determine latest CDN version"; exit 1; }
|
||||||
|
ISO_FILENAME="IncusOS_${ISO_VERSION}.iso"
|
||||||
|
info "Latest CDN version: ${ISO_VERSION}"
|
||||||
|
|
||||||
|
# Check if already on Proxmox
|
||||||
|
local iso_list
|
||||||
|
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null) || iso_list=""
|
||||||
|
if echo "$iso_list" | python3 -c "import json,sys; data=json.load(sys.stdin).get('data',[]); assert any('${ISO_FILENAME}' in d.get('volid','') for d in data)" 2>/dev/null; then
|
||||||
|
success "ISO already on Proxmox: ${ISO_FILENAME}"
|
||||||
|
else
|
||||||
|
step "Downloading ISO from CDN"
|
||||||
|
local dl_url="https://images.linuxcontainers.org/os/${ISO_VERSION}/x86_64/IncusOS_${ISO_VERSION}.iso.gz"
|
||||||
|
local tmp_gz="/tmp/${ISO_FILENAME}.gz"
|
||||||
|
local tmp_iso="/tmp/${ISO_FILENAME}"
|
||||||
|
curl -fSk -o "$tmp_gz" "$dl_url" 2>&1 | tail -1 || {
|
||||||
|
error "Failed to download ${dl_url}"; exit 1
|
||||||
|
}
|
||||||
|
info "Decompressing..."
|
||||||
|
gunzip -f "$tmp_gz"
|
||||||
|
step "Uploading to Proxmox"
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
curl -fsSk -H "$auth" -F "content=iso" \
|
||||||
|
-F "filename=@${tmp_iso};filename=${ISO_FILENAME}" "$url" &>/dev/null || {
|
||||||
|
error "Failed to upload ISO"; exit 1
|
||||||
|
}
|
||||||
|
rm -f "$tmp_iso"
|
||||||
|
ISO_UPLOADED_BY_US=true
|
||||||
|
success "Uploaded: ${ISO_FILENAME}"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
# File path
|
||||||
|
if [[ ! -f "$ISO_SOURCE" ]]; then
|
||||||
|
error "ISO file not found: ${ISO_SOURCE}"; exit 1
|
||||||
|
fi
|
||||||
|
local basename
|
||||||
|
basename=$(basename "$ISO_SOURCE")
|
||||||
|
ISO_FILENAME="$basename"
|
||||||
|
ISO_VERSION=$(echo "$basename" | sed -n 's/.*_\([0-9]*\)\..*/\1/p')
|
||||||
|
[[ -z "$ISO_VERSION" ]] && ISO_VERSION="custom"
|
||||||
|
|
||||||
|
# Upload to Proxmox
|
||||||
|
step "Uploading custom ISO: ${basename}"
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
curl -fsSk -H "$auth" -F "content=iso" \
|
||||||
|
-F "filename=@${ISO_SOURCE};filename=${basename}" "$url" &>/dev/null || {
|
||||||
|
error "Failed to upload ISO"; exit 1
|
||||||
|
}
|
||||||
|
ISO_UPLOADED_BY_US=true
|
||||||
|
success "Using custom ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Seed generation
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SEED_ISO=""
|
||||||
|
SEED_TMPDIR=""
|
||||||
|
|
||||||
|
generate_seed() {
|
||||||
|
step "Generating seed ISO"
|
||||||
|
SEED_TMPDIR=$(mktemp -d)
|
||||||
|
local seed_name="seed-${VM_NAME}.iso"
|
||||||
|
SEED_ISO="$seed_name"
|
||||||
|
local local_path="${SEED_TMPDIR}/${seed_name}"
|
||||||
|
|
||||||
|
local defaults_flag=""
|
||||||
|
if [[ "$APPLY_DEFAULTS" == true ]]; then
|
||||||
|
defaults_flag="--defaults"
|
||||||
|
else
|
||||||
|
defaults_flag="--no-defaults"
|
||||||
|
fi
|
||||||
|
|
||||||
|
"$SEED_TOOL" --format iso --app incus --output "$local_path" \
|
||||||
|
--force-install --force-reboot --hostname "$VM_NAME" \
|
||||||
|
$defaults_flag --quiet &>/dev/null
|
||||||
|
|
||||||
|
success "Seed ISO: ${seed_name} (apply_defaults=${APPLY_DEFAULTS})"
|
||||||
|
|
||||||
|
step "Uploading seed ISO"
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
curl -fsSk -H "$auth" -F "content=iso" \
|
||||||
|
-F "filename=@${local_path};filename=${seed_name}" "$url" &>/dev/null
|
||||||
|
success "Seed uploaded"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# VM lifecycle
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
create_vm() {
|
||||||
|
local json_payload
|
||||||
|
json_payload=$(python3 -c "
|
||||||
|
import json
|
||||||
|
payload = {
|
||||||
|
'vmid': ${VMID},
|
||||||
|
'name': '${VM_NAME}',
|
||||||
|
'description': '[incusos-lab:managed] investigate-boot',
|
||||||
|
'ostype': 'l26',
|
||||||
|
'bios': 'ovmf',
|
||||||
|
'machine': 'q35',
|
||||||
|
'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0',
|
||||||
|
'tpmstate0': '${PVE_STORAGE}:1,version=v2.0',
|
||||||
|
'cpu': 'host',
|
||||||
|
'cores': 4,
|
||||||
|
'memory': 4096,
|
||||||
|
'balloon': 0,
|
||||||
|
'scsihw': 'virtio-scsi-pci',
|
||||||
|
'scsi0': '${PVE_STORAGE}:50,iothread=1',
|
||||||
|
'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom',
|
||||||
|
'ide3': '${PVE_ISO_STORAGE}:iso/${SEED_ISO},media=cdrom',
|
||||||
|
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
||||||
|
'boot': 'order=ide2;scsi0',
|
||||||
|
'agent': 1,
|
||||||
|
}
|
||||||
|
pool = '${PVE_POOL}'
|
||||||
|
if pool:
|
||||||
|
payload['pool'] = pool
|
||||||
|
print(json.dumps(payload))
|
||||||
|
")
|
||||||
|
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/qemu"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
local resp
|
||||||
|
resp=$(curl -fsSk -H "$auth" -H "Content-Type: application/json" \
|
||||||
|
-X POST -d "$json_payload" "$url" 2>/dev/null) || { error "Failed to create VM"; return 1; }
|
||||||
|
|
||||||
|
# Wait for async task
|
||||||
|
local upid
|
||||||
|
upid=$(echo "$resp" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid=""
|
||||||
|
if [[ -n "$upid" ]]; then
|
||||||
|
local enc
|
||||||
|
enc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null)
|
||||||
|
local w=0
|
||||||
|
while [[ $w -lt 60 ]]; do
|
||||||
|
sleep 2; w=$((w + 2))
|
||||||
|
local st
|
||||||
|
st=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${enc}/status" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || st=""
|
||||||
|
[[ "$st" == "stopped" ]] && break
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup_vm() {
|
||||||
|
local status
|
||||||
|
status=$(pve_vm_status "$VMID" 2>/dev/null) || status="unknown"
|
||||||
|
[[ "$status" == "running" ]] && { pve_stop_vm "$VMID" || true; sleep 3; }
|
||||||
|
pve_destroy_vm "$VMID" || true
|
||||||
|
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${SEED_ISO}" &>/dev/null || true
|
||||||
|
if command -v incus &>/dev/null; then
|
||||||
|
local cur
|
||||||
|
cur=$(incus remote get-default 2>/dev/null) || true
|
||||||
|
[[ "$cur" == "$VM_NAME" ]] && { incus remote switch local 2>/dev/null || true; }
|
||||||
|
incus remote remove "$VM_NAME" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -rf "${SEED_TMPDIR:-}" 2>/dev/null || true
|
||||||
|
rm -f "/tmp/vm-${VMID}-screen.ppm" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# IncusOS API probing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
probe_incusos_api() {
|
||||||
|
# Probe the IncusOS REST API via the Incus /os/ proxy.
|
||||||
|
# Requires that the VM has been added as an incus remote.
|
||||||
|
local ip="$1"
|
||||||
|
local probe_file="${RUN_DIR}/api-probe.json"
|
||||||
|
local cert_file="${HOME}/.config/incus/client.crt"
|
||||||
|
local key_file="${HOME}/.config/incus/client.key"
|
||||||
|
|
||||||
|
if [[ ! -f "$cert_file" ]] || [[ ! -f "$key_file" ]]; then
|
||||||
|
log "API probe: no client cert available"
|
||||||
|
echo '{"error": "no client cert"}' > "$probe_file"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Probing IncusOS API at ${ip}..."
|
||||||
|
|
||||||
|
# Add as temporary incus remote
|
||||||
|
incus remote add "$VM_NAME" "https://${ip}:8443" --accept-certificate 2>/dev/null || true
|
||||||
|
|
||||||
|
# Probe storage config (scrub_schedule)
|
||||||
|
local storage_json
|
||||||
|
storage_json=$(incus query "${VM_NAME}:/os/1.0/system/storage" 2>/dev/null) || storage_json='{"error":"query failed"}'
|
||||||
|
|
||||||
|
# Probe update config (check_frequency)
|
||||||
|
local update_json
|
||||||
|
update_json=$(incus query "${VM_NAME}:/os/1.0/system/update" 2>/dev/null) || update_json='{"error":"query failed"}'
|
||||||
|
|
||||||
|
# Probe system info (os_version, hostname, uptime)
|
||||||
|
local system_json
|
||||||
|
system_json=$(incus query "${VM_NAME}:/os/1.0" 2>/dev/null) || system_json='{"error":"query failed"}'
|
||||||
|
|
||||||
|
# Write combined probe results
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
storage = json.loads('''${storage_json}''')
|
||||||
|
update = json.loads('''${update_json}''')
|
||||||
|
system = json.loads('''${system_json}''')
|
||||||
|
result = {
|
||||||
|
'storage': storage,
|
||||||
|
'update': update,
|
||||||
|
'system': system,
|
||||||
|
'scrub_schedule': storage.get('config', {}).get('scrub_schedule', ''),
|
||||||
|
'check_frequency': update.get('config', {}).get('check_frequency', ''),
|
||||||
|
'os_version': system.get('environment', {}).get('os_version', ''),
|
||||||
|
'hostname': system.get('environment', {}).get('hostname', ''),
|
||||||
|
'uptime': system.get('environment', {}).get('uptime', 0),
|
||||||
|
}
|
||||||
|
print(json.dumps(result, indent=2))
|
||||||
|
" > "$probe_file" 2>/dev/null || echo '{"error":"probe parse failed"}' > "$probe_file"
|
||||||
|
|
||||||
|
# Extract key values for summary
|
||||||
|
local scrub check_freq os_ver
|
||||||
|
scrub=$(python3 -c "import json; d=json.load(open('${probe_file}')); print(d.get('scrub_schedule',''))" 2>/dev/null) || scrub=""
|
||||||
|
check_freq=$(python3 -c "import json; d=json.load(open('${probe_file}')); print(d.get('check_frequency',''))" 2>/dev/null) || check_freq=""
|
||||||
|
os_ver=$(python3 -c "import json; d=json.load(open('${probe_file}')); print(d.get('os_version',''))" 2>/dev/null) || os_ver=""
|
||||||
|
|
||||||
|
log "API probe results:"
|
||||||
|
log " scrub_schedule: '${scrub}'"
|
||||||
|
log " check_frequency: '${check_freq}'"
|
||||||
|
log " os_version: '${os_ver}'"
|
||||||
|
|
||||||
|
# Store for summary
|
||||||
|
RUN_SCRUB_SCHEDULES+=("$scrub")
|
||||||
|
RUN_CHECK_FREQUENCIES+=("$check_freq")
|
||||||
|
RUN_OS_VERSIONS+=("$os_ver")
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Install monitoring
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
check_install_progress() {
|
||||||
|
local wr_bytes
|
||||||
|
wr_bytes=$(pve_vm_disk_writes "$VMID" 2>/dev/null) || wr_bytes=0
|
||||||
|
if [[ $wr_bytes -gt $PREV_WR_BYTES ]]; then
|
||||||
|
if [[ "$WRITES_STARTED" != true ]]; then
|
||||||
|
WRITES_STARTED=true
|
||||||
|
PHASE="installing"
|
||||||
|
log "Install started — disk writes detected"
|
||||||
|
fi
|
||||||
|
STABLE_COUNT=0
|
||||||
|
elif [[ "$WRITES_STARTED" == true ]]; then
|
||||||
|
STABLE_COUNT=$((STABLE_COUNT + 1))
|
||||||
|
if [[ $STABLE_COUNT -ge $STABLE_THRESHOLD ]]; then
|
||||||
|
local wr_mb=$(( wr_bytes / 1048576 ))
|
||||||
|
log "Install complete — ${wr_mb} MiB written, ${STABLE_COUNT} stable polls"
|
||||||
|
PHASE="install_done"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
PREV_WR_BYTES=$wr_bytes
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Single deploy cycle
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
run_single_deploy() {
|
||||||
|
local run_num="$1"
|
||||||
|
local deploy_start
|
||||||
|
deploy_start=$(date +%s)
|
||||||
|
|
||||||
|
# Reset state
|
||||||
|
PHASE="init"
|
||||||
|
PREV_WR_BYTES=0
|
||||||
|
WRITES_STARTED=false
|
||||||
|
STABLE_COUNT=0
|
||||||
|
FRAME_NUM=0
|
||||||
|
|
||||||
|
# Setup run directory
|
||||||
|
RUN_DIR="${RESULTS_DIR}/run-$(printf '%02d' "$run_num")"
|
||||||
|
mkdir -p "$RUN_DIR"
|
||||||
|
LOG_FILE="${RUN_DIR}/deploy.log"
|
||||||
|
touch "$LOG_FILE"
|
||||||
|
|
||||||
|
log "=== Run ${run_num}/${RUNS} ==="
|
||||||
|
log "ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
log "apply_defaults: ${APPLY_DEFAULTS}"
|
||||||
|
|
||||||
|
# Generate seed (fresh each time)
|
||||||
|
generate_seed
|
||||||
|
|
||||||
|
# Create VM
|
||||||
|
step "Run ${run_num}: Creating VM (VMID ${VMID})"
|
||||||
|
create_vm || { error "Failed to create VM"; return 1; }
|
||||||
|
success "VM created"
|
||||||
|
|
||||||
|
# --- Phase 1: Boot from ISO ---
|
||||||
|
step "Run ${run_num}: Starting VM (ISO boot)"
|
||||||
|
pve_start_vm "$VMID"
|
||||||
|
PHASE="iso_boot"
|
||||||
|
log "VM started — phase: iso_boot"
|
||||||
|
|
||||||
|
local elapsed=0
|
||||||
|
local last_bs_check=0
|
||||||
|
|
||||||
|
while [[ "$PHASE" == "iso_boot" ]] || [[ "$PHASE" == "installing" ]]; do
|
||||||
|
take_frame || true
|
||||||
|
local now; now=$(date +%s)
|
||||||
|
if [[ $((now - last_bs_check)) -ge $INSTALL_POLL ]]; then
|
||||||
|
check_install_progress
|
||||||
|
last_bs_check=$now
|
||||||
|
fi
|
||||||
|
sleep "$SCREENSHOT_INTERVAL"
|
||||||
|
elapsed=$((elapsed + SCREENSHOT_INTERVAL))
|
||||||
|
local status; status=$(pve_vm_status "$VMID" 2>/dev/null) || status="unknown"
|
||||||
|
[[ "$status" == "stopped" ]] && { log "VM stopped during install"; PHASE="install_done"; }
|
||||||
|
if [[ $elapsed -ge $INSTALL_TIMEOUT ]]; then
|
||||||
|
warn "Install timeout after ${INSTALL_TIMEOUT}s"
|
||||||
|
take_frame || true
|
||||||
|
PHASE="failed"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if [[ $((elapsed % 30)) -lt $SCREENSHOT_INTERVAL ]]; then
|
||||||
|
local wr_mb=$(( PREV_WR_BYTES / 1048576 ))
|
||||||
|
log "Progress: ${elapsed}s, ${wr_mb} MiB, phase=${PHASE}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
[[ "$PHASE" == "failed" ]] && return 1
|
||||||
|
|
||||||
|
# --- Phase 2: Transition ---
|
||||||
|
PHASE="transitioning"
|
||||||
|
take_frame || true
|
||||||
|
step "Run ${run_num}: Stopping VM, removing media"
|
||||||
|
pve_stop_vm "$VMID" || true
|
||||||
|
local sw=0
|
||||||
|
while [[ $sw -lt 30 ]]; do
|
||||||
|
local st; st=$(pve_vm_status "$VMID" 2>/dev/null) || st="unknown"
|
||||||
|
[[ "$st" == "stopped" ]] && break
|
||||||
|
sleep 2; sw=$((sw + 2))
|
||||||
|
done
|
||||||
|
pve_api PUT "/nodes/${PVE_NODE}/qemu/${VMID}/config" \
|
||||||
|
'{"delete":"ide2,ide3","boot":"order=scsi0"}' json &>/dev/null || {
|
||||||
|
error "Failed to remove install media"; PHASE="failed"; return 1
|
||||||
|
}
|
||||||
|
log "Media removed, boot=scsi0"
|
||||||
|
|
||||||
|
# --- Phase 3: Boot from disk ---
|
||||||
|
step "Run ${run_num}: Starting from disk"
|
||||||
|
pve_start_vm "$VMID"
|
||||||
|
PHASE="disk_boot"
|
||||||
|
log "Disk boot started"
|
||||||
|
|
||||||
|
local boot_elapsed=0
|
||||||
|
local ip_detected=""
|
||||||
|
local ip_started=false
|
||||||
|
|
||||||
|
while [[ "$PHASE" == "disk_boot" ]]; do
|
||||||
|
take_frame || true
|
||||||
|
sleep "$SCREENSHOT_INTERVAL"
|
||||||
|
boot_elapsed=$((boot_elapsed + SCREENSHOT_INTERVAL))
|
||||||
|
|
||||||
|
if [[ $boot_elapsed -ge 30 ]] && [[ "$ip_started" != true ]]; then
|
||||||
|
ip_started=true
|
||||||
|
log "Starting IP detection"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$ip_started" == true ]] && [[ $((boot_elapsed % 10)) -lt $SCREENSHOT_INTERVAL ]]; then
|
||||||
|
local ip
|
||||||
|
ip=$(pve_vm_ip "$VMID" 2>/dev/null) || ip=""
|
||||||
|
if [[ -n "$ip" ]]; then
|
||||||
|
[[ -z "$ip_detected" ]] && { ip_detected="$ip"; log "IP: ${ip}"; }
|
||||||
|
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||||
|
log "Port 8443 UP at ${ip} (${boot_elapsed}s)"
|
||||||
|
PHASE="ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $boot_elapsed -ge $BOOT_TIMEOUT ]]; then
|
||||||
|
warn "Boot timeout after ${BOOT_TIMEOUT}s"
|
||||||
|
take_frame || true
|
||||||
|
PHASE="failed"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $((boot_elapsed % 15)) -lt $SCREENSHOT_INTERVAL ]]; then
|
||||||
|
log "Disk boot: ${boot_elapsed}s, ip=${ip_detected:-none}, phase=${PHASE}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Take final frames
|
||||||
|
if [[ "$PHASE" == "ready" ]]; then
|
||||||
|
for _ in 1 2 3; do sleep "$SCREENSHOT_INTERVAL"; take_frame || true; done
|
||||||
|
fi
|
||||||
|
|
||||||
|
local deploy_end; deploy_end=$(date +%s)
|
||||||
|
local total=$((deploy_end - deploy_start))
|
||||||
|
|
||||||
|
log "Result: phase=${PHASE}, total=${total}s, frames=${FRAME_NUM}"
|
||||||
|
|
||||||
|
# --- Phase 4: API probing ---
|
||||||
|
if [[ "$PHASE" == "ready" ]] && [[ -n "$ip_detected" ]]; then
|
||||||
|
probe_incusos_api "$ip_detected"
|
||||||
|
else
|
||||||
|
RUN_SCRUB_SCHEDULES+=("N/A")
|
||||||
|
RUN_CHECK_FREQUENCIES+=("N/A")
|
||||||
|
RUN_OS_VERSIONS+=("N/A")
|
||||||
|
fi
|
||||||
|
|
||||||
|
RUN_TIMES+=("$total")
|
||||||
|
RUN_DIRS+=("$RUN_DIR")
|
||||||
|
|
||||||
|
if [[ "$PHASE" == "ready" ]]; then
|
||||||
|
RUN_OUTCOMES+=("PASS")
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
RUN_OUTCOMES+=("FAIL")
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Preflight
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
preflight() {
|
||||||
|
step "Preflight checks"
|
||||||
|
local missing=()
|
||||||
|
command -v sshpass &>/dev/null || missing+=("sshpass")
|
||||||
|
command -v python3 &>/dev/null || missing+=("python3")
|
||||||
|
command -v curl &>/dev/null || missing+=("curl")
|
||||||
|
[[ ${#missing[@]} -gt 0 ]] && { error "Missing: ${missing[*]}"; exit 1; }
|
||||||
|
|
||||||
|
python3 -c "from PIL import Image" &>/dev/null || warn "PIL not available — PPM screenshots"
|
||||||
|
|
||||||
|
if ! sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" "root@${PVE_HOST}" "echo ok" &>/dev/null; then
|
||||||
|
error "Cannot SSH to root@${PVE_HOST}"; exit 1
|
||||||
|
fi
|
||||||
|
success "SSH OK"
|
||||||
|
|
||||||
|
if [[ $VMID -lt 850 ]] || [[ $VMID -gt 869 ]]; then
|
||||||
|
error "VMID ${VMID} outside test range (850-869)"; exit 1
|
||||||
|
fi
|
||||||
|
success "VMID ${VMID} in range"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
echo -e "${BOLD}${SCRIPT_NAME}${RESET} v${VERSION}"
|
||||||
|
echo "Systematic IncusOS boot failure investigation"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [[ -z "$LABEL" ]]; then
|
||||||
|
case "$ISO_SOURCE" in
|
||||||
|
cdn) LABEL="cdn-latest" ;;
|
||||||
|
old) LABEL="old-202602200553" ;;
|
||||||
|
*) LABEL="custom-iso" ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo -e "${YELLOW}[dry-run] Preview mode${RESET}"
|
||||||
|
echo ""
|
||||||
|
echo "Configuration:"
|
||||||
|
echo " Runs: ${RUNS}"
|
||||||
|
echo " ISO source: ${ISO_SOURCE}"
|
||||||
|
echo " VMID: ${VMID}"
|
||||||
|
echo " Label: ${LABEL}"
|
||||||
|
echo " apply_defaults: ${APPLY_DEFAULTS}"
|
||||||
|
echo " Screenshots: every ${SCREENSHOT_INTERVAL}s"
|
||||||
|
echo ""
|
||||||
|
load_proxmox_yaml
|
||||||
|
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}"
|
||||||
|
echo ""
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Setup
|
||||||
|
step "Loading configuration"
|
||||||
|
load_proxmox_yaml
|
||||||
|
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
|
||||||
|
|
||||||
|
preflight
|
||||||
|
|
||||||
|
resolve_iso
|
||||||
|
|
||||||
|
# Setup results directory
|
||||||
|
RESULTS_DIR="${SCRIPT_DIR}/observe-runs/investigation_$(date '+%Y-%m-%d_%H-%M-%S')_${LABEL}"
|
||||||
|
mkdir -p "$RESULTS_DIR"
|
||||||
|
success "Results: ${RESULTS_DIR}/"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}Investigation plan:${RESET}"
|
||||||
|
echo " Runs: ${RUNS}"
|
||||||
|
echo " ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
echo " apply_defaults: ${APPLY_DEFAULTS}"
|
||||||
|
echo " VMID: ${VMID}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Trap for cleanup
|
||||||
|
trap_cleanup() {
|
||||||
|
echo ""
|
||||||
|
warn "Interrupted — cleaning up"
|
||||||
|
cleanup_vm 2>/dev/null || true
|
||||||
|
exit 130
|
||||||
|
}
|
||||||
|
trap trap_cleanup INT TERM
|
||||||
|
|
||||||
|
# Run the investigation cycles
|
||||||
|
for ((run=1; run<=RUNS; run++)); do
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}════════════════════════════════════════${RESET}"
|
||||||
|
echo -e "${BOLD} Run ${run}/${RUNS}${RESET}"
|
||||||
|
echo -e "${BOLD}════════════════════════════════════════${RESET}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Ensure VMID is free
|
||||||
|
if pve_vm_exists "$VMID"; then
|
||||||
|
warn "VMID ${VMID} exists — cleaning up from previous run"
|
||||||
|
cleanup_vm
|
||||||
|
sleep 5
|
||||||
|
fi
|
||||||
|
|
||||||
|
run_result=0
|
||||||
|
run_single_deploy "$run" || run_result=$?
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [[ $run_result -eq 0 ]]; then
|
||||||
|
echo -e " Result: ${GREEN}PASS${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " Result: ${RED}FAIL${RESET} (phase: ${PHASE})"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
step "Cleanup run ${run}"
|
||||||
|
cleanup_vm
|
||||||
|
sleep 3 # Brief pause between runs
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Results summary
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}╔══════════════════════════════════════════╗${RESET}"
|
||||||
|
echo -e "${BOLD}║ Investigation Results Summary ║${RESET}"
|
||||||
|
echo -e "${BOLD}╚══════════════════════════════════════════╝${RESET}"
|
||||||
|
echo ""
|
||||||
|
echo "ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
echo "ISO source: ${ISO_SOURCE}"
|
||||||
|
echo "apply_defaults: ${APPLY_DEFAULTS}"
|
||||||
|
echo "Runs: ${RUNS}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
pass_count=0
|
||||||
|
fail_count=0
|
||||||
|
|
||||||
|
for ((i=0; i<${#RUN_OUTCOMES[@]}; i++)); do
|
||||||
|
local_run=$((i + 1))
|
||||||
|
outcome="${RUN_OUTCOMES[$i]}"
|
||||||
|
time_s="${RUN_TIMES[$i]:-?}"
|
||||||
|
scrub="${RUN_SCRUB_SCHEDULES[$i]:-?}"
|
||||||
|
freq="${RUN_CHECK_FREQUENCIES[$i]:-?}"
|
||||||
|
osver="${RUN_OS_VERSIONS[$i]:-?}"
|
||||||
|
|
||||||
|
marker=""
|
||||||
|
case "$outcome" in
|
||||||
|
PASS) marker="${GREEN}PASS${RESET}"; pass_count=$((pass_count + 1)) ;;
|
||||||
|
FAIL) marker="${RED}FAIL${RESET}"; fail_count=$((fail_count + 1)) ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo -e " Run $(printf '%2d' $local_run): ${marker} time=${time_s}s scrub='${scrub}' freq='${freq}' os=${osver}"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e " Total: ${GREEN}${pass_count} PASS${RESET}, ${RED}${fail_count} FAIL${RESET} (of ${RUNS})"
|
||||||
|
|
||||||
|
if [[ $fail_count -gt 0 ]]; then
|
||||||
|
pct=$(( fail_count * 100 / RUNS ))
|
||||||
|
echo -e " Failure rate: ${RED}${pct}%${RESET}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Write structured results file
|
||||||
|
results_file="${RESULTS_DIR}/results.txt"
|
||||||
|
{
|
||||||
|
echo "# investigate-boot results"
|
||||||
|
echo "# Generated: $(date -Iseconds)"
|
||||||
|
echo "# ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
echo "# ISO source: ${ISO_SOURCE}"
|
||||||
|
echo "# apply_defaults: ${APPLY_DEFAULTS}"
|
||||||
|
echo "# Runs: ${RUNS}"
|
||||||
|
echo "#"
|
||||||
|
echo "# run|outcome|time_s|scrub_schedule|check_frequency|os_version|run_dir"
|
||||||
|
for ((i=0; i<${#RUN_OUTCOMES[@]}; i++)); do
|
||||||
|
echo "$((i+1))|${RUN_OUTCOMES[$i]}|${RUN_TIMES[$i]:-?}|${RUN_SCRUB_SCHEDULES[$i]:-?}|${RUN_CHECK_FREQUENCIES[$i]:-?}|${RUN_OS_VERSIONS[$i]:-?}|${RUN_DIRS[$i]:-?}"
|
||||||
|
done
|
||||||
|
} > "$results_file"
|
||||||
|
|
||||||
|
success "Results written to: ${results_file}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
exit $fail_count
|
||||||
|
|
@ -0,0 +1,578 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# investigate-parallel - Parallel IncusOS boot investigation with smart phase detection
|
||||||
|
#
|
||||||
|
# Runs N VMs simultaneously, monitors each independently, probes API on success.
|
||||||
|
# Uses aggressive port polling + screenshot capture for phase detection.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./investigate-parallel --count 4 --iso old --label baseline
|
||||||
|
# ./investigate-parallel --count 4 --iso new --label version-mismatch
|
||||||
|
# ./investigate-parallel --count 4 --iso /tmp/IncusOS-oc.iso --label oc-iso
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
|
||||||
|
PVE_API="${SCRIPT_DIR}/pve-api"
|
||||||
|
|
||||||
|
# Defaults
|
||||||
|
COUNT=4
|
||||||
|
START_VMID=850
|
||||||
|
ISO_MODE="old" # old, new, or file path
|
||||||
|
LABEL=""
|
||||||
|
DRY_RUN=false
|
||||||
|
BOOT_TIMEOUT=240 # longer for version-mismatch OS update path
|
||||||
|
|
||||||
|
# Proxmox config
|
||||||
|
PVE_HOST=""
|
||||||
|
PVE_NODE=""
|
||||||
|
PVE_STORAGE=""
|
||||||
|
PVE_ISO_STORAGE=""
|
||||||
|
PVE_BRIDGE=""
|
||||||
|
PVE_POOL=""
|
||||||
|
|
||||||
|
# ISO
|
||||||
|
ISO_FILENAME=""
|
||||||
|
ISO_VERSION=""
|
||||||
|
|
||||||
|
# Results
|
||||||
|
RESULTS_DIR=""
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Colors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
|
||||||
|
RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[0;33m'
|
||||||
|
CYAN='\033[0;36m' BOLD='\033[1m' DIM='\033[2m' RESET='\033[0m'
|
||||||
|
else
|
||||||
|
RED='' GREEN='' YELLOW='' CYAN='' BOLD='' DIM='' RESET=''
|
||||||
|
fi
|
||||||
|
|
||||||
|
step() { echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
||||||
|
success() { echo -e "${GREEN}[ok]${RESET} $*"; }
|
||||||
|
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
||||||
|
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
||||||
|
info() { echo -e "${DIM}[info]${RESET} $*"; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Argument parsing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--count) COUNT="${2:?'--count requires N'}"; shift 2 ;;
|
||||||
|
--iso) ISO_MODE="${2:?'--iso requires old|new|path'}"; shift 2 ;;
|
||||||
|
--label) LABEL="${2:?'--label requires string'}"; shift 2 ;;
|
||||||
|
--start-vmid) START_VMID="${2:?'--start-vmid requires N'}"; shift 2 ;;
|
||||||
|
--dry-run) DRY_RUN=true; shift ;;
|
||||||
|
--timeout) BOOT_TIMEOUT="${2:?'--timeout requires seconds'}"; shift 2 ;;
|
||||||
|
-h|--help)
|
||||||
|
echo "Usage: investigate-parallel [OPTIONS]"
|
||||||
|
echo " --count N Parallel VMs (default: 4, max 8)"
|
||||||
|
echo " --iso MODE old (202602200553), new (latest CDN), or file path"
|
||||||
|
echo " --label NAME Results label"
|
||||||
|
echo " --start-vmid N First VMID (default: 850)"
|
||||||
|
echo " --timeout N Boot timeout seconds (default: 240)"
|
||||||
|
echo " --dry-run Preview only"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*) error "Unknown: $1"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load environment & config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
load_env() {
|
||||||
|
local dir="$SCRIPT_DIR"
|
||||||
|
while [[ "$dir" != "/" ]]; do
|
||||||
|
[[ -f "${dir}/env" ]] && { source "${dir}/env"; return; }
|
||||||
|
dir="$(dirname "$dir")"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
load_env
|
||||||
|
|
||||||
|
[[ -z "${PROXMOX_TOKEN_SECRET:-}" ]] && { error "PROXMOX_TOKEN_SECRET not set"; exit 1; }
|
||||||
|
[[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]] && { error "PROXMOX_ROOT_PASSWORD not set"; exit 1; }
|
||||||
|
|
||||||
|
yaml_get() {
|
||||||
|
local val
|
||||||
|
val=$(awk -F': ' -v k="$1" '$1 == k {print $2; exit}' "${SCRIPT_DIR}/proxmox.yaml" 2>/dev/null) || val=""
|
||||||
|
val="${val#\"}" ; val="${val%\"}" ; val="${val#\'}" ; val="${val%\'}"
|
||||||
|
echo "${val:-${2:-}}"
|
||||||
|
}
|
||||||
|
|
||||||
|
PVE_HOST=$(yaml_get host)
|
||||||
|
PVE_NODE=$(yaml_get node pve)
|
||||||
|
PVE_STORAGE=$(yaml_get storage local-lvm)
|
||||||
|
PVE_ISO_STORAGE=$(yaml_get iso_storage local)
|
||||||
|
PVE_BRIDGE=$(yaml_get bridge vmbr0)
|
||||||
|
PVE_POOL=$(yaml_get pool)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Screenshot helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -o LogLevel=ERROR)
|
||||||
|
|
||||||
|
take_screenshot() {
|
||||||
|
local vmid="$1" output="$2"
|
||||||
|
local rppm="/tmp/vm-${vmid}-screen.ppm"
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" "root@${PVE_HOST}" \
|
||||||
|
"echo 'screendump ${rppm}' | qm monitor ${vmid}" &>/dev/null || return 1
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}:${rppm}" "/tmp/vm-${vmid}-screen.ppm" &>/dev/null || return 1
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" "root@${PVE_HOST}" \
|
||||||
|
"rm -f ${rppm}" &>/dev/null || true
|
||||||
|
python3 -c "from PIL import Image; Image.open('/tmp/vm-${vmid}-screen.ppm').save('${output}')" 2>/dev/null || {
|
||||||
|
mv "/tmp/vm-${vmid}-screen.ppm" "${output%.png}.ppm" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
rm -f "/tmp/vm-${vmid}-screen.ppm" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ISO resolution
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
resolve_iso() {
|
||||||
|
case "$ISO_MODE" in
|
||||||
|
old)
|
||||||
|
ISO_FILENAME="IncusOS_202602200553.iso"
|
||||||
|
ISO_VERSION="202602200553"
|
||||||
|
;;
|
||||||
|
new)
|
||||||
|
# Find latest on Proxmox
|
||||||
|
local latest
|
||||||
|
latest=$("$PVE_API" "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null | python3 -c "
|
||||||
|
import json, sys
|
||||||
|
data = json.load(sys.stdin).get('data', [])
|
||||||
|
isos = [d['volid'].split('/')[-1] for d in data if 'IncusOS_' in d.get('volid','') and '-oc' not in d['volid'].split('/')[-1].lower()]
|
||||||
|
isos.sort(reverse=True)
|
||||||
|
print(isos[0] if isos else '')
|
||||||
|
" 2>/dev/null)
|
||||||
|
if [[ -z "$latest" ]]; then
|
||||||
|
error "No IncusOS ISO found on Proxmox"; exit 1
|
||||||
|
fi
|
||||||
|
ISO_FILENAME="$latest"
|
||||||
|
ISO_VERSION=$(echo "$latest" | sed -n 's/IncusOS_\([0-9]*\)\..*/\1/p')
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
# File path — must already be uploaded
|
||||||
|
ISO_FILENAME="$(basename "$ISO_MODE")"
|
||||||
|
ISO_VERSION=$(echo "$ISO_FILENAME" | sed -n 's/.*_\([0-9]*\)\..*/\1/p')
|
||||||
|
[[ -z "$ISO_VERSION" ]] && ISO_VERSION="custom"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
success "ISO: ${ISO_FILENAME} (version ${ISO_VERSION})"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Single VM lifecycle (called as background job)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
deploy_single_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
local vm_name="$2"
|
||||||
|
local run_dir="$3"
|
||||||
|
local seed_iso="$4"
|
||||||
|
local result_file="${run_dir}/result.txt"
|
||||||
|
local log_file="${run_dir}/deploy.log"
|
||||||
|
|
||||||
|
_log() { echo "[$(date '+%H:%M:%S.%3N')] $*" >> "$log_file"; }
|
||||||
|
|
||||||
|
# Error trap: write FAIL result on any uncaught error
|
||||||
|
trap '_log "ERROR at line $LINENO: exit code $?"; echo "ERROR|line_${LINENO}|0|N/A|N/A|N/A" > "$result_file"' ERR
|
||||||
|
|
||||||
|
_log "=== VM ${vmid} (${vm_name}) ==="
|
||||||
|
_log "ISO: ${ISO_FILENAME}, seed: ${seed_iso}"
|
||||||
|
|
||||||
|
# Create VM
|
||||||
|
local json_payload
|
||||||
|
json_payload=$(python3 -c "
|
||||||
|
import json
|
||||||
|
p = {
|
||||||
|
'vmid': ${vmid}, 'name': '${vm_name}',
|
||||||
|
'description': '[incusos-lab:managed] investigate-parallel',
|
||||||
|
'ostype': 'l26', 'bios': 'ovmf', 'machine': 'q35',
|
||||||
|
'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0',
|
||||||
|
'tpmstate0': '${PVE_STORAGE}:1,version=v2.0',
|
||||||
|
'cpu': 'host', 'cores': 4, 'memory': 4096, 'balloon': 0,
|
||||||
|
'scsihw': 'virtio-scsi-pci', 'scsi0': '${PVE_STORAGE}:50,iothread=1',
|
||||||
|
'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom',
|
||||||
|
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_iso},media=cdrom',
|
||||||
|
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
||||||
|
'boot': 'order=ide2;scsi0', 'agent': 1,
|
||||||
|
}
|
||||||
|
pool = '${PVE_POOL}'
|
||||||
|
if pool: p['pool'] = pool
|
||||||
|
print(json.dumps(p))
|
||||||
|
")
|
||||||
|
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu" POST "$json_payload" >/dev/null 2>&1 || {
|
||||||
|
echo "ERROR|create_failed|0|N/A|N/A|N/A" > "$result_file"; return
|
||||||
|
}
|
||||||
|
sleep 15 # Wait for create task (allocating EFI, TPM, 50G disk)
|
||||||
|
_log "VM created"
|
||||||
|
|
||||||
|
# Start VM (ISO boot)
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" POST '{}' >/dev/null 2>&1 || {
|
||||||
|
_log "ERROR: Failed to start VM"; echo "ERROR|start_failed|0|N/A|N/A|N/A" > "$result_file"; return
|
||||||
|
}
|
||||||
|
_log "VM started"
|
||||||
|
|
||||||
|
local frame=0
|
||||||
|
|
||||||
|
# Monitor install via blockstat (screenshots every 15s to avoid SSH contention)
|
||||||
|
local elapsed=0 prev_wr=0 writes_started=false stable=0
|
||||||
|
while [[ $elapsed -lt 300 ]]; do
|
||||||
|
sleep 5; elapsed=$((elapsed + 5))
|
||||||
|
# Screenshot every 15s during install
|
||||||
|
if [[ $((elapsed % 15)) -lt 6 ]]; then
|
||||||
|
take_screenshot "$vmid" "${run_dir}/frame-$(printf '%03d' $((++frame))).png" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
local wr
|
||||||
|
wr=$("$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" 2>/dev/null | python3 -c "
|
||||||
|
import json,sys; d=json.load(sys.stdin).get('data',{}); print(d.get('blockstat',{}).get('scsi0',{}).get('wr_bytes',0))" 2>/dev/null) || wr=0
|
||||||
|
|
||||||
|
if [[ $wr -gt $prev_wr ]]; then
|
||||||
|
[[ "$writes_started" != true ]] && { writes_started=true; _log "Install writes started"; }
|
||||||
|
stable=0
|
||||||
|
elif [[ "$writes_started" == true ]]; then
|
||||||
|
stable=$((stable + 1))
|
||||||
|
[[ $stable -ge 3 ]] && { _log "Install complete ($(( wr / 1048576 )) MiB)"; break; }
|
||||||
|
fi
|
||||||
|
prev_wr=$wr
|
||||||
|
|
||||||
|
# Check if VM stopped
|
||||||
|
local st
|
||||||
|
st=$("$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['status'])" 2>/dev/null) || st="unknown"
|
||||||
|
[[ "$st" == "stopped" ]] && { _log "VM stopped during install"; break; }
|
||||||
|
done
|
||||||
|
|
||||||
|
# Take screenshot showing install-complete/media-error state
|
||||||
|
take_screenshot "$vmid" "${run_dir}/frame-$(printf '%03d' $((++frame))).png" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Transition: stop, remove media, boot from disk
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" POST '{}' >/dev/null 2>&1 || true
|
||||||
|
sleep 8
|
||||||
|
# Wait for stop to complete
|
||||||
|
for _ in 1 2 3 4 5; do
|
||||||
|
local vm_st
|
||||||
|
vm_st=$("$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['status'])" 2>/dev/null) || vm_st="unknown"
|
||||||
|
[[ "$vm_st" == "stopped" ]] && break
|
||||||
|
sleep 3
|
||||||
|
done
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/config" "PUT" '{"delete":"ide2,ide3","boot":"order=scsi0"}' >/dev/null 2>&1 || {
|
||||||
|
_log "ERROR: Failed to remove media"; echo "ERROR|media_remove|0|N/A|N/A|N/A" > "$result_file"; return
|
||||||
|
}
|
||||||
|
_log "Media removed"
|
||||||
|
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" POST '{}' >/dev/null 2>&1 || {
|
||||||
|
_log "ERROR: Failed to start from disk"; echo "ERROR|disk_start|0|N/A|N/A|N/A" > "$result_file"; return
|
||||||
|
}
|
||||||
|
_log "Disk boot started"
|
||||||
|
local boot_start
|
||||||
|
boot_start=$(date +%s)
|
||||||
|
|
||||||
|
# Poll for port 8443 with periodic screenshots
|
||||||
|
local ip="" boot_elapsed=0 port_up=false
|
||||||
|
while [[ $boot_elapsed -lt $BOOT_TIMEOUT ]]; do
|
||||||
|
sleep 5; boot_elapsed=$(( $(date +%s) - boot_start ))
|
||||||
|
# Screenshot every 15s during boot
|
||||||
|
if [[ $((boot_elapsed % 15)) -lt 6 ]]; then
|
||||||
|
take_screenshot "$vmid" "${run_dir}/frame-$(printf '%03d' $((++frame))).png" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Try to find IP (after 15s)
|
||||||
|
if [[ $boot_elapsed -ge 15 ]] && [[ -z "$ip" ]]; then
|
||||||
|
# Get MAC from Proxmox config (only once)
|
||||||
|
if [[ -z "${mac:-}" ]]; then
|
||||||
|
mac=$("$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null | python3 -c "
|
||||||
|
import json, sys, re
|
||||||
|
net0 = json.load(sys.stdin).get('data',{}).get('net0','')
|
||||||
|
m = re.search(r'([0-9A-Fa-f:]{17})', net0)
|
||||||
|
print(m.group(1).lower() if m else '')
|
||||||
|
" 2>/dev/null) || mac=""
|
||||||
|
[[ -n "$mac" ]] && _log "MAC: ${mac}"
|
||||||
|
fi
|
||||||
|
if [[ -n "${mac:-}" ]]; then
|
||||||
|
# Check ARP table (don't flush — other VMs are using it)
|
||||||
|
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) {print $1; exit}') || ip=""
|
||||||
|
# If not found, do a targeted ARP probe: ping a range of likely DHCP IPs
|
||||||
|
if [[ -z "$ip" ]]; then
|
||||||
|
# Only sweep every 30 seconds to avoid overload
|
||||||
|
local now_s
|
||||||
|
now_s=$(date +%s)
|
||||||
|
if [[ $(( now_s - ${_last_sweep:-0} )) -ge 25 ]]; then
|
||||||
|
_last_sweep=$now_s
|
||||||
|
_log "Running ARP sweep"
|
||||||
|
# Sweep in smaller batches to avoid overwhelming
|
||||||
|
for octet in $(seq 1 254); do
|
||||||
|
ping -c 1 -W 1 "192.168.1.${octet}" &>/dev/null &
|
||||||
|
done
|
||||||
|
wait 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
fi
|
||||||
|
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) {print $1; exit}') || ip=""
|
||||||
|
fi
|
||||||
|
if [[ -n "$ip" ]]; then
|
||||||
|
if ping -c 1 -W 2 "$ip" &>/dev/null; then
|
||||||
|
_log "IP detected: ${ip}"
|
||||||
|
else
|
||||||
|
_log "IP ${ip} not responding, retrying"
|
||||||
|
ip=""
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check port 8443
|
||||||
|
if [[ -n "$ip" ]]; then
|
||||||
|
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||||
|
port_up=true
|
||||||
|
_log "Port 8443 UP at ${ip} (${boot_elapsed}s)"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ $((boot_elapsed % 30)) -lt 6 ]] && _log "Disk boot: ${boot_elapsed}s, ip=${ip:-none}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Take final screenshot
|
||||||
|
take_screenshot "$vmid" "${run_dir}/frame-$(printf '%03d' $((++frame))).png" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Probe API if successful
|
||||||
|
local scrub="N/A" freq="N/A" osver="N/A"
|
||||||
|
if [[ "$port_up" == true ]] && [[ -n "$ip" ]]; then
|
||||||
|
# Add incus remote and probe
|
||||||
|
incus remote add "$vm_name" "https://${ip}:8443" --accept-certificate 2>/dev/null || true
|
||||||
|
local storage_json update_json system_json
|
||||||
|
storage_json=$(incus query "${vm_name}:/os/1.0/system/storage" 2>/dev/null) || storage_json='{}'
|
||||||
|
update_json=$(incus query "${vm_name}:/os/1.0/system/update" 2>/dev/null) || update_json='{}'
|
||||||
|
system_json=$(incus query "${vm_name}:/os/1.0" 2>/dev/null) || system_json='{}'
|
||||||
|
|
||||||
|
scrub=$(echo "$storage_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('config',{}).get('scrub_schedule','EMPTY'))" 2>/dev/null) || scrub="error"
|
||||||
|
freq=$(echo "$update_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('config',{}).get('check_frequency','EMPTY'))" 2>/dev/null) || freq="error"
|
||||||
|
osver=$(echo "$system_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('environment',{}).get('os_version','EMPTY'))" 2>/dev/null) || osver="error"
|
||||||
|
|
||||||
|
_log "API: scrub='${scrub}' freq='${freq}' os=${osver}"
|
||||||
|
|
||||||
|
# Save full probe data
|
||||||
|
echo "$storage_json" > "${run_dir}/api-storage.json" 2>/dev/null || true
|
||||||
|
echo "$update_json" > "${run_dir}/api-update.json" 2>/dev/null || true
|
||||||
|
echo "$system_json" > "${run_dir}/api-system.json" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Cleanup remote
|
||||||
|
local cur_remote
|
||||||
|
cur_remote=$(incus remote get-default 2>/dev/null) || true
|
||||||
|
[[ "$cur_remote" == "$vm_name" ]] && incus remote switch local 2>/dev/null || true
|
||||||
|
incus remote remove "$vm_name" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Determine outcome:
|
||||||
|
# - PASS = port up AND scrub_schedule is set (genuine success)
|
||||||
|
# - CRONTAB_BUG = port up but scrub_schedule empty (detected during error window)
|
||||||
|
# - FAIL = port never came up (could be bug + bad timing, or IP detection failure)
|
||||||
|
local outcome="FAIL"
|
||||||
|
if [[ "$port_up" == true ]]; then
|
||||||
|
if [[ -n "$scrub" ]] && [[ "$scrub" != "" ]] && [[ "$scrub" != "EMPTY" ]] && [[ "$scrub" != "N/A" ]]; then
|
||||||
|
outcome="PASS"
|
||||||
|
else
|
||||||
|
outcome="CRONTAB_BUG"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "${outcome}|${boot_elapsed}|${scrub}|${freq}|${osver}|${ip:-none}" > "$result_file"
|
||||||
|
_log "RESULT: ${outcome} boot=${boot_elapsed}s scrub='${scrub}' freq='${freq}' os=${osver}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cleanup single VM
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
cleanup_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
local seed_iso="$2"
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" POST >/dev/null 2>&1 || true
|
||||||
|
sleep 3
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" DELETE >/dev/null 2>&1 || true
|
||||||
|
"$PVE_API" "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_iso}" DELETE >/dev/null 2>&1 || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
echo -e "${BOLD}investigate-parallel${RESET} — Parallel IncusOS boot investigation"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
[[ -z "$LABEL" ]] && LABEL="${ISO_MODE}"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo " Count: ${COUNT}, ISO: ${ISO_MODE}, VMIDs: ${START_VMID}-$((START_VMID + COUNT - 1))"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
step "Resolving ISO"
|
||||||
|
resolve_iso
|
||||||
|
|
||||||
|
RESULTS_DIR="${SCRIPT_DIR}/observe-runs/parallel_$(date '+%Y-%m-%d_%H-%M-%S')_${LABEL}"
|
||||||
|
mkdir -p "$RESULTS_DIR"
|
||||||
|
success "Output: ${RESULTS_DIR}/"
|
||||||
|
|
||||||
|
# Generate seeds and create run dirs
|
||||||
|
step "Generating ${COUNT} seeds"
|
||||||
|
declare -a VM_NAMES=() VM_SEEDS=() VM_DIRS=()
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
vmid=$((START_VMID + i))
|
||||||
|
vm_name="inv-${vmid}"
|
||||||
|
seed_name="seed-inv-${vmid}.iso"
|
||||||
|
run_dir="${RESULTS_DIR}/vm-${vmid}"
|
||||||
|
mkdir -p "$run_dir"
|
||||||
|
|
||||||
|
VM_NAMES+=("$vm_name")
|
||||||
|
VM_SEEDS+=("$seed_name")
|
||||||
|
VM_DIRS+=("$run_dir")
|
||||||
|
|
||||||
|
local_seed="/tmp/${seed_name}"
|
||||||
|
"$SEED_TOOL" --format iso --app incus --output "$local_seed" \
|
||||||
|
--force-install --force-reboot --hostname "$vm_name" \
|
||||||
|
--defaults --quiet &>/dev/null
|
||||||
|
|
||||||
|
# Upload seed (use Python to avoid bash ! quoting issues with API token)
|
||||||
|
url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||||
|
python3 -c "
|
||||||
|
import subprocess, os, sys
|
||||||
|
# Read token_id from proxmox.yaml
|
||||||
|
token_id = ''
|
||||||
|
yaml_file = '${SCRIPT_DIR}/proxmox.yaml'
|
||||||
|
for line in open(yaml_file):
|
||||||
|
line = line.strip()
|
||||||
|
if line.startswith('api_token_id:'):
|
||||||
|
token_id = line.split(': ', 1)[1].strip().strip(\"'\\\"\")
|
||||||
|
secret = os.environ.get('PROXMOX_TOKEN_SECRET', '')
|
||||||
|
if not token_id or not secret:
|
||||||
|
sys.exit(1)
|
||||||
|
r = subprocess.run([
|
||||||
|
'curl', '-fsSk',
|
||||||
|
'-H', 'Authorization: PVEAPIToken=' + token_id + '=' + secret,
|
||||||
|
'-F', 'content=iso',
|
||||||
|
'-F', 'filename=@${local_seed};filename=${seed_name}',
|
||||||
|
'${url}'
|
||||||
|
], capture_output=True)
|
||||||
|
sys.exit(r.returncode)
|
||||||
|
" 2>/dev/null
|
||||||
|
rm -f "$local_seed"
|
||||||
|
info "Seed $((i+1))/${COUNT}: ${seed_name}"
|
||||||
|
done
|
||||||
|
success "All seeds uploaded"
|
||||||
|
|
||||||
|
# Check VMIDs are free
|
||||||
|
step "Checking VMIDs"
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
vmid=$((START_VMID + i))
|
||||||
|
if "$PVE_API" "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" >/dev/null 2>&1; then
|
||||||
|
error "VMID ${vmid} already in use"; exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
success "VMIDs ${START_VMID}-$((START_VMID + COUNT - 1)) available"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}Launching ${COUNT} VMs in parallel${RESET}"
|
||||||
|
echo " ISO: ${ISO_FILENAME} (${ISO_VERSION})"
|
||||||
|
echo " VMIDs: ${START_VMID}-$((START_VMID + COUNT - 1))"
|
||||||
|
echo " Boot timeout: ${BOOT_TIMEOUT}s"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Trap for cleanup
|
||||||
|
trap_cleanup() {
|
||||||
|
echo ""
|
||||||
|
warn "Interrupted — cleaning up all VMs"
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
cleanup_vm "$((START_VMID + i))" "${VM_SEEDS[$i]}" &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
exit 130
|
||||||
|
}
|
||||||
|
trap trap_cleanup INT TERM
|
||||||
|
|
||||||
|
# Launch all VMs in parallel
|
||||||
|
pids=()
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
vmid=$((START_VMID + i))
|
||||||
|
( set +e; deploy_single_vm "$vmid" "${VM_NAMES[$i]}" "${VM_DIRS[$i]}" "${VM_SEEDS[$i]}" 2>"${VM_DIRS[$i]}/stderr.log" ) &
|
||||||
|
pids+=($!)
|
||||||
|
info "Started VM ${vmid} (PID $!)"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Wait for all to complete
|
||||||
|
for pid in "${pids[@]}"; do
|
||||||
|
wait "$pid" || true
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}════════════════════════════════════════════════${RESET}"
|
||||||
|
echo -e "${BOLD} Results (ISO: ${ISO_FILENAME})${RESET}"
|
||||||
|
echo -e "${BOLD}════════════════════════════════════════════════${RESET}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
pass=0 fail=0 crontab_bug=0
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
vmid=$((START_VMID + i))
|
||||||
|
rfile="${VM_DIRS[$i]}/result.txt"
|
||||||
|
if [[ -f "$rfile" ]]; then
|
||||||
|
IFS='|' read -r outcome boot_s scrub freq osver vm_ip < "$rfile"
|
||||||
|
if [[ "$outcome" == "PASS" ]]; then
|
||||||
|
marker="${GREEN}PASS${RESET}"; pass=$((pass + 1))
|
||||||
|
elif [[ "$outcome" == "CRONTAB_BUG" ]]; then
|
||||||
|
marker="${YELLOW}CRONTAB_BUG${RESET}"; crontab_bug=$((crontab_bug + 1))
|
||||||
|
else
|
||||||
|
marker="${RED}FAIL${RESET}"; fail=$((fail + 1))
|
||||||
|
fi
|
||||||
|
echo -e " ${vmid}: ${marker} boot=${boot_s}s scrub='${scrub}' freq='${freq}' os=${osver} ip=${vm_ip}"
|
||||||
|
else
|
||||||
|
echo -e " ${vmid}: ${RED}NO RESULT${RESET}"
|
||||||
|
fail=$((fail + 1))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e " Total: ${GREEN}${pass} PASS${RESET}, ${YELLOW}${crontab_bug} CRONTAB_BUG${RESET}, ${RED}${fail} FAIL/NO_IP${RESET} (of ${COUNT})"
|
||||||
|
total_bad=$((crontab_bug + fail))
|
||||||
|
[[ $total_bad -gt 0 ]] && echo -e " Bug rate: ${YELLOW}$(( crontab_bug * 100 / COUNT ))%${RESET} confirmed, ${RED}$(( fail * 100 / COUNT ))%${RESET} unknown (IP detection failure)"
|
||||||
|
|
||||||
|
# Write summary
|
||||||
|
{
|
||||||
|
echo "# investigate-parallel results"
|
||||||
|
echo "# $(date -Iseconds)"
|
||||||
|
echo "# ISO: ${ISO_FILENAME} (${ISO_VERSION})"
|
||||||
|
echo "# Count: ${COUNT}, VMIDs: ${START_VMID}-$((START_VMID + COUNT - 1))"
|
||||||
|
echo "#"
|
||||||
|
echo "# vmid|outcome|boot_s|scrub_schedule|check_frequency|os_version|ip"
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
vmid=$((START_VMID + i))
|
||||||
|
rfile="${VM_DIRS[$i]}/result.txt"
|
||||||
|
[[ -f "$rfile" ]] && echo "${vmid}|$(cat "$rfile")"
|
||||||
|
done
|
||||||
|
} > "${RESULTS_DIR}/results.txt"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Cleanup all VMs
|
||||||
|
step "Cleaning up"
|
||||||
|
for ((i=0; i<COUNT; i++)); do
|
||||||
|
cleanup_vm "$((START_VMID + i))" "${VM_SEEDS[$i]}" &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
sleep 3
|
||||||
|
success "All cleaned up"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Screenshots: ${RESULTS_DIR}/vm-*/frame-*.png"
|
||||||
|
echo "Results: ${RESULTS_DIR}/results.txt"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
exit $fail
|
||||||
|
|
@ -0,0 +1,60 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Proxmox API helper — avoids bash ! quoting issues in interactive shells."""
|
||||||
|
import json, os, ssl, sys, urllib.request
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Load config
|
||||||
|
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
env_file = os.path.join(os.path.dirname(script_dir), "env")
|
||||||
|
if os.path.exists(env_file):
|
||||||
|
for line in open(env_file):
|
||||||
|
line = line.strip()
|
||||||
|
if line.startswith("export "):
|
||||||
|
line = line[7:]
|
||||||
|
if "=" in line and not line.startswith("#"):
|
||||||
|
k, v = line.split("=", 1)
|
||||||
|
os.environ[k] = v
|
||||||
|
|
||||||
|
yaml_file = os.path.join(script_dir, "proxmox.yaml")
|
||||||
|
cfg = {}
|
||||||
|
if os.path.exists(yaml_file):
|
||||||
|
for line in open(yaml_file):
|
||||||
|
line = line.strip()
|
||||||
|
if ": " in line and not line.startswith("#"):
|
||||||
|
k, v = line.split(": ", 1)
|
||||||
|
cfg[k] = v.strip("'\"")
|
||||||
|
|
||||||
|
host = cfg.get("host", "")
|
||||||
|
node = cfg.get("node", "pve")
|
||||||
|
token_id = cfg.get("api_token_id", "")
|
||||||
|
secret = os.environ.get("PROXMOX_TOKEN_SECRET", "")
|
||||||
|
|
||||||
|
if not host or not token_id or not secret:
|
||||||
|
print("Error: missing host/token_id/secret", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
ctx = ssl.create_default_context()
|
||||||
|
ctx.check_hostname = False
|
||||||
|
ctx.verify_mode = ssl.CERT_NONE
|
||||||
|
|
||||||
|
method = "GET"
|
||||||
|
endpoint = sys.argv[1] if len(sys.argv) > 1 else f"/nodes/{node}/qemu"
|
||||||
|
if len(sys.argv) > 2:
|
||||||
|
method = sys.argv[2]
|
||||||
|
data = sys.argv[3].encode() if len(sys.argv) > 3 else None
|
||||||
|
|
||||||
|
url = f"https://{host}:8006/api2/json{endpoint}"
|
||||||
|
req = urllib.request.Request(url, method=method, data=data)
|
||||||
|
req.add_header("Authorization", f"PVEAPIToken={token_id}={secret}")
|
||||||
|
if data:
|
||||||
|
req.add_header("Content-Type", "application/json")
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = urllib.request.urlopen(req, context=ctx)
|
||||||
|
print(resp.read().decode())
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
print(json.dumps({"error": str(e), "code": e.code}))
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,914 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# test-boot-reliability - Stress test IncusOS first-boot reliability
|
||||||
|
#
|
||||||
|
# Tests whether IncusOS nodes reliably come up after installation by deploying
|
||||||
|
# multiple single-node VMs and checking port 8443 health.
|
||||||
|
#
|
||||||
|
# Three test groups with different seed configurations:
|
||||||
|
# A) No update.yaml in seed (old behavior, state defaults only)
|
||||||
|
# B) update.yaml with check_frequency: "6h" (current default)
|
||||||
|
# C) update.yaml with check_frequency: "1h" (verifies seed takes effect)
|
||||||
|
#
|
||||||
|
# Runs batches of VMs in parallel, checks health, cleans up between batches.
|
||||||
|
# Reports per-group pass/fail rates at the end.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./test-boot-reliability # 20 deploys in batches of 4
|
||||||
|
# ./test-boot-reliability --count 12 # 12 deploys
|
||||||
|
# ./test-boot-reliability --batch-size 2 # 2 parallel per batch
|
||||||
|
# ./test-boot-reliability --dry-run # Preview without deploying
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
|
||||||
|
|
||||||
|
# Defaults
|
||||||
|
TOTAL_COUNT=20
|
||||||
|
BATCH_SIZE=4
|
||||||
|
START_VMID=850
|
||||||
|
DRY_RUN=false
|
||||||
|
HEALTH_TIMEOUT=150 # seconds to wait for port 8443 after boot
|
||||||
|
INSTALL_TIMEOUT=600 # seconds to wait for install to complete
|
||||||
|
|
||||||
|
# Proxmox connection (from proxmox.yaml)
|
||||||
|
PVE_HOST=""
|
||||||
|
PVE_NODE=""
|
||||||
|
PVE_STORAGE=""
|
||||||
|
PVE_ISO_STORAGE=""
|
||||||
|
PVE_BRIDGE=""
|
||||||
|
PVE_POOL=""
|
||||||
|
PVE_API_TOKEN_ID=""
|
||||||
|
|
||||||
|
# IncusOS ISO (must already be on Proxmox)
|
||||||
|
ISO_FILENAME=""
|
||||||
|
|
||||||
|
# Results directory (each background job writes a file)
|
||||||
|
RESULTS_DIR=""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Colors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
setup_colors() {
|
||||||
|
if [[ "${NO_COLOR:-}" == "1" ]] || [[ "${TERM:-}" == "dumb" ]]; then
|
||||||
|
BOLD="" RESET="" RED="" GREEN="" YELLOW="" CYAN="" DIM=""
|
||||||
|
else
|
||||||
|
BOLD="\033[1m" RESET="\033[0m" RED="\033[31m" GREEN="\033[32m"
|
||||||
|
YELLOW="\033[33m" CYAN="\033[36m" DIM="\033[2m"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
setup_colors
|
||||||
|
|
||||||
|
step() { echo -e "${BOLD}[step]${RESET} $*"; }
|
||||||
|
success() { echo -e "${GREEN}[ok]${RESET} $*"; }
|
||||||
|
warn() { echo -e "${YELLOW}[warn]${RESET} $*"; }
|
||||||
|
error() { echo -e "${RED}[error]${RESET} $*"; }
|
||||||
|
info() { echo -e "${DIM}[info]${RESET} $*"; }
|
||||||
|
detail() { echo -e "${DIM} $*${RESET}"; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Argument parsing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--count) TOTAL_COUNT="${2:?'--count requires a number'}"; shift 2 ;;
|
||||||
|
--batch-size) BATCH_SIZE="${2:?'--batch-size requires a number'}"; shift 2 ;;
|
||||||
|
--start-vmid) START_VMID="${2:?'--start-vmid requires a number'}"; shift 2 ;;
|
||||||
|
--dry-run) DRY_RUN=true; shift ;;
|
||||||
|
-h|--help)
|
||||||
|
echo "Usage: test-boot-reliability [OPTIONS]"
|
||||||
|
echo ""
|
||||||
|
echo "Options:"
|
||||||
|
echo " --count N Total deploys (default: 20)"
|
||||||
|
echo " --batch-size N Parallel VMs per batch (default: 4)"
|
||||||
|
echo " --start-vmid N First VMID to use (default: 850)"
|
||||||
|
echo " --dry-run Preview without deploying"
|
||||||
|
echo " -h, --help Show this help"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*) error "Unknown option: $1"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load environment
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Auto-load env file for PROXMOX_TOKEN_SECRET
|
||||||
|
load_env() {
|
||||||
|
local dir="$SCRIPT_DIR"
|
||||||
|
while [[ "$dir" != "/" ]]; do
|
||||||
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "${dir}/env"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
dir="$(dirname "$dir")"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
load_env
|
||||||
|
|
||||||
|
if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then
|
||||||
|
error "PROXMOX_TOKEN_SECRET not set. Create an env file or export it."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Load proxmox.yaml (simple key: value format, no nested structures)
|
||||||
|
yaml_get() {
|
||||||
|
local file="$1" key="$2" default="${3:-}"
|
||||||
|
local val
|
||||||
|
val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val=""
|
||||||
|
# Strip surrounding quotes
|
||||||
|
val="${val#\"}" ; val="${val%\"}"
|
||||||
|
val="${val#\'}" ; val="${val%\'}"
|
||||||
|
echo "${val:-$default}"
|
||||||
|
}
|
||||||
|
|
||||||
|
load_proxmox_yaml() {
|
||||||
|
local yaml_file=""
|
||||||
|
if [[ -f "${SCRIPT_DIR}/proxmox.yaml" ]]; then
|
||||||
|
yaml_file="${SCRIPT_DIR}/proxmox.yaml"
|
||||||
|
elif [[ -f "proxmox.yaml" ]]; then
|
||||||
|
yaml_file="proxmox.yaml"
|
||||||
|
else
|
||||||
|
error "proxmox.yaml not found"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
PVE_HOST=$(yaml_get "$yaml_file" "host")
|
||||||
|
PVE_NODE=$(yaml_get "$yaml_file" "node" "pve")
|
||||||
|
PVE_STORAGE=$(yaml_get "$yaml_file" "storage" "local-lvm")
|
||||||
|
PVE_ISO_STORAGE=$(yaml_get "$yaml_file" "iso_storage" "local")
|
||||||
|
PVE_BRIDGE=$(yaml_get "$yaml_file" "bridge" "vmbr0")
|
||||||
|
PVE_POOL=$(yaml_get "$yaml_file" "pool")
|
||||||
|
PVE_API_TOKEN_ID=$(yaml_get "$yaml_file" "api_token_id")
|
||||||
|
|
||||||
|
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
|
||||||
|
detail "Pool: ${PVE_POOL}, Token: ${PVE_API_TOKEN_ID}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Proxmox API helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
pve_api() {
|
||||||
|
local method="$1"
|
||||||
|
local endpoint="$2"
|
||||||
|
local data="${3:-}"
|
||||||
|
local content_type="${4:-form}"
|
||||||
|
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json${endpoint}"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
|
||||||
|
local curl_args=(-fsSk -H "$auth")
|
||||||
|
if [[ "$content_type" == "json" ]]; then
|
||||||
|
curl_args+=(-H "Content-Type: application/json")
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$method" in
|
||||||
|
GET) curl "${curl_args[@]}" "$url" ;;
|
||||||
|
POST) curl "${curl_args[@]}" -X POST ${data:+-d "$data"} "$url" ;;
|
||||||
|
PUT) curl "${curl_args[@]}" -X PUT ${data:+-d "$data"} "$url" ;;
|
||||||
|
DELETE) curl "${curl_args[@]}" -X DELETE "$url" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_exists() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_status() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" 2>/dev/null | \
|
||||||
|
python3 -c "import json,sys; print(json.load(sys.stdin)['data']['status'])" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_start_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_stop_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_destroy_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_disk_writes() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" 2>/dev/null | \
|
||||||
|
python3 -c "
|
||||||
|
import json,sys
|
||||||
|
d = json.load(sys.stdin).get('data',{})
|
||||||
|
bs = d.get('blockstat',{}).get('scsi0',{})
|
||||||
|
print(bs.get('wr_bytes',0))
|
||||||
|
" 2>/dev/null || echo "0"
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_mac() {
|
||||||
|
local vmid="$1"
|
||||||
|
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null | \
|
||||||
|
python3 -c "
|
||||||
|
import json, sys, re
|
||||||
|
data = json.load(sys.stdin).get('data', {})
|
||||||
|
net0 = data.get('net0', '')
|
||||||
|
m = re.search(r'([0-9A-Fa-f:]{17})', net0)
|
||||||
|
print(m.group(1).lower() if m else '')
|
||||||
|
" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_vm_ip() {
|
||||||
|
# ARP-based IP detection matching incusos-proxmox pve_vm_ip().
|
||||||
|
# Strategy: flush stale ARP → broadcast ping → check ARP → subnet sweep fallback.
|
||||||
|
local vmid="$1"
|
||||||
|
local mac
|
||||||
|
mac=$(pve_vm_mac "$vmid") || return 1
|
||||||
|
[[ -z "$mac" ]] && return 1
|
||||||
|
|
||||||
|
ip neigh flush nud stale 2>/dev/null || true
|
||||||
|
|
||||||
|
local bridge_subnet
|
||||||
|
bridge_subnet=$(ip -4 addr show | awk '/inet / && !/127\.0\.0/ {print $2; exit}')
|
||||||
|
|
||||||
|
# Try broadcast ping first (fast, but many hosts ignore it)
|
||||||
|
if [[ -n "$bridge_subnet" ]]; then
|
||||||
|
local broadcast
|
||||||
|
broadcast=$(python3 -c "import ipaddress; print(ipaddress.ip_interface('${bridge_subnet}').network.broadcast_address)" 2>/dev/null) || true
|
||||||
|
if [[ -n "$broadcast" ]]; then
|
||||||
|
ping -c 2 -W 1 -b "$broadcast" &>/dev/null || true
|
||||||
|
sleep 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check ARP table (filter by state)
|
||||||
|
local ip
|
||||||
|
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/ || /STALE/) {print $1; exit}') || true
|
||||||
|
|
||||||
|
if [[ -n "$ip" ]] && ping -c 1 -W 1 "$ip" &>/dev/null; then
|
||||||
|
echo "$ip"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Broadcast didn't work — parallel ping sweep of the subnet
|
||||||
|
if [[ -n "$bridge_subnet" ]]; then
|
||||||
|
local hosts
|
||||||
|
hosts=$(python3 -c "
|
||||||
|
import ipaddress
|
||||||
|
net = ipaddress.ip_interface('${bridge_subnet}').network
|
||||||
|
for h in net.hosts():
|
||||||
|
print(h)
|
||||||
|
" 2>/dev/null)
|
||||||
|
if [[ -n "$hosts" ]]; then
|
||||||
|
while IFS= read -r host_ip; do
|
||||||
|
ping -c 1 -W 1 "$host_ip" &>/dev/null &
|
||||||
|
done <<< "$hosts"
|
||||||
|
wait
|
||||||
|
sleep 1
|
||||||
|
|
||||||
|
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/) {print $1; exit}') || true
|
||||||
|
|
||||||
|
if [[ -n "$ip" ]] && ping -c 1 -W 1 "$ip" &>/dev/null; then
|
||||||
|
echo "$ip"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Find IncusOS ISO already on Proxmox
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
find_incusos_iso() {
|
||||||
|
step "Finding IncusOS ISO on Proxmox"
|
||||||
|
local iso_list
|
||||||
|
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null) || iso_list=""
|
||||||
|
|
||||||
|
ISO_FILENAME=$(echo "$iso_list" | python3 -c "
|
||||||
|
import json, sys
|
||||||
|
data = json.load(sys.stdin).get('data', [])
|
||||||
|
isos = [d['volid'].split('/')[-1] for d in data if 'IncusOS_' in d.get('volid', '')]
|
||||||
|
if isos:
|
||||||
|
isos.sort()
|
||||||
|
print(isos[-1])
|
||||||
|
" 2>/dev/null) || ISO_FILENAME=""
|
||||||
|
|
||||||
|
if [[ -z "$ISO_FILENAME" ]]; then
|
||||||
|
error "No IncusOS ISO found on Proxmox storage '${PVE_ISO_STORAGE}'"
|
||||||
|
error "Deploy a lab first to upload the ISO, or use incusos-proxmox --phase upload"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Using ISO: ${ISO_FILENAME}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Seed generation per group
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
generate_seed() {
|
||||||
|
local group="$1" # A, B, or C
|
||||||
|
local name="$2"
|
||||||
|
local output="$3"
|
||||||
|
|
||||||
|
case "$group" in
|
||||||
|
A)
|
||||||
|
# No update.yaml -- uses incusos-seed to generate correct format,
|
||||||
|
# then removes update.yaml from the seed directory before ISO creation.
|
||||||
|
# This tests whether omitting update.yaml causes boot failures.
|
||||||
|
local tmpdir
|
||||||
|
tmpdir=$(mktemp -d)
|
||||||
|
|
||||||
|
# Generate full seed via incusos-seed to a tar, extract, remove update.yaml
|
||||||
|
"$SEED_TOOL" --format tar --app incus --output "${tmpdir}/seed.tar" \
|
||||||
|
--force-install --force-reboot --hostname "$name" \
|
||||||
|
--defaults --quiet &>/dev/null
|
||||||
|
|
||||||
|
local seeddir="${tmpdir}/seedfiles"
|
||||||
|
mkdir -p "$seeddir"
|
||||||
|
tar xf "${tmpdir}/seed.tar" -C "$seeddir"
|
||||||
|
rm -f "${seeddir}/update.yaml"
|
||||||
|
|
||||||
|
# Generate ISO from the modified seed directory
|
||||||
|
if command -v genisoimage &>/dev/null; then
|
||||||
|
genisoimage -V SEED_DATA -J -r -o "$output" "$seeddir" 2>/dev/null
|
||||||
|
elif command -v mkisofs &>/dev/null; then
|
||||||
|
mkisofs -V SEED_DATA -J -r -o "$output" "$seeddir" 2>/dev/null
|
||||||
|
fi
|
||||||
|
rm -rf "$tmpdir"
|
||||||
|
;;
|
||||||
|
B)
|
||||||
|
# update.yaml with check_frequency: "6h" (default/current behavior)
|
||||||
|
"$SEED_TOOL" --format iso --app incus --output "$output" \
|
||||||
|
--force-install --force-reboot --hostname "$name" \
|
||||||
|
--defaults --quiet &>/dev/null
|
||||||
|
;;
|
||||||
|
C)
|
||||||
|
# update.yaml with check_frequency: "1h" (verify seed takes effect)
|
||||||
|
# Uses incusos-seed to generate correct format, then patches
|
||||||
|
# update.yaml to use a different check_frequency.
|
||||||
|
local tmpdir
|
||||||
|
tmpdir=$(mktemp -d)
|
||||||
|
|
||||||
|
# Generate full seed via incusos-seed
|
||||||
|
"$SEED_TOOL" --format tar --app incus --output "${tmpdir}/seed.tar" \
|
||||||
|
--force-install --force-reboot --hostname "$name" \
|
||||||
|
--defaults --quiet &>/dev/null
|
||||||
|
|
||||||
|
local seeddir="${tmpdir}/seedfiles"
|
||||||
|
mkdir -p "$seeddir"
|
||||||
|
tar xf "${tmpdir}/seed.tar" -C "$seeddir"
|
||||||
|
|
||||||
|
# Patch check_frequency to 1h
|
||||||
|
cat > "${seeddir}/update.yaml" <<'YAML'
|
||||||
|
# Update configuration
|
||||||
|
# Patched: check_frequency set to 1h for test group C
|
||||||
|
channel: "stable"
|
||||||
|
auto_reboot: false
|
||||||
|
check_frequency: "1h"
|
||||||
|
YAML
|
||||||
|
|
||||||
|
# Generate ISO from the modified seed directory
|
||||||
|
if command -v genisoimage &>/dev/null; then
|
||||||
|
genisoimage -V SEED_DATA -J -r -o "$output" "$seeddir" 2>/dev/null
|
||||||
|
elif command -v mkisofs &>/dev/null; then
|
||||||
|
mkisofs -V SEED_DATA -J -r -o "$output" "$seeddir" 2>/dev/null
|
||||||
|
fi
|
||||||
|
rm -rf "$tmpdir"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Upload seed ISO to Proxmox
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
upload_seed_iso() {
|
||||||
|
local local_path="$1"
|
||||||
|
local filename="$2"
|
||||||
|
|
||||||
|
pve_api POST "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" \
|
||||||
|
"" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
# Upload via curl multipart
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
|
||||||
|
curl -fsSk -H "$auth" \
|
||||||
|
-F "content=iso" \
|
||||||
|
-F "filename=@${local_path};filename=${filename}" \
|
||||||
|
"$url" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Create VM (minimal IncusOS settings)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
create_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
local name="$2"
|
||||||
|
local seed_iso="$3"
|
||||||
|
|
||||||
|
# Proxmox API requires JSON for VM creation -- comma-separated disk/net
|
||||||
|
# specs are misinterpreted when sent as form-encoded data.
|
||||||
|
local json_payload
|
||||||
|
json_payload=$(python3 -c "
|
||||||
|
import json
|
||||||
|
payload = {
|
||||||
|
'vmid': ${vmid},
|
||||||
|
'name': '${name}',
|
||||||
|
'description': '[incusos-lab:managed] test-boot-reliability',
|
||||||
|
'ostype': 'l26',
|
||||||
|
'bios': 'ovmf',
|
||||||
|
'machine': 'q35',
|
||||||
|
'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0',
|
||||||
|
'tpmstate0': '${PVE_STORAGE}:1,version=v2.0',
|
||||||
|
'cpu': 'host',
|
||||||
|
'cores': 4,
|
||||||
|
'memory': 4096,
|
||||||
|
'balloon': 0,
|
||||||
|
'scsihw': 'virtio-scsi-pci',
|
||||||
|
'scsi0': '${PVE_STORAGE}:50,iothread=1',
|
||||||
|
'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom',
|
||||||
|
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_iso},media=cdrom',
|
||||||
|
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
||||||
|
'boot': 'order=ide2;scsi0',
|
||||||
|
'agent': 1,
|
||||||
|
}
|
||||||
|
pool = '${PVE_POOL}'
|
||||||
|
if pool:
|
||||||
|
payload['pool'] = pool
|
||||||
|
print(json.dumps(payload))
|
||||||
|
")
|
||||||
|
|
||||||
|
local url="https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/qemu"
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
|
||||||
|
local create_response
|
||||||
|
create_response=$(curl -fsSk -H "$auth" -H "Content-Type: application/json" \
|
||||||
|
-X POST -d "$json_payload" "$url" 2>/dev/null) || return 1
|
||||||
|
|
||||||
|
# VM creation is async -- wait for the UPID task to complete
|
||||||
|
local upid
|
||||||
|
upid=$(echo "$create_response" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid=""
|
||||||
|
if [[ -n "$upid" ]]; then
|
||||||
|
local encoded_upid
|
||||||
|
encoded_upid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null)
|
||||||
|
local wait_elapsed=0
|
||||||
|
while [[ $wait_elapsed -lt 60 ]]; do
|
||||||
|
sleep 2
|
||||||
|
wait_elapsed=$((wait_elapsed + 2))
|
||||||
|
local task_status
|
||||||
|
task_status=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${encoded_upid}/status" 2>/dev/null | \
|
||||||
|
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || task_status=""
|
||||||
|
if [[ "$task_status" == "stopped" ]]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Single VM deploy + health check
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
write_result() {
|
||||||
|
local vmid="$1"
|
||||||
|
local group="$2"
|
||||||
|
local outcome="$3"
|
||||||
|
local detail="$4"
|
||||||
|
echo "${outcome}|${detail}" > "${RESULTS_DIR}/${vmid}-${group}.result"
|
||||||
|
}
|
||||||
|
|
||||||
|
deploy_and_check() {
|
||||||
|
local vmid="$1"
|
||||||
|
local name="$2"
|
||||||
|
local group="$3"
|
||||||
|
local seed_iso="$4"
|
||||||
|
|
||||||
|
info "[$name] Group ${group}: creating VM (VMID ${vmid})"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "[$name] [dry-run] Would create, install, check health"
|
||||||
|
write_result "$vmid" "$group" "dry-run" "skipped"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create VM
|
||||||
|
if ! create_vm "$vmid" "$name" "$seed_iso"; then
|
||||||
|
error "[$name] Failed to create VM"
|
||||||
|
write_result "$vmid" "$group" "ERROR" "create failed"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Start VM
|
||||||
|
if ! pve_start_vm "$vmid"; then
|
||||||
|
error "[$name] Failed to start VM"
|
||||||
|
write_result "$vmid" "$group" "ERROR" "start failed"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Monitor installation (blockstat polling)
|
||||||
|
local elapsed=0
|
||||||
|
local prev_wr=0
|
||||||
|
local writes_started=false
|
||||||
|
local stable_count=0
|
||||||
|
local install_done=false
|
||||||
|
|
||||||
|
while [[ $elapsed -lt $INSTALL_TIMEOUT ]]; do
|
||||||
|
sleep 5
|
||||||
|
elapsed=$((elapsed + 5))
|
||||||
|
|
||||||
|
local status
|
||||||
|
status=$(pve_vm_status "$vmid" 2>/dev/null) || status="unknown"
|
||||||
|
|
||||||
|
if [[ "$status" == "stopped" ]]; then
|
||||||
|
install_done=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$status" == "running" ]]; then
|
||||||
|
local wr
|
||||||
|
wr=$(pve_vm_disk_writes "$vmid" 2>/dev/null) || wr=0
|
||||||
|
if [[ $wr -gt $prev_wr ]]; then
|
||||||
|
writes_started=true
|
||||||
|
stable_count=0
|
||||||
|
elif [[ "$writes_started" == true ]]; then
|
||||||
|
stable_count=$((stable_count + 1))
|
||||||
|
if [[ $stable_count -ge 3 ]]; then
|
||||||
|
install_done=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
prev_wr=$wr
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ "$install_done" != true ]]; then
|
||||||
|
warn "[$name] Install timeout after ${INSTALL_TIMEOUT}s"
|
||||||
|
write_result "$vmid" "$group" "FAIL" "install timeout"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Stop VM if still running
|
||||||
|
local cur_status
|
||||||
|
cur_status=$(pve_vm_status "$vmid" 2>/dev/null) || cur_status="unknown"
|
||||||
|
if [[ "$cur_status" == "running" ]]; then
|
||||||
|
pve_stop_vm "$vmid" || true
|
||||||
|
sleep 5
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Remove install media, set boot order (must use JSON to avoid
|
||||||
|
# form-encoding ambiguity with commas and equals signs)
|
||||||
|
pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" \
|
||||||
|
'{"delete":"ide2,ide3","boot":"order=scsi0"}' json &>/dev/null || true
|
||||||
|
|
||||||
|
# Start from disk
|
||||||
|
if ! pve_start_vm "$vmid"; then
|
||||||
|
error "[$name] Failed to start from disk"
|
||||||
|
write_result "$vmid" "$group" "FAIL" "disk boot failed"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Health check: wait for port 8443
|
||||||
|
info "[$name] Waiting for port 8443..."
|
||||||
|
local health_elapsed=0
|
||||||
|
local healthy=false
|
||||||
|
|
||||||
|
# Wait 45s for VM to boot and get DHCP lease.
|
||||||
|
# Stagger by vmid mod to reduce ARP sweep contention across parallel jobs.
|
||||||
|
local stagger=$(( (vmid % 4) * 5 ))
|
||||||
|
sleep $((45 + stagger))
|
||||||
|
health_elapsed=$((45 + stagger))
|
||||||
|
|
||||||
|
# Detect IP with retries
|
||||||
|
local ip=""
|
||||||
|
local ip_retry=0
|
||||||
|
while [[ $ip_retry -lt 6 ]] && [[ -z "$ip" ]]; do
|
||||||
|
ip=$(pve_vm_ip "$vmid" 2>/dev/null) || ip=""
|
||||||
|
if [[ -n "$ip" ]]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 10
|
||||||
|
health_elapsed=$((health_elapsed + 10))
|
||||||
|
ip_retry=$((ip_retry + 1))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -z "$ip" ]]; then
|
||||||
|
warn "[$name] Could not detect IP"
|
||||||
|
write_result "$vmid" "$group" "FAIL" "no IP detected"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Poll port 8443
|
||||||
|
while [[ $health_elapsed -lt $HEALTH_TIMEOUT ]]; do
|
||||||
|
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||||
|
healthy=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
health_elapsed=$((health_elapsed + 5))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ "$healthy" == true ]]; then
|
||||||
|
success "[$name] Group ${group}: PASS — port 8443 up at ${ip} (${health_elapsed}s)"
|
||||||
|
local result_detail="healthy at ${ip} in ${health_elapsed}s"
|
||||||
|
|
||||||
|
# For group C, verify check_frequency via IncusOS API
|
||||||
|
if [[ "$group" == "C" ]]; then
|
||||||
|
local freq="?"
|
||||||
|
local cert_file="${HOME}/.config/incus/client.crt"
|
||||||
|
local key_file="${HOME}/.config/incus/client.key"
|
||||||
|
if [[ -f "$cert_file" ]] && [[ -f "$key_file" ]]; then
|
||||||
|
freq=$(curl -sk --cert "$cert_file" --key "$key_file" \
|
||||||
|
"https://${ip}:8443/1.0/system/update" 2>/dev/null | \
|
||||||
|
python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('metadata',d).get('check_frequency','?'))" 2>/dev/null) || freq="?"
|
||||||
|
fi
|
||||||
|
detail "[$name] check_frequency from API: ${freq}"
|
||||||
|
result_detail="healthy at ${ip}, check_frequency=${freq}"
|
||||||
|
fi
|
||||||
|
write_result "$vmid" "$group" "PASS" "$result_detail"
|
||||||
|
else
|
||||||
|
warn "[$name] Group ${group}: FAIL — port 8443 not reachable at ${ip} after ${HEALTH_TIMEOUT}s"
|
||||||
|
write_result "$vmid" "$group" "FAIL" "port 8443 unreachable at ${ip}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cleanup a VM
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
cleanup_vm() {
|
||||||
|
local vmid="$1"
|
||||||
|
local name="$2"
|
||||||
|
local seed_iso="$3"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Stop if running
|
||||||
|
local status
|
||||||
|
status=$(pve_vm_status "$vmid" 2>/dev/null) || status="unknown"
|
||||||
|
if [[ "$status" == "running" ]]; then
|
||||||
|
pve_stop_vm "$vmid" || true
|
||||||
|
sleep 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Destroy
|
||||||
|
pve_destroy_vm "$vmid" || true
|
||||||
|
|
||||||
|
# Remove seed ISO from storage
|
||||||
|
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_iso}" &>/dev/null || true
|
||||||
|
|
||||||
|
# Remove incus remote
|
||||||
|
if command -v incus &>/dev/null; then
|
||||||
|
local current_remote
|
||||||
|
current_remote=$(incus remote get-default 2>/dev/null) || true
|
||||||
|
if [[ "$current_remote" == "$name" ]]; then
|
||||||
|
incus remote switch local 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
incus remote remove "$name" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
echo -e "${BOLD}IncusOS Boot Reliability Test${RESET}"
|
||||||
|
echo ""
|
||||||
|
echo "Test design:"
|
||||||
|
echo " Group A: no update.yaml in seed (state defaults only)"
|
||||||
|
echo " Group B: update.yaml with check_frequency: \"6h\" (current default)"
|
||||||
|
echo " Group C: update.yaml with check_frequency: \"1h\" (verify seed effect)"
|
||||||
|
echo ""
|
||||||
|
echo "Deploys: ${TOTAL_COUNT} total, ${BATCH_SIZE} parallel per batch"
|
||||||
|
echo "VMIDs: ${START_VMID}-$((START_VMID + TOTAL_COUNT - 1))"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
step "Loading configuration"
|
||||||
|
load_proxmox_yaml
|
||||||
|
find_incusos_iso
|
||||||
|
|
||||||
|
# Assign groups: distribute evenly across A, B, C
|
||||||
|
# Round-robin: 0→A, 1→B, 2→C, 3→A, 4→B, ...
|
||||||
|
declare -a VM_GROUPS=()
|
||||||
|
declare -a VM_NAMES=()
|
||||||
|
declare -a VM_VMIDS=()
|
||||||
|
declare -a VM_SEEDS=()
|
||||||
|
|
||||||
|
groups=("A" "B" "C")
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
group="${groups[$((idx % 3))]}"
|
||||||
|
vmid=$((START_VMID + idx))
|
||||||
|
name="boot-test-${group,,}-${vmid}"
|
||||||
|
seed_name="seed-${name}.iso"
|
||||||
|
|
||||||
|
VM_GROUPS+=("$group")
|
||||||
|
VM_NAMES+=("$name")
|
||||||
|
VM_VMIDS+=("$vmid")
|
||||||
|
VM_SEEDS+=("$seed_name")
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}Test matrix:${RESET}"
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
echo " VMID ${VM_VMIDS[$idx]}: ${VM_NAMES[$idx]} (Group ${VM_GROUPS[$idx]})"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check for VMID collisions
|
||||||
|
step "Checking for VMID collisions"
|
||||||
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
if pve_vm_exists "${VM_VMIDS[$idx]}"; then
|
||||||
|
error "VMID ${VM_VMIDS[$idx]} already exists. Choose a different --start-vmid range."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
success "No collisions"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Generate and upload seed ISOs
|
||||||
|
step "Generating seed ISOs"
|
||||||
|
SEED_TMPDIR=$(mktemp -d)
|
||||||
|
RESULTS_DIR=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$SEED_TMPDIR" "$RESULTS_DIR"' EXIT
|
||||||
|
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
local_seed="${SEED_TMPDIR}/${VM_SEEDS[$idx]}"
|
||||||
|
generate_seed "${VM_GROUPS[$idx]}" "${VM_NAMES[$idx]}" "$local_seed"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "[dry-run] Would generate seed: ${VM_SEEDS[$idx]} (Group ${VM_GROUPS[$idx]})"
|
||||||
|
else
|
||||||
|
upload_seed_iso "$local_seed" "${VM_SEEDS[$idx]}"
|
||||||
|
detail "Uploaded: ${VM_SEEDS[$idx]} (Group ${VM_GROUPS[$idx]})"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
success "All seed ISOs generated and uploaded"
|
||||||
|
|
||||||
|
# Run batches
|
||||||
|
echo ""
|
||||||
|
total_batches=$(( (TOTAL_COUNT + BATCH_SIZE - 1) / BATCH_SIZE ))
|
||||||
|
batch_num=0
|
||||||
|
|
||||||
|
for ((batch_start=0; batch_start<TOTAL_COUNT; batch_start+=BATCH_SIZE)); do
|
||||||
|
batch_num=$((batch_num + 1))
|
||||||
|
batch_end=$((batch_start + BATCH_SIZE))
|
||||||
|
[[ $batch_end -gt $TOTAL_COUNT ]] && batch_end=$TOTAL_COUNT
|
||||||
|
batch_count=$((batch_end - batch_start))
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
step "Batch ${batch_num}/${total_batches}: deploying ${batch_count} VMs (VMIDs $((START_VMID + batch_start))-$((START_VMID + batch_end - 1)))"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Deploy all VMs in this batch in parallel
|
||||||
|
pids=()
|
||||||
|
for ((idx=batch_start; idx<batch_end; idx++)); do
|
||||||
|
deploy_and_check "${VM_VMIDS[$idx]}" "${VM_NAMES[$idx]}" \
|
||||||
|
"${VM_GROUPS[$idx]}" "${VM_SEEDS[$idx]}" &
|
||||||
|
pids+=($!)
|
||||||
|
done
|
||||||
|
|
||||||
|
# Wait for all in this batch to complete
|
||||||
|
for pid in "${pids[@]}"; do
|
||||||
|
wait "$pid" || true
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
step "Batch ${batch_num} complete — cleaning up"
|
||||||
|
|
||||||
|
# Cleanup all VMs in this batch
|
||||||
|
for ((idx=batch_start; idx<batch_end; idx++)); do
|
||||||
|
cleanup_vm "${VM_VMIDS[$idx]}" "${VM_NAMES[$idx]}" "${VM_SEEDS[$idx]}"
|
||||||
|
detail "Cleaned: ${VM_NAMES[$idx]}"
|
||||||
|
done
|
||||||
|
success "Batch ${batch_num} cleaned"
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Results summary (read from result files)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}========================================${RESET}"
|
||||||
|
echo -e "${BOLD} Test Results Summary${RESET}"
|
||||||
|
echo -e "${BOLD}========================================${RESET}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Per-group stats
|
||||||
|
for group in A B C; do
|
||||||
|
total=0
|
||||||
|
pass=0
|
||||||
|
fail=0
|
||||||
|
err=0
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
if [[ "${VM_GROUPS[$idx]}" == "$group" ]]; then
|
||||||
|
total=$((total + 1))
|
||||||
|
rfile="${RESULTS_DIR}/${VM_VMIDS[$idx]}-${group}.result"
|
||||||
|
if [[ -f "$rfile" ]]; then
|
||||||
|
outcome=$(cut -d'|' -f1 "$rfile")
|
||||||
|
case "$outcome" in
|
||||||
|
PASS) pass=$((pass + 1)) ;;
|
||||||
|
FAIL) fail=$((fail + 1)) ;;
|
||||||
|
ERROR) err=$((err + 1)) ;;
|
||||||
|
esac
|
||||||
|
else
|
||||||
|
err=$((err + 1)) # no result file = something went wrong
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
label=""
|
||||||
|
case "$group" in
|
||||||
|
A) label="No update.yaml (state defaults)" ;;
|
||||||
|
B) label="update.yaml check_frequency=6h" ;;
|
||||||
|
C) label="update.yaml check_frequency=1h" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [[ $fail -eq 0 ]] && [[ $err -eq 0 ]]; then
|
||||||
|
echo -e " Group ${group} (${label}):"
|
||||||
|
echo -e " ${GREEN}${pass}/${total} PASS${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " Group ${group} (${label}):"
|
||||||
|
echo -e " ${GREEN}${pass} PASS${RESET}, ${RED}${fail} FAIL${RESET}, ${RED}${err} ERROR${RESET} (of ${total})"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}Per-VM details:${RESET}"
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
rfile="${RESULTS_DIR}/${VM_VMIDS[$idx]}-${VM_GROUPS[$idx]}.result"
|
||||||
|
outcome="MISSING"
|
||||||
|
rdetail="no result file"
|
||||||
|
if [[ -f "$rfile" ]]; then
|
||||||
|
outcome=$(cut -d'|' -f1 "$rfile")
|
||||||
|
rdetail=$(cut -d'|' -f2- "$rfile")
|
||||||
|
fi
|
||||||
|
|
||||||
|
marker=""
|
||||||
|
case "$outcome" in
|
||||||
|
PASS) marker="${GREEN}PASS${RESET}" ;;
|
||||||
|
FAIL) marker="${RED}FAIL${RESET}" ;;
|
||||||
|
ERROR) marker="${RED}ERROR${RESET}" ;;
|
||||||
|
*) marker="${DIM}${outcome}${RESET}" ;;
|
||||||
|
esac
|
||||||
|
echo -e " ${VM_VMIDS[$idx]} ${VM_NAMES[$idx]} Group ${VM_GROUPS[$idx]}: ${marker} -- ${rdetail}"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Overall assessment
|
||||||
|
total_fail=0
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
rfile="${RESULTS_DIR}/${VM_VMIDS[$idx]}-${VM_GROUPS[$idx]}.result"
|
||||||
|
if [[ -f "$rfile" ]]; then
|
||||||
|
outcome=$(cut -d'|' -f1 "$rfile")
|
||||||
|
if [[ "$outcome" == "FAIL" ]] || [[ "$outcome" == "ERROR" ]]; then
|
||||||
|
total_fail=$((total_fail + 1))
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
total_fail=$((total_fail + 1))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ $total_fail -eq 0 ]]; then
|
||||||
|
success "All ${TOTAL_COUNT} deploys passed — no boot failures detected"
|
||||||
|
else
|
||||||
|
warn "${total_fail}/${TOTAL_COUNT} deploys failed"
|
||||||
|
echo ""
|
||||||
|
echo "Analysis:"
|
||||||
|
for group in A B C; do
|
||||||
|
g_fail=0
|
||||||
|
g_total=0
|
||||||
|
for ((idx=0; idx<TOTAL_COUNT; idx++)); do
|
||||||
|
if [[ "${VM_GROUPS[$idx]}" == "$group" ]]; then
|
||||||
|
g_total=$((g_total + 1))
|
||||||
|
rfile="${RESULTS_DIR}/${VM_VMIDS[$idx]}-${group}.result"
|
||||||
|
outcome=""
|
||||||
|
[[ -f "$rfile" ]] && outcome=$(cut -d'|' -f1 "$rfile")
|
||||||
|
if [[ "$outcome" == "FAIL" ]] || [[ "$outcome" == "ERROR" ]] || [[ -z "$outcome" ]]; then
|
||||||
|
g_fail=$((g_fail + 1))
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo " Group ${group}: ${g_fail}/${g_total} failures"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done."
|
||||||
|
|
@ -0,0 +1,172 @@
|
||||||
|
# IncusOS Boot Failure Investigation Report
|
||||||
|
|
||||||
|
Date: 2026-02-21
|
||||||
|
ISO tested: IncusOS_202602210344.iso (latest CDN)
|
||||||
|
Filed as: IncusOS issue #843
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
The "ERROR invalid crontab expression" boot failure occurs on **67% of first
|
||||||
|
boots** (8/12 API-verified deploys). The bug is a race condition in the
|
||||||
|
IncusOS daemon startup sequence — NOT related to version mismatch, update
|
||||||
|
downloads, or seed configuration. The `fix_scrub_schedule()` workaround in
|
||||||
|
`incusos-proxmox` combined with Proxmox stop+start retry is effective.
|
||||||
|
|
||||||
|
## Test Methodology
|
||||||
|
|
||||||
|
- **Tool**: `investigate-parallel` — deploys N VMs simultaneously on Proxmox,
|
||||||
|
monitors install via blockstat, transitions to disk boot, detects IP via
|
||||||
|
ARP sweep, probes IncusOS REST API, captures console screenshots.
|
||||||
|
- **Environment**: Proxmox VE 9.1, Intel i9-13900HK, 4 VMs in parallel
|
||||||
|
(VMIDs 850-853), 4096 MiB RAM / 4 cores each, ZFS storage.
|
||||||
|
- **ISO**: IncusOS_202602210344.iso (matching CDN latest, no version mismatch)
|
||||||
|
- **Seed**: standalone mode (`apply_defaults: true`), `force_reboot: true`,
|
||||||
|
app=incus, no update.yaml.
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
### Batches with reliable IP detection (API-verified)
|
||||||
|
|
||||||
|
| Batch | PASS | CRONTAB_BUG | Bug Rate | Notes |
|
||||||
|
|-------|------|-------------|----------|-------|
|
||||||
|
| v3 | 1 | 3 | 75% | First batch with working IP detection |
|
||||||
|
| v4 | 1 | 3 | 75% | Consistent with v3 |
|
||||||
|
| v5 | 2 | 2 | 50% | Slightly better |
|
||||||
|
| **Total** | **4** | **8** | **67%** | **12 VMs total** |
|
||||||
|
|
||||||
|
### Earlier batches (screenshot-verified, IP detection unreliable)
|
||||||
|
|
||||||
|
| Batch | PASS | CRONTAB_BUG | Notes |
|
||||||
|
|-------|------|-------------|-------|
|
||||||
|
| v1 | 1* | 3 | *VM 850 showed "System is ready" on console |
|
||||||
|
| v2 | 4* | 0 | *All showed "System is ready" on console |
|
||||||
|
|
||||||
|
*Batch v1-v2 had broken ARP-based IP detection (broadcast ping disabled).
|
||||||
|
Console screenshots confirm actual state.
|
||||||
|
|
||||||
|
### Combined (20 VMs)
|
||||||
|
|
||||||
|
- **Genuine PASS**: 9 (45%)
|
||||||
|
- **CRONTAB_BUG**: 11 (55%)
|
||||||
|
- **Overall bug rate: 55-67%** depending on which batches are included
|
||||||
|
|
||||||
|
## Key Findings
|
||||||
|
|
||||||
|
### 1. Version mismatch is NOT the cause
|
||||||
|
|
||||||
|
Stgraber's hypothesis that the bug relates to ISO version != CDN latest was
|
||||||
|
tested and **ruled out**. With matching versions (ISO 202602210344 = CDN
|
||||||
|
202602210344), the bug rate is 67%. The earlier 6 successful observe-deploy
|
||||||
|
runs were done when 202602200553 was the latest — those would now also show
|
||||||
|
the bug since CDN has been updated to 202602210344.
|
||||||
|
|
||||||
|
### 2. Sysext downloads happen on EVERY first boot
|
||||||
|
|
||||||
|
Even with matching ISO version, the first boot downloads:
|
||||||
|
1. SecureBoot update (~1 second)
|
||||||
|
2. Application sysext (Incus, ~2 minutes from CDN)
|
||||||
|
|
||||||
|
These are NOT in the ISO. The sysext is always fetched from the update
|
||||||
|
provider. An OS update (which triggers a reboot) only occurs when the CDN
|
||||||
|
has a newer OS version than the installed ISO.
|
||||||
|
|
||||||
|
### 3. The error window allows false-positive detection
|
||||||
|
|
||||||
|
When the bug hits:
|
||||||
|
1. `initialize()` creates state.txt with correct ScrubSchedule
|
||||||
|
2. REST API starts (port 8443 opens on the IncusOS daemon)
|
||||||
|
3. Incus application starts (port 8443 now proxies to Incus)
|
||||||
|
4. `registerJobs()` finds empty ScrubSchedule → ERROR
|
||||||
|
5. 15-second sleep before `os.Exit(1)`
|
||||||
|
6. During this 15-second window, port 8443 is reachable and API works
|
||||||
|
7. After exit, daemon restarts → corrupt state.txt → crash loop
|
||||||
|
|
||||||
|
**Detection**: `scrub_schedule` empty in API response = bug hit.
|
||||||
|
`scrub_schedule` = `"0 4 * * 0"` = genuine success.
|
||||||
|
|
||||||
|
### 4. Console messages are definitive
|
||||||
|
|
||||||
|
| Last console line | Status |
|
||||||
|
|-------------------|--------|
|
||||||
|
| `INFO System is ready` | Genuine success |
|
||||||
|
| `ERROR invalid crontab expression` | Crontab bug |
|
||||||
|
|
||||||
|
### 5. The bug is genuinely random
|
||||||
|
|
||||||
|
Same ISO, same VM specs, same network, same Proxmox host — some VMs
|
||||||
|
succeed and some fail within the same batch. No positional pattern
|
||||||
|
(VMID, creation order). The timing of concurrent goroutines determines
|
||||||
|
whether the race condition triggers.
|
||||||
|
|
||||||
|
### 6. Version mismatch path adds significant time
|
||||||
|
|
||||||
|
When ISO version != CDN latest (tested with old ISO 202602200553 against
|
||||||
|
CDN 202602210344), the first boot downloads a ~3 GB OS update and reboots.
|
||||||
|
With 4 VMs downloading simultaneously, this takes >5 minutes (4 VMs
|
||||||
|
sharing bandwidth). The second boot then goes through the normal startup.
|
||||||
|
This path was not fully tested for bug rate due to timeout constraints.
|
||||||
|
|
||||||
|
## Timeline (typical first boot from disk)
|
||||||
|
|
||||||
|
```
|
||||||
|
T+0s UEFI firmware
|
||||||
|
T+5s "IncusOS is starting..."
|
||||||
|
T+10s state.LoadOrCreate() → initialize() → ScrubSchedule="0 4 * * 0"
|
||||||
|
T+10s "Auto-generating encryption recovery key"
|
||||||
|
T+15s "System is starting up" (REST API starts)
|
||||||
|
T+20s "Bringing up the network"
|
||||||
|
T+25s "Downloading SecureBoot update" → "Applying Secure Boot certificate"
|
||||||
|
T+25s "Downloading application update" (Incus sysext, ~2 min)
|
||||||
|
T+140s "Bringing up the local storage"
|
||||||
|
T+142s "Starting application" → "Initializing application"
|
||||||
|
T+145s "Application TLS certificate fingerprint" (port 8443 now open)
|
||||||
|
T+147s registerJobs() — SUCCESS: "System is ready"
|
||||||
|
— or FAILURE: "ERROR invalid crontab expression"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Root Cause Analysis
|
||||||
|
|
||||||
|
The race condition occurs between `initialize()` setting ScrubSchedule and
|
||||||
|
`registerJobs()` reading it. Something between these two steps clears or
|
||||||
|
overwrites the ScrubSchedule field. The most likely candidates:
|
||||||
|
|
||||||
|
1. **Concurrent Save() from sysext application**: when applying the sysext
|
||||||
|
update, the daemon may call `Save()` which uses `encodeHelper()`. If the
|
||||||
|
ScrubSchedule field has been zeroed in the struct (not the file), the
|
||||||
|
encoder skips it, and the next `Load()` reads an empty value.
|
||||||
|
|
||||||
|
2. **API handler race**: the REST API is active during startup. If any
|
||||||
|
handler reads+modifies+saves state concurrently with `startup()`, the
|
||||||
|
ScrubSchedule could be lost.
|
||||||
|
|
||||||
|
3. **Non-atomic writes**: `Save()` uses `os.WriteFile()` (truncate+write).
|
||||||
|
A concurrent `Load()` during write could read a partial file.
|
||||||
|
|
||||||
|
The `defer s.Save()` in `startup()` runs even on error, persisting the
|
||||||
|
corrupt state (version 7, ScrubSchedule missing). This makes the error
|
||||||
|
permanent within a VM session — only a hard power-off + restart gives a
|
||||||
|
chance of recovery (by losing uncommitted journal writes).
|
||||||
|
|
||||||
|
## Recommendations
|
||||||
|
|
||||||
|
### For upstream (IncusOS issue #843):
|
||||||
|
1. **Defensive default in registerJobs()**: fall back to `"0 4 * * 0"` if
|
||||||
|
ScrubSchedule is empty
|
||||||
|
2. **Atomic state writes**: use write-to-temp + rename pattern in Save()
|
||||||
|
3. **Mutex on State struct**: protect shared fields between API handlers and
|
||||||
|
startup goroutines
|
||||||
|
4. **Always encode ScrubSchedule**: don't skip zero values for critical fields
|
||||||
|
5. **Don't defer Save() unconditionally**: only save on success path
|
||||||
|
|
||||||
|
### For deployment (incusos-proxmox):
|
||||||
|
1. **3 retries** gives 96.3% success rate (1 - 0.67^3) — current default
|
||||||
|
2. **fix_scrub_schedule()** heals the root cause via API — already implemented
|
||||||
|
3. **Monitor scrub_schedule** on every deployed node — already implemented
|
||||||
|
4. **Use latest ISO** to avoid the long OS update + reboot path (saves ~5 min)
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- Investigation tool: `incusos/investigate-parallel`
|
||||||
|
- API helper: `incusos/pve-api`
|
||||||
|
- Results: `incusos/observe-runs/parallel_2026-02-21_*`
|
||||||
|
- Screenshots: `incusos/observe-runs/parallel_2026-02-21_*/vm-*/frame-*.png`
|
||||||
Loading…
Reference in New Issue