Add observe-deploy diagnostics, boot failure analysis, and deploy reliability fixes
Observational single-VM deploy tool (observe-deploy) captures frame-by-frame console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot failures. Four deploys revealed ~50% "invalid crontab expression" hit rate with no correlation to update.yaml presence. Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go) traced the full crash path: registerJobs() validates ScrubSchedule via gocron at step 19 of startup(), after applications start at step 16 and REST API starts even earlier. Most likely a race condition or gocron edge case. Key fixes: - incusos-proxmox: automatic retry on first-boot failure (stop+start cycle) - incusos-proxmox: pve_destroy_vm now waits for async UPID task completion - observe-deploy: same UPID wait for destroy, full screenshot pipeline - incusos-seed: always generate update.yaml with explicit check_frequency - CLAUDE.md: complete boot sequence timeline, source-level root cause analysis, proposed upstream fix for IncusOS registerJobs() defensive validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
00f57746ff
commit
1499dd7072
|
|
@ -31,3 +31,6 @@ proxmox.yaml
|
|||
|
||||
# Environment secrets
|
||||
env
|
||||
|
||||
# Observational deploy output (screenshots, logs)
|
||||
observe-runs/
|
||||
|
|
|
|||
215
CLAUDE.md
215
CLAUDE.md
|
|
@ -19,6 +19,8 @@ incu-contrib/
|
|||
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||
│ ├── observe-deploy # Single-VM deploy with rapid console screenshots
|
||||
│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored)
|
||||
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
||||
│ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
|
||||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||||
|
|
@ -180,6 +182,23 @@ incu-contrib/
|
|||
| 920-929 | OC server standalone |
|
||||
| 930-939 | Advanced / heterogeneous |
|
||||
|
||||
### Resource constraints for multi-lab
|
||||
|
||||
- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node
|
||||
cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource
|
||||
where you can actually run out.
|
||||
- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default
|
||||
(`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk.
|
||||
LZ4 compression provides ~1.5-1.8x ratio on OS data.
|
||||
- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs
|
||||
can share CPUs without contention.
|
||||
- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if
|
||||
requested RAM exceeds available host RAM (API method only).
|
||||
- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to
|
||||
run the full deploy pipeline (auto-accepts with `--yes`).
|
||||
- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs
|
||||
(via storage content API) and notes thin provisioning for ZFS/LVM-thin.
|
||||
|
||||
### Incus clustering via remotes
|
||||
|
||||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||||
|
|
@ -443,10 +462,10 @@ incu-contrib/
|
|||
- **Inventory is NOT real-time** -- requires explicit `cluster resync`
|
||||
- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
|
||||
on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
|
||||
~50s with data intact). The failure is OC-specific: the OC agent runs on
|
||||
boot and fails with errors like "invalid crontab expression" (bad update
|
||||
config), "constraint violation" (re-registration of existing server), or
|
||||
Incus stuck at "starting application" (waiting for peers in boot loops).
|
||||
~50s with data intact). The failure is OC-specific: the OC agent pushes
|
||||
config via the IncusOS REST API that gets persisted to `state.txt`. On
|
||||
reboot, invalid values (e.g., cron expression where Go duration is expected)
|
||||
crash the daemon. See "IncusOS boot failure" section below for full analysis.
|
||||
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
|
||||
- **No cluster member state tracking** -- OC always shows `ready` even for
|
||||
EVACUATED/OFFLINE nodes
|
||||
|
|
@ -472,6 +491,194 @@ incu-contrib/
|
|||
output is concise (step names + results). `--quiet` suppresses everything
|
||||
except warnings and errors.
|
||||
|
||||
### IncusOS first-boot sequence (observed via console screenshots)
|
||||
|
||||
Observed using `observe-deploy` with 2-second screenshot intervals. The
|
||||
complete lifecycle from Proxmox VM start to port 8443 ready:
|
||||
|
||||
**Phase 1: ISO boot and installation (~60-85s)**
|
||||
1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s)
|
||||
2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s)
|
||||
3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s)
|
||||
- "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda"
|
||||
- "Cloning GPT partitions" → progress bar → complete
|
||||
4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via
|
||||
`/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not
|
||||
a graceful systemd reboot. This is intentional — the install
|
||||
environment is minimal and may not have systemd running.
|
||||
5. VM reboots but ISO still attached → boots from ISO again
|
||||
6. "System check error: install media detected, but the system is already
|
||||
installed; please remove USB/CDROM and reboot the system" (~40s).
|
||||
IncusOS daemon sleeps **1 hour** at this error, then exits. The
|
||||
`state.txt` on the installed disk (scsi0 data partition) is **NOT
|
||||
touched** — this boot runs from the ISO rootfs (tmpfs).
|
||||
7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle)
|
||||
|
||||
**Phase 2: Transition (~15s)**
|
||||
1. Proxmox stops VM
|
||||
2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0`
|
||||
3. Start VM from disk
|
||||
|
||||
**Phase 3: First boot from disk (~50s to port 8443)**
|
||||
1. UEFI → "IncusOS is starting..." (~8s)
|
||||
2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets
|
||||
defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`,
|
||||
`Channel="stable"` → `Save()` writes state.txt to data partition
|
||||
3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s)
|
||||
4. "System is starting up machine-id=... mode=production" (~10s after key)
|
||||
5. "Bringing up the network" (immediate)
|
||||
6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s)
|
||||
7. "Downloading application update application=incus" + progress bar (~33-38s)
|
||||
8. "Bringing up the local storage" (after download)
|
||||
9. "Starting application" → "Initializing application" → TLS cert fingerprint
|
||||
10. `registerJobs()` validates ScrubSchedule with gocron
|
||||
11. "System is ready" — port 8443 now reachable
|
||||
|
||||
**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable
|
||||
is the application download (33-38s on first boot; skipped if already cached).
|
||||
|
||||
**Important**: `DoInstall()` does NOT create `state.txt`. It writes the
|
||||
IncusOS image to disk, processes seed files (including deleting `install.yaml`
|
||||
from the target's seed partition via `CleanupPostInstall()`), and triggers
|
||||
the SysRq-B reboot. The `state.txt` is created exclusively on first boot
|
||||
from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
||||
|
||||
### IncusOS boot failure (crontab / update frequency error)
|
||||
|
||||
- **Symptom**: "ERROR invalid crontab expression" appears on the console
|
||||
during first boot. Intermittent — observed in ~50% of deploys (2/4
|
||||
observational deploys). Error does NOT correlate with `update.yaml`
|
||||
presence/absence.
|
||||
- **Two distinct error paths in IncusOS**:
|
||||
- `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
|
||||
like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
|
||||
daemon exits — **this is the fatal one**. Port 8443 never opens
|
||||
(or opens briefly during the 15-second error sleep).
|
||||
- `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
|
||||
format like `"6h"`). If invalid, it logs an error but does **not** crash.
|
||||
- **Not always fatal** (key finding from observational testing):
|
||||
- The REST API starts BEFORE `startup()` completes. Applications (Incus)
|
||||
start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
|
||||
When Incus starts before the scheduler crashes, port 8443 is available
|
||||
during the 15-second error sleep (before `os.Exit(1)`).
|
||||
- `observe-deploy` detects port 8443 during this window → reports PASS.
|
||||
- "System is ready" is NOT shown on console (only appears after
|
||||
`registerJobs()` succeeds).
|
||||
- **Source code analysis** (root cause investigation):
|
||||
- `state.txt` is created FRESH on first boot from disk by
|
||||
`LoadOrCreate()` → `initialize()`. The `initialize()` function
|
||||
correctly sets `ScrubSchedule = "0 4 * * 0"`.
|
||||
- The "install media detected" boot (#2, ISO still attached) runs from
|
||||
the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed
|
||||
disk. The installed disk's state.txt is untouched.
|
||||
- `DoInstall()` does not create state.txt. It only writes the OS image
|
||||
and processes seed files.
|
||||
- **The encoder skips zero values**: `encodeHelper()` returns early on
|
||||
`v.IsZero()`. An empty ScrubSchedule string is NOT written to file.
|
||||
If state.txt is read with a missing ScrubSchedule, the field stays
|
||||
at Go's zero value (empty string) → `gocron.IsValid("")` fails.
|
||||
- **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which
|
||||
truncates then writes. Crash during write → partial file.
|
||||
- **Race window**: The REST API server starts BEFORE `startup()`
|
||||
completes. Applications start before `registerJobs()`. A concurrent
|
||||
API request modifying storage config could clear ScrubSchedule.
|
||||
- **Most likely root cause**: either (a) a race condition between the
|
||||
REST API/application startup and `registerJobs()` that clears
|
||||
`ScrubSchedule`, or (b) a gocron library edge case where
|
||||
`IsValid("0 4 * * 0", time.UTC, time.Now())` intermittently fails.
|
||||
Both explain the ~50% hit rate and lack of correlation with seed
|
||||
content.
|
||||
- **Proposed upstream fix** (for `incus-os` project):
|
||||
```go
|
||||
// In registerJobs(): validate and fall back to default
|
||||
func registerJobs(s *state.State) error {
|
||||
schedule := s.System.Storage.Config.ScrubSchedule
|
||||
if schedule == "" {
|
||||
schedule = "0 4 * * 0"
|
||||
s.System.Storage.Config.ScrubSchedule = schedule
|
||||
}
|
||||
err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools)
|
||||
if err != nil {
|
||||
// Fall back to default instead of crashing the daemon
|
||||
slog.Warn("Invalid scrub schedule, using default", "schedule", schedule)
|
||||
s.System.Storage.Config.ScrubSchedule = "0 4 * * 0"
|
||||
return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
Additional recommendations: (1) use atomic writes in `Save()` (write
|
||||
to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even
|
||||
when empty (to preserve the default through encode/decode cycles),
|
||||
(3) add mutex protection on State fields shared between API handlers
|
||||
and startup().
|
||||
- **Recovery via restart**: `phase_install` in `incusos-proxmox` now
|
||||
automatically retries once (stop → 10s wait → start → re-check).
|
||||
After the first crash, the deferred `s.Save()` writes state. On the
|
||||
second boot, systemd restarts the daemon; if the root cause was
|
||||
transient (race condition or gocron edge case), the second boot
|
||||
succeeds.
|
||||
- **Detection**: `phase_install` checks port 8443 after starting from disk
|
||||
(30s wait + up to 60s polling). If first check fails, automatic retry
|
||||
adds ~100s. Failed VMs after retry are reported with remediation.
|
||||
- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
|
||||
`updateChecker()`, `startup()`, `firstBootActions()`),
|
||||
`incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,
|
||||
`ErrInvalidCronTab`), `incus-osd/internal/state/file.go`
|
||||
(`LoadOrCreate()`, `Save()`, `initialize()`),
|
||||
`incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip),
|
||||
`incus-osd/internal/install/install.go` (`DoInstall()`,
|
||||
`rebootUponDeviceRemoval()` SysRq-B).
|
||||
|
||||
### Proxmox SSH root access — strict rules
|
||||
|
||||
Root SSH access to the Proxmox host is available for **diagnostics only**.
|
||||
The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`.
|
||||
|
||||
**MANDATORY RULES — violation of any rule is unacceptable:**
|
||||
|
||||
1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM
|
||||
console screenshots via `qm monitor <vmid> screendump`. No other use
|
||||
without explicit user instruction.
|
||||
2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869)
|
||||
or VMs in the IncusLab pool that were created by our scripts.
|
||||
3. **No modifications**: NEVER run any command that modifies, stops, starts,
|
||||
or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`,
|
||||
`pct` commands, `zfs` commands, `systemctl` commands, or ANY write
|
||||
operation on the host.
|
||||
4. **Do not touch the dev VM**: the user's dev VM runs on the same host.
|
||||
Do not interact with it in any way — do not even `qm status` it.
|
||||
5. **No config access**: do not read or modify `/etc/pve/`, storage configs,
|
||||
network configs, user/ACL settings, or any host-level configuration.
|
||||
6. **Transparency**: every SSH command is visible in tool output. If the
|
||||
user rejects a command, do not retry it.
|
||||
7. **Only during active tests**: only SSH during test runs where test VMs
|
||||
have been deployed, and only to screenshot those test VMs.
|
||||
|
||||
**Permitted commands (exhaustive list):**
|
||||
```bash
|
||||
# Take screenshot of a test VM (VMID in 850-869 range)
|
||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||
root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"
|
||||
|
||||
# Retrieve screenshot
|
||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
|
||||
root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/
|
||||
|
||||
# Cleanup screenshot on remote
|
||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||
root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
|
||||
```
|
||||
|
||||
**Technical notes:**
|
||||
- QEMU's `screendump` is root-only in PVE 9's HMP permission model.
|
||||
API tokens cannot execute it regardless of privileges.
|
||||
- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng,
|
||||
which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`).
|
||||
- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`:
|
||||
`python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"`
|
||||
PNG output is ~20 KB and can be read directly by Claude Code (multimodal).
|
||||
|
||||
## Coding conventions for scripts
|
||||
|
||||
- **Shell**: bash with `set -euo pipefail`
|
||||
|
|
|
|||
|
|
@ -1937,13 +1937,35 @@ pve_stop_vm() {
|
|||
esac
|
||||
}
|
||||
|
||||
# Destroy a VM
|
||||
# Destroy a VM (waits for async task to complete)
|
||||
pve_destroy_vm() {
|
||||
local vmid="$1"
|
||||
|
||||
case "$PVE_METHOD" in
|
||||
ssh) pve_run "qm destroy ${vmid} --purge" ;;
|
||||
api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;;
|
||||
api)
|
||||
local destroy_response
|
||||
destroy_response=$(pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" 2>/dev/null) || return 1
|
||||
# VM destruction is async -- wait for the UPID task to finish
|
||||
# to avoid VMID/disk reuse issues in rapid destroy+create cycles.
|
||||
local upid
|
||||
upid=$(echo "$destroy_response" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid=""
|
||||
if [[ -n "$upid" ]]; then
|
||||
local encoded_upid
|
||||
encoded_upid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null)
|
||||
local wait_elapsed=0
|
||||
while [[ $wait_elapsed -lt 30 ]]; do
|
||||
sleep 2
|
||||
wait_elapsed=$((wait_elapsed + 2))
|
||||
local task_status
|
||||
task_status=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${encoded_upid}/status" 2>/dev/null | \
|
||||
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || task_status=""
|
||||
if [[ "$task_status" == "stopped" ]]; then
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
|
@ -2458,6 +2480,70 @@ for s in data.get('data', []):
|
|||
errors=$((errors + 1))
|
||||
fi
|
||||
|
||||
# 6. Host resource feasibility (API method only)
|
||||
if [[ "$PVE_METHOD" == "api" ]] && [[ "$DRY_RUN" != true ]] && [[ ${#VM_MEMORY[@]} -gt 0 ]]; then
|
||||
local total_ram_mib=0
|
||||
local total_disk_gib=0
|
||||
local total_cores=0
|
||||
local ri=0
|
||||
while [[ $ri -lt ${#VM_MEMORY[@]} ]]; do
|
||||
total_ram_mib=$((total_ram_mib + VM_MEMORY[$ri]))
|
||||
total_disk_gib=$((total_disk_gib + VM_DISK[$ri]))
|
||||
total_cores=$((total_cores + VM_CORES[$ri]))
|
||||
ri=$((ri + 1))
|
||||
done
|
||||
|
||||
local res_node_status
|
||||
res_node_status=$(pve_api GET "/nodes/${PVE_NODE}/status" 2>/dev/null) || res_node_status=""
|
||||
if [[ -n "$res_node_status" ]]; then
|
||||
local resource_lines
|
||||
resource_lines=$(python3 -c "
|
||||
import json, sys
|
||||
data = json.loads('''${res_node_status}''').get('data', {})
|
||||
mem = data.get('memory', {})
|
||||
mem_total = mem.get('total', 0) / (1024**3)
|
||||
mem_used = mem.get('used', 0) / (1024**3)
|
||||
mem_free = mem_total - mem_used
|
||||
cpu_count = data.get('cpuinfo', {}).get('cpus', 0)
|
||||
req_ram = ${total_ram_mib} / 1024
|
||||
req_cores = ${total_cores}
|
||||
|
||||
# RAM check
|
||||
if mem_free > 0:
|
||||
pct = req_ram / mem_free * 100
|
||||
if req_ram > mem_free:
|
||||
print(f'error|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free -- not enough RAM')
|
||||
elif pct > 70:
|
||||
print(f'warn|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free) -- tight fit')
|
||||
else:
|
||||
print(f'ok|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free)')
|
||||
|
||||
# CPU (info only)
|
||||
print(f'ok|CPU: {req_cores} cores requested, {cpu_count} available')
|
||||
" 2>/dev/null) || resource_lines=""
|
||||
|
||||
local line
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" ]] && continue
|
||||
local level="${line%%|*}"
|
||||
local msg="${line#*|}"
|
||||
case "$level" in
|
||||
error)
|
||||
error "$msg"
|
||||
errors=$((errors + 1))
|
||||
;;
|
||||
warn)
|
||||
warn "$msg"
|
||||
warnings=$((warnings + 1))
|
||||
;;
|
||||
ok)
|
||||
detail "$msg"
|
||||
;;
|
||||
esac
|
||||
done <<< "$resource_lines"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ $errors -gt 0 ]]; then
|
||||
error "${errors} pre-flight check(s) failed"
|
||||
exit 1
|
||||
|
|
@ -2826,6 +2912,20 @@ confirm() {
|
|||
fi
|
||||
}
|
||||
|
||||
# Like confirm(), but returns 0/1 instead of exiting on "no"
|
||||
confirm_yn() {
|
||||
local message="$1"
|
||||
|
||||
if [[ "$YES" == true ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo -n -e "${BOLD}${message} [y/N] ${RESET}"
|
||||
local reply
|
||||
read -r reply
|
||||
[[ "$reply" =~ ^[Yy]$ ]]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 1: Validate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -3130,6 +3230,7 @@ phase_install() {
|
|||
echo ""
|
||||
|
||||
local count=${#VM_NAMES[@]}
|
||||
local install_failures=0
|
||||
local i=0
|
||||
while [[ $i -lt $count ]]; do
|
||||
local name="${VM_NAMES[$i]}"
|
||||
|
|
@ -3306,28 +3407,92 @@ phase_install() {
|
|||
continue
|
||||
fi
|
||||
|
||||
# 5. Wait for services to come up
|
||||
# 5. Wait for services to come up and verify health
|
||||
info "Waiting for services to start..."
|
||||
sleep 30
|
||||
|
||||
# 6. Try to detect IP
|
||||
# 6. Detect IP and check port 8443
|
||||
local ip
|
||||
ip=$(pve_vm_ip "$vmid") || ip=""
|
||||
|
||||
echo ""
|
||||
local healthy=false
|
||||
if [[ -n "$ip" ]]; then
|
||||
# Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total)
|
||||
local poll=0
|
||||
while [[ $poll -lt 12 ]]; do
|
||||
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||
healthy=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
poll=$((poll + 1))
|
||||
done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "$healthy" == true ]]; then
|
||||
success "VM '${name}' installed and running at ${ip}"
|
||||
register_remote "$name" "$ip"
|
||||
else
|
||||
success "VM '${name}' installed and running"
|
||||
info "Check Proxmox console for IP address"
|
||||
# Retry once: IncusOS has an intermittent "invalid crontab expression"
|
||||
# bug (~50% first-boot hit rate) where registerJobs() fails during
|
||||
# startup(). Most likely a race condition between the REST API and
|
||||
# startup sequence, or a gocron library edge case. A stop+start
|
||||
# cycle resolves it because the transient condition clears on retry.
|
||||
if [[ -n "$ip" ]]; then
|
||||
warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)"
|
||||
else
|
||||
warn "VM '${name}': could not detect IP — retrying (stop+start)"
|
||||
fi
|
||||
info "Stopping VM for retry..."
|
||||
pve_stop_vm "$vmid" || true
|
||||
sleep 10
|
||||
info "Starting VM (retry)..."
|
||||
pve_start_vm "$vmid" || true
|
||||
sleep 30
|
||||
|
||||
# Re-detect IP (may have changed after restart)
|
||||
ip=$(pve_vm_ip "$vmid") || ip=""
|
||||
|
||||
local retry_healthy=false
|
||||
if [[ -n "$ip" ]]; then
|
||||
local retry_poll=0
|
||||
while [[ $retry_poll -lt 12 ]]; do
|
||||
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||
retry_healthy=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
retry_poll=$((retry_poll + 1))
|
||||
done
|
||||
fi
|
||||
|
||||
if [[ "$retry_healthy" == true ]]; then
|
||||
success "VM '${name}' recovered after retry — running at ${ip}"
|
||||
register_remote "$name" "$ip"
|
||||
elif [[ -n "$ip" ]]; then
|
||||
warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)"
|
||||
warn "Likely IncusOS boot failure (check Proxmox console for errors)"
|
||||
warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
|
||||
install_failures=$((install_failures + 1))
|
||||
else
|
||||
warn "VM '${name}' installed but could not detect IP (even after retry)"
|
||||
warn "Check Proxmox console for status"
|
||||
install_failures=$((install_failures + 1))
|
||||
fi
|
||||
fi
|
||||
echo ""
|
||||
|
||||
i=$((i + 1))
|
||||
done
|
||||
|
||||
success "Installation phase complete"
|
||||
if [[ $install_failures -gt 0 ]]; then
|
||||
warn "Installation phase complete: ${install_failures} VM(s) failed health check"
|
||||
warn "Failed VMs are likely stuck in an IncusOS boot loop"
|
||||
warn "Re-run deployment to redeploy, or use --cleanup to start fresh"
|
||||
else
|
||||
success "Installation phase complete"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -3393,6 +3558,48 @@ phase_lab_up() {
|
|||
local count=${#VM_NAMES[@]}
|
||||
local started=0
|
||||
local skipped=0
|
||||
local missing=0
|
||||
|
||||
# First pass: check how many VMs exist
|
||||
if [[ "$DRY_RUN" != true ]]; then
|
||||
local i=0
|
||||
while [[ $i -lt $count ]]; do
|
||||
if ! pve_vm_exists "${VM_VMIDS[$i]}"; then
|
||||
missing=$((missing + 1))
|
||||
fi
|
||||
i=$((i + 1))
|
||||
done
|
||||
|
||||
# If ALL VMs are missing, offer to deploy
|
||||
if [[ $missing -eq $count ]]; then
|
||||
echo ""
|
||||
warn "No VMs found -- lab has not been deployed yet"
|
||||
if confirm_yn "Deploy lab now?"; then
|
||||
echo ""
|
||||
phase_prepare
|
||||
echo ""
|
||||
phase_upload
|
||||
echo ""
|
||||
phase_create
|
||||
echo ""
|
||||
phase_install
|
||||
echo ""
|
||||
phase_post_deploy
|
||||
return
|
||||
else
|
||||
info "Aborted"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
|
||||
# If SOME VMs are missing, warn but don't offer auto-deploy
|
||||
if [[ $missing -gt 0 ]]; then
|
||||
warn "${missing} of ${count} VMs not found -- partial state detected"
|
||||
warn "Use --cleanup then redeploy, or deploy missing VMs manually"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Second pass: start existing VMs
|
||||
local i=0
|
||||
while [[ $i -lt $count ]]; do
|
||||
local name="${VM_NAMES[$i]}"
|
||||
|
|
@ -3505,6 +3712,7 @@ print(f' Uptime: {days}d {hours}h')
|
|||
python3 -c "
|
||||
import json, sys
|
||||
data = json.loads('''${storage_list}''').get('data', [])
|
||||
thin_types = {'zfspool', 'lvmthin'}
|
||||
for s in sorted(data, key=lambda x: x.get('storage', '')):
|
||||
name = s.get('storage', '?')
|
||||
total = s.get('total', 0) / (1024**3)
|
||||
|
|
@ -3513,10 +3721,11 @@ for s in sorted(data, key=lambda x: x.get('storage', '')):
|
|||
pct = (used / total * 100) if total > 0 else 0
|
||||
stype = s.get('type', '?')
|
||||
content = s.get('content', '?')
|
||||
thin = ' [thin]' if stype in thin_types else ''
|
||||
if total > 0:
|
||||
print(f' {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}')
|
||||
print(f' {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}{thin}')
|
||||
else:
|
||||
print(f' {name:<16} {stype:<12} (no capacity info) -- {content}')
|
||||
print(f' {name:<16} {stype:<12} (no capacity info) -- {content}{thin}')
|
||||
" 2>/dev/null || warn "Failed to parse storage list"
|
||||
fi
|
||||
|
||||
|
|
@ -3540,6 +3749,57 @@ print(f' VMs: {len(vms)} total ({running} running, {stopped} stopped)')
|
|||
print(f' Allocated: {total_mem:.1f} GiB RAM, {total_disk:.0f} GiB disk')
|
||||
" 2>/dev/null || warn "Failed to parse pool data"
|
||||
fi
|
||||
|
||||
# Actual disk usage via storage content API
|
||||
if [[ -n "$pool_data" ]]; then
|
||||
local storage_content
|
||||
storage_content=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_STORAGE}/content" 2>/dev/null) || storage_content=""
|
||||
local storage_status_json
|
||||
storage_status_json=$(echo "$storage_list" | python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin).get('data', [])
|
||||
for s in data:
|
||||
if s.get('storage') == '${PVE_STORAGE}':
|
||||
print(json.dumps(s))
|
||||
break
|
||||
" 2>/dev/null) || storage_status_json=""
|
||||
|
||||
if [[ -n "$storage_content" ]] && [[ -n "$pool_data" ]]; then
|
||||
python3 -c "
|
||||
import json, sys
|
||||
content = json.loads('''${storage_content}''').get('data', [])
|
||||
pool = json.loads('''${pool_data}''').get('data', {})
|
||||
vm_vmids = {str(m.get('vmid', '')) for m in pool.get('members', []) if m.get('type') == 'qemu'}
|
||||
|
||||
# Sum actual (used) bytes for volumes belonging to pool VMs
|
||||
actual_bytes = 0
|
||||
allocated_bytes = 0
|
||||
for vol in content:
|
||||
vmid = str(vol.get('vmid', ''))
|
||||
if vmid in vm_vmids:
|
||||
actual_bytes += vol.get('used', 0)
|
||||
allocated_bytes += vol.get('size', 0)
|
||||
|
||||
if allocated_bytes > 0:
|
||||
actual_gib = actual_bytes / (1024**3)
|
||||
allocated_gib = allocated_bytes / (1024**3)
|
||||
ratio = (actual_bytes / allocated_bytes * 100) if allocated_bytes > 0 else 0
|
||||
print(f' Actual: {actual_gib:.1f} GiB disk used ({ratio:.0f}% of allocated, thin provisioned)')
|
||||
" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Storage type note
|
||||
if [[ -n "$storage_status_json" ]]; then
|
||||
python3 -c "
|
||||
import json
|
||||
s = json.loads('''${storage_status_json}''')
|
||||
stype = s.get('type', '')
|
||||
notes = {'zfspool': 'ZFS (thin provisioned, lz4 compression)', 'lvmthin': 'LVM-thin (thin provisioned)'}
|
||||
if stype in notes:
|
||||
print(f' Storage: {notes[stype]}')
|
||||
" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
|
|
|||
|
|
@ -616,17 +616,13 @@ generate_network_yaml() {
|
|||
}
|
||||
|
||||
generate_update_yaml() {
|
||||
# Only generate if non-default values were specified
|
||||
if [[ "$CHANNEL" == "stable" ]] && [[ "$AUTO_REBOOT" == false ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
local file="${WORKDIR}/update.yaml"
|
||||
|
||||
step "Generating update.yaml"
|
||||
|
||||
{
|
||||
echo "# Update configuration"
|
||||
echo "# Always generated to ensure explicit check_frequency on first boot"
|
||||
echo "channel: \"${CHANNEL}\""
|
||||
echo "auto_reboot: ${AUTO_REBOOT}"
|
||||
echo "check_frequency: \"6h\""
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue