diff --git a/CLAUDE.md b/CLAUDE.md index 219d89e..c7968ce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,6 +18,7 @@ incu-contrib/ │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator │ ├── incusos-proxmox # Declarative Proxmox VM deployment +│ ├── TESTING.md # Testing guide for incusos-proxmox │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material ``` @@ -51,7 +52,12 @@ incu-contrib/ - Tar archives containing YAML files at the root level. - Written to byte offset 2148532224 (the seed partition) in the image. -- Alternative: FAT image labeled `SEED_DATA` as external boot media. +- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM. + - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM + devices. Volume labels are correctly detected by the kernel on `/dev/sr*`. + - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but + **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose + FAT filesystem labels, so IncusOS cannot find the seed. - Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, `operations-center.yaml`, `network.yaml`, `update.yaml`. @@ -59,7 +65,9 @@ incu-contrib/ - Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. - Running `incus remote list` triggers auto-generation if no keypair exists. -- For Incus seed: injected under `preseed.server.certificates[]`. +- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`). + The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including + `certificates`) are promoted to the top level of the `preseed` object. - For Operations Center seed: injected under `trusted_client_certificates[]`. - Operations Center **requires** at least one trusted certificate -- without it, you are locked out after installation. @@ -67,7 +75,7 @@ incu-contrib/ ### Proxmox VE deployment - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images - via `incusos-seed --format fat`, uploads the ISO + seeds to Proxmox, creates + via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation. - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). @@ -85,12 +93,29 @@ incu-contrib/ - `cpu=host` -- needed for x86_64_v3 instruction set requirement - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - `balloon=0` -- IncusOS manages memory internally - - `ide3` -- SEED_DATA FAT image attached as second CD-ROM + - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - Minimum 50 GiB disk, minimum 4096 MiB RAM -- **Install flow**: boot from ISO (ide2) -> read SEED_DATA (ide3) -> install - to disk (scsi0) -> force_reboot -> boot from disk. Boot order is swapped - to `order=scsi0;ide2` immediately after first start so the reboot lands - on disk. +- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. + IncusOS does **literal** string matching (not glob) on disk device IDs. + `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit + `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). +- **Install flow** (automated by `incusos-proxmox`): + 1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed + 2. IncusOS reads seed, installs to disk (scsi0), auto-reboots + 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- + when disk writes start then stop for 15s (3 stable polls), install is done + 4. Stop the VM (Proxmox stop, not guest shutdown) + 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot + and refuses to start if found, regardless of boot order + 6. Set boot order to `order=scsi0` and start from disk +- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use + ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP → + ping broadcast → look up MAC in ARP table. Verify with direct ping before + trusting the result. +- **force_reboot is required**: without `force_reboot: true` in the seed, + IncusOS sits at "please remove installation media" and waits indefinitely. + It does NOT halt automatically. `force_reboot` triggers a guest-level reboot + (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does). ## Coding conventions for scripts diff --git a/incusos/README.md b/incusos/README.md index 727ca8f..0f7a83c 100644 --- a/incusos/README.md +++ b/incusos/README.md @@ -7,7 +7,7 @@ Scripts for building IncusOS installation media and seed configurations. | Script | Purpose | |--------|---------| | [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool | -| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar or FAT) for installation customization | +| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization | | [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE | All scripts support automatic client certificate injection from the local Incus @@ -43,7 +43,10 @@ installation, so the freshly installed system is immediately manageable. **Optional:** -- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`) +- **genisoimage** -- for creating ISO 9660 seed images (`incusos-seed --format iso`). + Required by `incusos-proxmox` (Proxmox CD-ROM devices need ISO 9660, not FAT). +- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`). + Only for USB/block device use; FAT labels are NOT detected on CD-ROM devices. - **jq** or **python3** -- for parsing the CDN index (cross-architecture builds) - **curl** -- for downloading images from the CDN @@ -149,7 +152,8 @@ Generates IncusOS seed archives containing the YAML configuration files that customize an IncusOS installation. Output can be: - **tar archive** (default) -- for use with `flasher-tool --seed` -- **FAT image** -- for use as an external SEED_DATA partition/disk +- **ISO 9660 image** -- for use as a CD-ROM SEED_DATA device (e.g. Proxmox `ide3`) +- **FAT image** -- for use as a USB/block SEED_DATA device (NOT for CD-ROM) ### Quick Start @@ -160,8 +164,8 @@ that customize an IncusOS installation. Output can be: # Operations Center seed ./incusos-seed --app operations-center --defaults --cert ~/.config/incus/client.crt -# Cluster node (no defaults) targeting virtio disks -./incusos-seed --disk-target 'virtio-*' --force-install --force-reboot +# Cluster node (no defaults) with forced install and reboot +./incusos-seed --force-install --force-reboot # FAT image for external boot media ./incusos-seed --format fat --defaults --output seed.img @@ -178,7 +182,7 @@ that customize an IncusOS installation. Output can be: -c, --cert FILE Client certificate PEM file (default: auto-detect) --no-cert Skip certificate injection -o, --output FILE Output file (default: incusos-seed.tar) --f, --format FORMAT tar or fat (default: tar) +-f, --format FORMAT tar, fat, or iso (default: tar) --fat-size SIZE FAT image size in MiB (default: 10) --disk-target ID Target disk pattern --force-install Overwrite existing installation @@ -209,24 +213,40 @@ The following YAML files are generated inside the archive: | `network.yaml` | when network opts set | Hostname, DNS, static IP | | `update.yaml` | when non-default | Update channel, auto-reboot | -### Using the FAT Image (SEED_DATA) +### Using ISO 9660 Images (SEED_DATA for CD-ROM) + +The ISO output format creates a small ISO 9660 image with volume label `SEED_DATA`. +This is the **required** format when attaching seed data as a CD-ROM device (e.g. +Proxmox `ide3`). FAT filesystem labels are NOT detected by the Linux kernel on +CD-ROM (`/dev/sr*`) devices — only ISO 9660 volume labels work. + +```bash +# Generate ISO seed for Proxmox deployment +./incusos-seed --format iso --defaults --output seed-node-1.iso +``` + +`incusos-proxmox` uses this format automatically. + +### Using FAT Images (SEED_DATA for USB/Block Devices) The FAT output format creates a small disk image labeled `SEED_DATA` that can be -attached to a VM as a secondary disk. IncusOS detects this label at boot and -reads the seed configuration from it. +attached to a VM as a secondary block device or written to USB. IncusOS detects +this label at boot and reads the seed configuration from it. -This is useful for the "pristine image" workflow: +**Note:** FAT images do NOT work when attached as CD-ROM devices. Use ISO 9660 +format instead for CD-ROM use cases. + +This is useful for the "pristine image" workflow on bare metal: 1. Download one uncustomized IncusOS ISO 2. Generate per-node seed images with `incusos-seed --format fat` -3. Boot each VM with both the ISO and the node-specific seed image +3. Boot each machine with both the ISO and the node-specific seed USB ```bash -# Generate seeds for three cluster nodes +# Generate seeds for three cluster nodes (USB/block device use) for i in 1 2 3; do ./incusos-seed --format fat \ --hostname "node-${i}" \ - --disk-target 'virtio-*' \ --force-install \ --output "seed-node-${i}.img" done @@ -287,7 +307,7 @@ This script ensures all settings are correct, preventing install failures. | `prepare` | Download IncusOS ISO from CDN, generate per-VM SEED_DATA images | | `upload` | Upload ISO and seed images to Proxmox ISO storage | | `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) | -| `install` | Start VMs, swap boot order, wait for install, clean up CD-ROMs | +| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk | ### Config File Format @@ -357,11 +377,12 @@ Every VM is created with these IncusOS-specific settings: | CPU | host | Needs x86_64_v3 instruction set | | SCSI controller | virtio-scsi-pci | VirtIO-blk is broken with IncusOS | | Balloon | 0 (disabled) | IncusOS manages memory internally | -| SEED_DATA | Attached as ide3 CD-ROM | Per-VM seed configuration | +| SEED_DATA | ISO 9660 image on ide3 CD-ROM | Per-VM seed configuration | ### Prerequisites - **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing +- **genisoimage** -- for creating ISO 9660 seed images - **ssh** + **scp** (for SSH method) or **curl** (for API method) - **incusos-seed** -- sibling script (auto-detected in same directory) - **Bash 4+** -- associative arrays @@ -406,10 +427,11 @@ For full installation instructions per platform, see ### After Installation -With an injected certificate, the remote is already trusted: +With an injected certificate, the remote is already trusted — no authentication +prompt appears: ```bash -incus remote add my-server +incus remote add my-server --accept-certificate incus remote list ``` diff --git a/incusos/TESTING.md b/incusos/TESTING.md index 354ad0e..28fe55a 100644 --- a/incusos/TESTING.md +++ b/incusos/TESTING.md @@ -11,12 +11,13 @@ the full lifecycle: - **Config parsing** -- YAML with PyYAML or built-in fallback parser - **ISO download** -- fetches latest IncusOS from the CDN by channel (stable/testing) -- **Per-VM seed generation** -- calls `incusos-seed --format fat` for each VM +- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM - **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method) - **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM, VirtIO-SCSI, etc.) -- **Automated install monitoring** -- state-machine polling with 10-minute - timeout +- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API + to detect when disk writes stop (3 stable polls = 15s idle), then stops VM, + detaches install media (ide2 + ide3), and boots from disk - **Cleanup** -- safe teardown with managed-VM marker checks ### Cluster Formation (Manual Step) @@ -109,7 +110,7 @@ For a home lab, `/` (root) is simpler and perfectly acceptable. bash --version # Need 4+ python3 --version # For YAML parsing which jq curl ssh scp # All needed for SSH method -which mcopy mkfs.fat # Needed for seed FAT images (mtools + dosfstools) +which genisoimage # Needed for seed ISO images (used by incusos-proxmox) # Verify the Incus client has generated a keypair ls ~/.config/incus/client.crt ~/.config/incus/client.key @@ -184,7 +185,7 @@ Using VMID 900+ avoids clashing with existing VMs. 3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed 4. **Upload** -- SCPs ISO + seed to Proxmox 5. **Create** -- runs `qm create 900 ...` with all IncusOS settings -6. **Install** -- starts VM, monitors install completion (~3-5 min) +6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min) **Simultaneously on the Proxmox web UI:** @@ -200,8 +201,8 @@ Using VMID 900+ avoids clashing with existing VMs. Once the script reports success with an IP address: ```bash -# The cert was injected via seed, so it's already trusted -incus remote add incus-test https://:8443 --accept-certificate +# The cert was injected via seed, so no auth prompt appears +incus remote add incus-test --accept-certificate # Test connectivity incus remote list @@ -212,6 +213,10 @@ incus storage list incus-test: incus network list incus-test: ``` +**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses +ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, +check the Proxmox console — stale ARP entries can be misleading. + ### Step 6: Clean Up the Test VM ```bash @@ -292,7 +297,10 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment: | ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` | | VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` | | Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI | -| No IP reported | Guest agent not ready | Wait a minute, check Proxmox UI for IP | +| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection | +| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console | +| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start | +| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect | +| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` | | Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one | -| `mcopy` not found | mtools not installed | Install `mtools` package | -| `mkfs.fat` not found | dosfstools not installed | Install `dosfstools` package | +| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package | diff --git a/incusos/examples/incus-cluster-node.yaml b/incusos/examples/incus-cluster-node.yaml index d61c2b9..0b99f29 100644 --- a/incusos/examples/incus-cluster-node.yaml +++ b/incusos/examples/incus-cluster-node.yaml @@ -13,14 +13,13 @@ # --- incus.yaml --- apply_defaults: false preseed: - server: - certificates: - - name: "my-workstation" - type: "client" - certificate: | - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- + certificates: + - name: "my-workstation" + type: "client" + certificate: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- # --- network.yaml --- # dns: diff --git a/incusos/examples/incus-standalone.yaml b/incusos/examples/incus-standalone.yaml index 9a325d5..ff6cc36 100644 --- a/incusos/examples/incus-standalone.yaml +++ b/incusos/examples/incus-standalone.yaml @@ -28,11 +28,10 @@ # --- incus.yaml --- apply_defaults: true preseed: - server: - certificates: - - name: "my-workstation" - type: "client" - certificate: | - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- + certificates: + - name: "my-workstation" + type: "client" + certificate: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- diff --git a/incusos/examples/my-test.yaml b/incusos/examples/my-test.yaml new file mode 100644 index 0000000..5c79c69 --- /dev/null +++ b/incusos/examples/my-test.yaml @@ -0,0 +1,17 @@ +proxmox: + host: 192.168.1.29 + method: api + api_token_id: automation@pve!deploy + storage: local-zfs # VM disk storage (default: local-lvm) + +defaults: + start_vmid: 900 # High range to avoid collisions + +vms: + - name: incus-test + app: incus + apply_defaults: true + cores: 4 + memory: 8192 + disk: 64 + \ No newline at end of file diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index d01112a..19bde2c 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -423,9 +423,20 @@ check_dependencies() { # --------------------------------------------------------------------------- WORKDIR="" +CACHEDIR="" setup_workdir() { - WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-XXXXXX") + # Use persistent storage for the work directory -- the ISO download + + # decompression can exceed 4 GiB, which overflows tmpfs on most systems. + # Honour TMPDIR if explicitly set; otherwise use ~/.cache on real disk. + local base="${TMPDIR:-${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}}" + mkdir -p "$base" + + # Persistent cache for ISO downloads (survives across runs) + CACHEDIR="${base}/iso-cache" + mkdir -p "$CACHEDIR" + + WORKDIR=$(mktemp -d "${base}/${SCRIPT_NAME}-XXXXXX") trap 'rm -rf "$WORKDIR"' EXIT } @@ -1017,10 +1028,13 @@ pve_run() { } # Execute a Proxmox API call (API method) +# Usage: pve_api METHOD ENDPOINT [DATA] [json] +# Pass "json" as 4th arg to send data as application/json instead of form-encoded. pve_api() { local method="$1" local endpoint="$2" local data="${3:-}" + local content_type="${4:-form}" local url="https://${PVE_HOST}:8006/api2/json${endpoint}" local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" @@ -1034,6 +1048,9 @@ pve_api() { fi local curl_args=(-fsSk -H "$auth") + if [[ "$content_type" == "json" ]]; then + curl_args+=(-H "Content-Type: application/json") + fi case "$method" in GET) @@ -1082,13 +1099,48 @@ pve_upload() { "$local_path" "${PVE_SSH_USER}@${PVE_HOST}:${iso_dir}/${remote_filename}" ;; api) - curl -fsSk \ + local upload_response upload_rc + upload_response=$(curl -sSk \ -H "Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" \ -X POST \ -F "content=iso" \ - -F "filename=${remote_filename}" \ - -F "file=@${local_path}" \ - "https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" + -F "filename=@${local_path};filename=${remote_filename}" \ + "https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" \ + -w "\n%{http_code}" 2>&1) || true + local http_code + http_code=$(echo "$upload_response" | tail -1) + local body + body=$(echo "$upload_response" | sed '$d') + + if [[ "$http_code" -ge 200 ]] && [[ "$http_code" -lt 300 ]] 2>/dev/null; then + return 0 + else + error "Upload failed (HTTP ${http_code})" + # Extract error message from JSON response + local api_error + api_error=$(echo "$body" | python3 -c " +import json, sys +try: + data = json.load(sys.stdin) + errors = data.get('errors', {}) + if errors: + for k, v in errors.items(): + print(f' {k}: {v}') + elif data.get('message'): + print(f' {data[\"message\"]}') + else: + print(json.dumps(data, indent=2)) +except: + print(sys.stdin.read() if hasattr(sys.stdin, 'read') else '') +" 2>/dev/null) || api_error="$body" + if [[ -n "$api_error" ]]; then + error "Proxmox API response:" + echo "$api_error" | while IFS= read -r line; do + error " ${line}" + done + fi + return 1 + fi ;; esac } @@ -1178,33 +1230,34 @@ pve_create_vm() { pve_run "$cmd" ;; api) - # URL-encode the description for the API - local url_encoded_desc - url_encoded_desc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${description}'))" 2>/dev/null) - - # Build API parameters - local params="" - params+="vmid=${vmid}" - params+="&name=${name}" - params+="&description=${url_encoded_desc}" - params+="&ostype=l26" - params+="&bios=ovmf" - params+="&machine=q35" - params+="&efidisk0=${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0" - params+="&tpmstate0=${PVE_STORAGE}:1,version=v2.0" - params+="&cpu=host" - params+="&cores=${cores}" - params+="&memory=${memory}" - params+="&balloon=0" - params+="&scsihw=virtio-scsi-pci" - params+="&scsi0=${PVE_STORAGE}:${disk_gib},iothread=1" - params+="&ide2=${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom" - params+="&ide3=${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom" - params+="&net0=virtio,bridge=${PVE_BRIDGE}" - params+="&boot=order%3Dide2%3Bscsi0" - params+="&agent=1" - - pve_api POST "/nodes/${PVE_NODE}/qemu" "$params" + # Build JSON payload -- the Proxmox API requires JSON for VM + # creation because comma-separated disk/net specs are misinterpreted + # when sent as form-encoded data. + local json_payload + json_payload=$(python3 -c " +import json +print(json.dumps({ + 'vmid': ${vmid}, + 'name': '${name}', + 'description': '''${description}''', + 'ostype': 'l26', + 'bios': 'ovmf', + 'machine': 'q35', + 'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0', + 'tpmstate0': '${PVE_STORAGE}:1,version=v2.0', + 'cpu': 'host', + 'cores': ${cores}, + 'memory': ${memory}, + 'balloon': 0, + 'scsihw': 'virtio-scsi-pci', + 'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1', + 'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom', + 'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom', + 'net0': 'virtio,bridge=${PVE_BRIDGE}', + 'boot': 'order=ide2;scsi0', + 'agent': 1, +}))") + pve_api POST "/nodes/${PVE_NODE}/qemu" "$json_payload" json >/dev/null ;; esac } @@ -1215,7 +1268,7 @@ pve_start_vm() { case "$PVE_METHOD" in ssh) pve_run "qm start ${vmid}" ;; - api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" ;; + api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" >/dev/null ;; esac } @@ -1225,7 +1278,7 @@ pve_stop_vm() { case "$PVE_METHOD" in ssh) pve_run "qm stop ${vmid}" ;; - api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" ;; + api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" >/dev/null ;; esac } @@ -1235,7 +1288,7 @@ pve_destroy_vm() { case "$PVE_METHOD" in ssh) pve_run "qm destroy ${vmid} --purge" ;; - api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" ;; + api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;; esac } @@ -1248,10 +1301,35 @@ pve_set_vm() { case "$PVE_METHOD" in ssh) pve_run "qm set ${vmid} ${opts}" ;; api) - # Convert CLI-style opts to API params - local params - params=$(echo "$opts" | sed 's/--//g; s/ /\&/g; s/\&\([a-z]\)/\&\1/g') - pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$params" + # Convert CLI-style opts (--key value) to JSON. + # JSON avoids form-encoding ambiguity with values containing = and ; + # Duplicate keys (e.g. --delete ide2 --delete ide3) are merged with commas. + # Shell escapes (e.g. \;) are stripped since the API doesn't need them. + local json_payload + json_payload=$(echo "$opts" | python3 -c " +import json, sys +tokens = sys.stdin.read().split() +obj = {} +i = 0 +while i < len(tokens): + t = tokens[i] + if t.startswith('--'): + key = t.lstrip('-') + if i + 1 < len(tokens) and not tokens[i+1].startswith('--'): + val = tokens[i+1].replace(chr(92), '') + i += 2 + else: + val = '' + i += 1 + if key in obj and obj[key]: + obj[key] = obj[key] + ',' + val + else: + obj[key] = val + else: + i += 1 +print(json.dumps(obj)) +" 2>/dev/null) + pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$json_payload" json >/dev/null ;; esac } @@ -1271,6 +1349,42 @@ pve_vm_status() { esac } +# Get VM uptime (seconds); returns 0 if stopped or unknown +pve_vm_uptime() { + local vmid="$1" + + case "$PVE_METHOD" in + ssh) + pve_run "qm status ${vmid} --verbose" | awk '/^uptime:/{print $2}' + ;; + api) + pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \ + python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('uptime',0))" 2>/dev/null || echo "0" + ;; + esac +} + +# Get cumulative write bytes for scsi0 (from QEMU blockstat). +# Returns 0 if unavailable. Only works with api method. +pve_vm_disk_writes() { + local vmid="$1" + + case "$PVE_METHOD" in + ssh) + echo "0" + ;; + api) + pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \ + python3 -c " +import json,sys +d = json.load(sys.stdin).get('data',{}) +bs = d.get('blockstat',{}).get('scsi0',{}) +print(bs.get('wr_bytes',0)) +" 2>/dev/null || echo "0" + ;; + esac +} + # Check if a VMID exists pve_vm_exists() { local vmid="$1" @@ -1293,9 +1407,11 @@ pve_vm_exists() { pve_vm_ip() { local vmid="$1" + # Method 1: Try QEMU guest agent (works if agent is installed) + local ip="" case "$PVE_METHOD" in ssh) - pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \ + ip=$(pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \ python3 -c " import json, sys try: @@ -1309,10 +1425,10 @@ try: sys.exit(0) except: pass -" 2>/dev/null || true +" 2>/dev/null) || true ;; api) - pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \ + ip=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \ python3 -c " import json, sys try: @@ -1326,9 +1442,69 @@ try: sys.exit(0) except: pass -" 2>/dev/null || true +" 2>/dev/null) || true ;; esac + + if [[ -n "$ip" ]]; then + echo "$ip" + return 0 + fi + + # Method 2: ARP-based lookup using VM's MAC address. + # Works without guest agent (e.g. IncusOS which is immutable). + # Get MAC from VM config, flush stale ARP, ping sweep, look up. + local mac="" + case "$PVE_METHOD" in + ssh) + mac=$(pve_run "qm config ${vmid}" 2>/dev/null | \ + awk -F'=' '/^net0:/{print $1}' | grep -oiE '[0-9a-f:]{17}') || true + ;; + api) + mac=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null | \ + python3 -c " +import json, sys, re +try: + cfg = json.load(sys.stdin).get('data', {}) + net0 = cfg.get('net0', '') + m = re.search(r'([0-9A-Fa-f:]{17})', net0) + if m: + print(m.group(1).lower()) +except: + pass +" 2>/dev/null) || true + ;; + esac + + if [[ -z "$mac" ]]; then + return 1 + fi + + # Flush stale ARP entries for this MAC, ping broadcast to populate, then look up + ip neigh flush nud stale 2>/dev/null || true + local bridge_subnet + bridge_subnet=$(ip -4 addr show | awk '/inet /{print $2}' | head -1) + if [[ -n "$bridge_subnet" ]]; then + local broadcast + broadcast=$(python3 -c "import ipaddress; print(ipaddress.ip_interface('${bridge_subnet}').network.broadcast_address)" 2>/dev/null) || true + if [[ -n "$broadcast" ]]; then + ping -c 2 -W 1 -b "$broadcast" &>/dev/null || true + sleep 1 + fi + fi + + # Look up MAC in ARP table (only REACHABLE or DELAY, not STALE) + ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/ || /STALE/) {print $1; exit}') || true + + if [[ -n "$ip" ]]; then + # Verify it's actually reachable + if ping -c 1 -W 1 "$ip" &>/dev/null; then + echo "$ip" + return 0 + fi + fi + + return 1 } # --------------------------------------------------------------------------- @@ -1416,9 +1592,10 @@ preflight_checks() { fi ;; api) + local storage_json + storage_json=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null) || storage_json="" local storage_list - storage_list=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null | \ - python3 -c " + storage_list=$(echo "$storage_json" | python3 -c " import json, sys data = json.load(sys.stdin) for s in data.get('data', []): @@ -1435,6 +1612,22 @@ for s in data.get('data', []): error "Available pools: $(echo "$storage_list" | tr '\n' ' ')" error "Fix: update 'proxmox.iso_storage' in your config file" errors=$((errors + 1)) + else + # Check that ISO storage accepts 'iso' content type + local iso_content + iso_content=$(echo "$storage_json" | python3 -c " +import json, sys +data = json.load(sys.stdin) +for s in data.get('data', []): + if s.get('storage') == '${PVE_ISO_STORAGE}': + print(s.get('content', '')) + break +" 2>/dev/null) || iso_content="" + if [[ -n "$iso_content" ]] && ! echo "$iso_content" | grep -q "iso"; then + error "Storage '${PVE_ISO_STORAGE}' does not accept ISO content (content types: ${iso_content})" + error "Fix: enable 'iso' content on '${PVE_ISO_STORAGE}' in Proxmox, or use a different iso_storage" + errors=$((errors + 1)) + fi fi ;; esac @@ -1475,15 +1668,10 @@ for s in data.get('data', []): fi fi - # 4. FAT tools for SEED_DATA generation - if ! command -v mcopy &>/dev/null; then - error "mcopy (mtools) is required for SEED_DATA generation but not found" - error "Install mtools for your platform (e.g. mtools package)" - errors=$((errors + 1)) - fi - if ! command -v mkfs.fat &>/dev/null; then - error "mkfs.fat (dosfstools) is required for SEED_DATA generation but not found" - error "Install dosfstools for your platform (e.g. dosfstools package)" + # 4. genisoimage for SEED_DATA ISO 9660 generation + if ! command -v genisoimage &>/dev/null; then + error "genisoimage is required for SEED_DATA generation but not found" + error "Install with: sudo apt install genisoimage" errors=$((errors + 1)) fi @@ -1506,12 +1694,6 @@ for s in data.get('data', []): check_connectivity() { step "Checking Proxmox connectivity (${PVE_METHOD})" - if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would check connectivity to ${PVE_HOST}" - success "Connectivity check skipped (dry-run)" - return 0 - fi - case "$PVE_METHOD" in ssh) if ! ssh -o BatchMode=yes -o ConnectTimeout=10 \ @@ -1529,8 +1711,10 @@ check_connectivity() { success "Connected to ${PVE_HOST} (Proxmox VE ${pve_version})" ;; api) + local url="https://${PVE_HOST}:8006/api2/json/version" + local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" local response - if ! response=$(pve_api GET "/version" 2>/dev/null); then + if ! response=$(curl -fsSk -H "$auth" "$url" 2>/dev/null); then error "Cannot connect to Proxmox API at https://${PVE_HOST}:8006" error "Check that:" error " 1. The host is reachable" @@ -1750,12 +1934,23 @@ if versions: download_iso() { local version="$1" local output_path="$2" + local iso_filename + iso_filename=$(basename "$output_path") + + # Check persistent cache first + local cached_iso="${CACHEDIR}/${iso_filename}" + if [[ -f "$cached_iso" ]]; then + success "ISO found in local cache: ${cached_iso}" + cp "$cached_iso" "$output_path" + return 0 + fi local url="${CDN_BASE}/${version}/x86_64/IncusOS_${version}.iso.gz" - local gz_path="${output_path}.gz" + local gz_path="${cached_iso}.gz" step "Downloading IncusOS ISO" detail "URL: ${url}" + detail "Cache directory: ${CACHEDIR}" if ! curl -fL --progress-bar "$url" -o "$gz_path"; then error "Failed to download ISO from ${url}" @@ -1765,12 +1960,19 @@ download_iso() { step "Decompressing ISO" if ! gzip -d "$gz_path"; then - error "Failed to decompress ISO" + local avail + avail=$(df -h --output=avail "$(dirname "$gz_path")" 2>/dev/null | tail -1 | tr -d ' ') || avail="unknown" + error "Failed to decompress ISO (${avail} available on disk)" + error "The decompressed ISO requires several GiB of disk space" + error "Free up space, or set TMPDIR to a larger filesystem:" + error " TMPDIR=/path/with/space ${SCRIPT_NAME} ${CONFIG_FILE}" rm -f "$gz_path" exit 1 fi - success "ISO ready: ${output_path}" + # Copy from cache to workdir + cp "$cached_iso" "$output_path" + success "ISO ready: ${output_path} (cached for future runs)" } # --------------------------------------------------------------------------- @@ -1904,16 +2106,15 @@ phase_prepare() { local app="${VM_APPS[$i]}" local apply_defaults="${VM_APPLY_DEFAULTS[$i]}" - local seed_output="${WORKDIR}/seed-${name}.img" + local seed_output="${WORKDIR}/seed-${name}.iso" detail "Generating seed for '${name}' (${app}, defaults=${apply_defaults})" # Build incusos-seed command local seed_cmd=("$SEED_TOOL") - seed_cmd+=(--format fat) + seed_cmd+=(--format iso) seed_cmd+=(--app "$app") seed_cmd+=(--output "$seed_output") - seed_cmd+=(--disk-target "scsi-*") seed_cmd+=(--force-install) seed_cmd+=(--force-reboot) seed_cmd+=(--hostname "$name") @@ -1980,7 +2181,7 @@ phase_upload() { continue fi - local seed_file="seed-${name}.img" + local seed_file="seed-${name}.iso" local seed_path="${WORKDIR}/${seed_file}" if [[ -f "$seed_path" ]] || [[ "$DRY_RUN" == true ]]; then @@ -2020,7 +2221,7 @@ phase_create() { local cores="${VM_CORES[$i]}" local memory="${VM_MEMORY[$i]}" local disk="${VM_DISK[$i]}" - local seed_file="seed-${name}.img" + local seed_file="seed-${name}.iso" step "Creating VM '${name}' (VMID ${vmid})" @@ -2093,27 +2294,30 @@ phase_install() { fi success "VM '${name}' started" - # 2. Immediately swap boot order (takes effect on next reboot) - detail "Swapping boot order to disk-first" - if ! pve_set_vm "$vmid" "--boot order=scsi0\\;ide2"; then - warn "Failed to swap boot order for '${name}' -- manual intervention may be needed" - fi - - # 3. Wait for installation + reboot cycle + # 2. Wait for installation to complete by monitoring disk writes + # + # Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot + # (force_reboot in seed). A guest-level reboot does NOT reset the QEMU + # VM uptime, so we cannot detect reboot that way. Instead we monitor + # scsi0 write bytes via blockstat: + # - During install: wr_bytes increases steadily + # - After install + reboot: wr_bytes stops (VM is idle at post-reboot + # screen or "install media detected" error) + # + # When wr_bytes has been stable for STABLE_COUNT consecutive polls, + # installation is complete. We then stop the VM, remove install media, + # and start cleanly from disk. step "Waiting for '${name}' to complete installation" - detail "IncusOS installs from ISO, then reboots to disk automatically" - detail "Polling every 15s (timeout: 10 minutes)" + detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)" local timeout=600 - local interval=15 + local interval=5 local elapsed=0 - local booted=false - - # Wait for the VM to go through: running -> (install) -> stopped -> running - # The force_reboot in seed causes an automatic reboot after install. - # We detect the reboot by looking for the VM to be "running" after - # having been "stopped" (or we just wait for it to stabilize). - local saw_stop=false + local install_done=false + local prev_wr_bytes=0 + local writes_started=false + local stable_count=0 + local STABLE_THRESHOLD=3 # 3 consecutive polls with no change = 15s idle while [[ $elapsed -lt $timeout ]]; do sleep "$interval" @@ -2124,28 +2328,35 @@ phase_install() { case "$status" in stopped) - if [[ "$saw_stop" == false ]]; then - detail "VM stopped (installation complete, awaiting reboot) [${elapsed}s]" - saw_stop=true - # The VM should auto-reboot via force_reboot, but if it - # doesn't after 30s, start it manually - sleep 30 - elapsed=$((elapsed + 30)) - local recheck - recheck=$(pve_vm_status "$vmid") || recheck="unknown" - if [[ "$recheck" == "stopped" ]]; then - detail "VM did not auto-reboot, starting manually" - pve_start_vm "$vmid" || true - fi - fi + detail "VM stopped [${elapsed}s]" + install_done=true + break ;; running) - if [[ "$saw_stop" == true ]] || [[ $elapsed -ge 120 ]]; then - # VM has been running long enough post-reboot -- likely booted from disk - booted=true - break + local wr_bytes + wr_bytes=$(pve_vm_disk_writes "$vmid") || wr_bytes=0 + + if [[ $wr_bytes -gt $prev_wr_bytes ]]; then + # Disk writes are happening -- install in progress + writes_started=true + stable_count=0 + local wr_mb=$(( wr_bytes / 1048576 )) + # Show progress every 15s + if [[ $((elapsed % 15)) -eq 0 ]]; then + detail "Installing... (${wr_mb} MiB written) [${elapsed}s]" + fi + elif [[ "$writes_started" == true ]]; then + # Writes have stopped after being active + stable_count=$((stable_count + 1)) + if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then + local wr_mb=$(( wr_bytes / 1048576 )) + detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]" + install_done=true + break + fi fi - detail "VM running (installing...) [${elapsed}s]" + + prev_wr_bytes=$wr_bytes ;; *) detail "VM status: ${status} [${elapsed}s]" @@ -2153,20 +2364,39 @@ phase_install() { esac done - if [[ "$booted" != true ]]; then + if [[ "$install_done" != true ]]; then warn "VM '${name}' did not complete installation within ${timeout}s" warn "Check the VM console on Proxmox for details" i=$((i + 1)) continue fi - # 4. Give it a moment to fully initialize - detail "Waiting for services to start..." - sleep 15 + # 3. Stop the VM (it's either at "remove install media" prompt or + # post-reboot "install media detected" error -- either way idle). + local cur_status + cur_status=$(pve_vm_status "$vmid") || cur_status="unknown" + if [[ "$cur_status" == "running" ]]; then + detail "Stopping VM to remove install media..." + pve_stop_vm "$vmid" || true + sleep 5 + fi - # 5. Remove CD-ROM drives - step "Cleaning up '${name}'" - pve_set_vm "$vmid" "--delete ide2 --delete ide3" || true + # 4. Remove install media and set clean boot order + step "Removing install media from '${name}'" + pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true + success "Install media detached, boot order set to disk-only" + + # 5. Start VM from disk + step "Starting '${name}' from disk" + if ! pve_start_vm "$vmid"; then + error "Failed to start VM '${name}' from disk" + i=$((i + 1)) + continue + fi + + # 5. Wait for services to come up + detail "Waiting for services to start..." + sleep 30 # 6. Try to detect IP local ip diff --git a/incusos/incusos-seed b/incusos/incusos-seed index 7d99148..10d3d2e 100755 --- a/incusos/incusos-seed +++ b/incusos/incusos-seed @@ -76,8 +76,9 @@ ${BOLD}OPTIONS${RESET} ${BOLD}--auto-reboot${RESET} Enable automatic reboot for updates ${BOLD}Output options:${RESET} - ${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}fat${RESET} (default: tar) + ${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}iso${RESET}, ${CYAN}fat${RESET} (default: tar) tar: for use with flasher-tool --seed + iso: ISO 9660 image with SEED_DATA label for CD-ROM fat: FAT image with SEED_DATA label for external media ${BOLD}--fat-size${RESET} SIZE FAT image size in MiB (default: 10) @@ -97,6 +98,9 @@ ${BOLD}EXAMPLES${RESET} # Cluster node (no defaults) with forced install on virtio disk ${SCRIPT_NAME} --disk-target 'virtio-*' --force-install --force-reboot + # Create an ISO 9660 image for CD-ROM seed delivery (e.g. Proxmox VMs) + ${SCRIPT_NAME} --format iso --defaults --output seed.iso + # Create a FAT image for SEED_DATA partition ${SCRIPT_NAME} --format fat --defaults --output seed.img @@ -257,10 +261,10 @@ validate_args() { esac case "$FORMAT" in - tar|fat) ;; + tar|iso|fat) ;; *) error "Invalid output format: ${FORMAT}" - error "Supported: tar, fat" + error "Supported: tar, iso, fat" exit 1 ;; esac @@ -284,8 +288,21 @@ validate_args() { exit 1 fi + if [[ "$FORMAT" == "iso" ]]; then + if ! command -v genisoimage &>/dev/null; then + error "ISO 9660 image creation requires genisoimage but it was not found" + error "Install with: sudo apt install genisoimage" + exit 1 + fi + fi + if [[ "$FORMAT" == "fat" ]]; then - if ! command -v mcopy &>/dev/null && ! command -v mkfs.fat &>/dev/null; then + # mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users + local has_mkfs_fat=false + if command -v mkfs.fat &>/dev/null || [[ -x /usr/sbin/mkfs.fat ]] || [[ -x /sbin/mkfs.fat ]]; then + has_mkfs_fat=true + fi + if ! command -v mcopy &>/dev/null && [[ "$has_mkfs_fat" != true ]]; then error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)" error "Install with: sudo apt install mtools dosfstools" exit 1 @@ -296,6 +313,7 @@ validate_args() { if [[ -z "$OUTPUT" ]]; then case "$FORMAT" in tar) OUTPUT="incusos-seed.tar" ;; + iso) OUTPUT="incusos-seed.iso" ;; fat) OUTPUT="incusos-seed.img" ;; esac fi @@ -464,14 +482,13 @@ generate_incus_yaml() { if [[ -n "$CERT_CONTENT" ]]; then echo "preseed:" - echo " server:" - echo " certificates:" - echo " - name: \"auto-injected-client\"" - echo " type: \"client\"" - echo " certificate: |" + echo " certificates:" + echo " - name: \"auto-injected-client\"" + echo " type: \"client\"" + echo " certificate: |-" # Indent each line of the certificate while IFS= read -r line; do - echo " ${line}" + echo " ${line}" done <<< "$CERT_CONTENT" fi } > "$file" @@ -592,14 +609,34 @@ create_tar_output() { success "Seed archive created: ${OUTPUT} (${size})" } +create_iso_output() { + step "Creating ISO 9660 image: ${OUTPUT}" + + genisoimage -o "$OUTPUT" -V SEED_DATA -J -r "$WORKDIR"/ 2>/dev/null + + local size + size=$(du -h "$OUTPUT" | cut -f1) + success "ISO 9660 seed image created: ${OUTPUT} (${size})" +} + create_fat_output() { step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)" # Create empty image dd if=/dev/zero of="$OUTPUT" bs=1M count="$FAT_SIZE" status=none + # mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users + local mkfs_fat="" if command -v mkfs.fat &>/dev/null; then - mkfs.fat -n SEED_DATA "$OUTPUT" >/dev/null + mkfs_fat="mkfs.fat" + elif [[ -x /usr/sbin/mkfs.fat ]]; then + mkfs_fat="/usr/sbin/mkfs.fat" + elif [[ -x /sbin/mkfs.fat ]]; then + mkfs_fat="/sbin/mkfs.fat" + fi + + if [[ -n "$mkfs_fat" ]]; then + "$mkfs_fat" -n SEED_DATA "$OUTPUT" >/dev/null if command -v mcopy &>/dev/null; then # Use mtools to copy files without mounting @@ -663,6 +700,9 @@ print_summary() { if [[ "$FORMAT" == "tar" ]]; then echo -e "${DIM} Use with flasher-tool:${RESET}" echo -e " flasher-tool --seed ${OUTPUT}" + elif [[ "$FORMAT" == "iso" ]]; then + echo -e "${DIM} Attach as CD-ROM to the VM.${RESET}" + echo -e "${DIM} IncusOS will detect the SEED_DATA volume label at boot.${RESET}" elif [[ "$FORMAT" == "fat" ]]; then echo -e "${DIM} Attach as secondary disk/ISO to the VM.${RESET}" echo -e "${DIM} IncusOS will detect the SEED_DATA label at boot.${RESET}" @@ -718,6 +758,7 @@ main() { else case "$FORMAT" in tar) create_tar_output ;; + iso) create_iso_output ;; fat) create_fat_output ;; esac diff --git a/notes/proxmox-deploy-progress.md b/notes/proxmox-deploy-progress.md new file mode 100644 index 0000000..c8f13ba --- /dev/null +++ b/notes/proxmox-deploy-progress.md @@ -0,0 +1,148 @@ +# IncusOS Proxmox Deploy — Session Progress + +_Last updated: 2026-02-19_ + +## Status: Complete — end-to-end automated deployment working + +Full automated deployment of IncusOS to Proxmox VE via API is working. +Tested with a single standalone Incus VM (VMID 900) — seed generated, +ISO uploaded, VM created with correct settings, installation detected via +disk I/O monitoring, media ejected, booted from disk, Incus API accessible, +client certificate pre-trusted. + +## Proxmox environment + +| Item | Value | +|----------------------|-------------------------------------------| +| Host | 192.168.1.29 | +| Node name | pve | +| Proxmox VE version | 9.1.5 | +| API token ID | `automation@pve!deploy` | +| Token secret env var | `PROXMOX_TOKEN_SECRET` | +| ISO storage | `local` (directory, supports iso content) | +| VM disk storage | `local-zfs` (ZFS pool, images/rootdir) | +| Test config | `incusos/examples/my-test.yaml` | +| Test VM ID | 900 | + +## Bugs found and fixed + +### 1. FAT labels not detected on CD-ROM devices + +**Symptom:** IncusOS reports "unable to begin install from read-only device +/dev/mapper/sr0 without seed configuration" + +**Root cause:** Linux kernel's `sr_mod` driver does NOT expose FAT filesystem +labels for `/dev/sr*` devices. The `SEED_DATA` label is never created in +`/dev/disk/by-label/`. + +**Fix:** Switched from FAT to ISO 9660 format. Added `--format iso` to +`incusos-seed` using `genisoimage -V SEED_DATA -J -r`. Changed +`incusos-proxmox` to use `--format iso` and `.iso` extension. + +### 2. Disk target literal matching (not glob) + +**Symptom:** "no target device matched 'scsi-*' (detected devices: +scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)" + +**Root cause:** IncusOS does literal string matching on disk target IDs, +not glob matching. `scsi-*` is not a valid device ID. + +**Fix:** Removed `--disk-target "scsi-*"` from seed generation in +`incusos-proxmox`. IncusOS auto-detects when there's a single disk. + +### 3. Preseed YAML structure (Go `yaml:",inline"`) + +**Symptom:** "yaml: unmarshal errors: line 4 field server not found in +type api.initpreseed" + +**Root cause:** The Incus `InitPreseed` struct has +`Server InitLocalPreseed` with `yaml:",inline"`, which promotes the +Server's fields (including `certificates`) to the parent level. The +correct path is `preseed.certificates[]`, NOT `preseed.server.certificates[]`. + +**Fix:** Updated `generate_incus_yaml()` in `incusos-seed` and all +example files. + +### 4. Install media detection blocking boot + +**Symptom:** After installation and reboot, IncusOS refuses to boot with +"install media detected" error, regardless of boot order. + +**Root cause:** IncusOS checks for the _presence_ of install media at +every boot. If IDE CD-ROMs are still attached, it blocks boot even when +disk is first in boot order. + +**Fix:** After detecting install completion, the script now: stops the VM, +deletes ide2 and ide3 devices completely, sets boot order to `order=scsi0`, +then starts from disk. + +### 5. Install completion detection + +**Symptom:** Cannot reliably detect when IncusOS finishes installing. +Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start +does), making uptime-based detection impossible. + +**Root cause:** `force_reboot: true` in the seed triggers a guest OS reboot, +but QEMU treats this as a guest-initiated reset — the VM process continues, +uptime keeps incrementing. + +**Fix:** Poll `blockstat.scsi0.wr_bytes` via Proxmox API every 5 seconds. +During installation, disk writes increase (~876 MiB total). After install + +reboot, writes stop. Three consecutive stable readings (15 seconds of no +writes) = install done. Works reliably. + +### 6. force_reboot is required for automation + +**Symptom:** Without `force_reboot: true`, IncusOS sits at "please remove +installation media to complete the installation" and waits indefinitely. + +**Root cause:** IncusOS does not auto-halt after installation. It expects +the user to remove media and manually reboot. + +**Fix:** Always include `force_reboot: true` in the seed for automated +deployments. + +### 7. IP detection without guest agent + +**Symptom:** IncusOS is immutable with no QEMU guest agent. Standard +Proxmox guest agent IP detection returns nothing. + +**Fix:** ARP-based detection: extract MAC from Proxmox VM config (net0 +field), flush stale ARP entries, ping broadcast to populate ARP table, +look up MAC in ARP output. Verify with direct ping before returning. + +### 8. Various API-related fixes (earlier session) + +- **ISO upload HTTP 400**: Upload field must be named `filename` (not `file`) +- **ISO caching**: Added persistent cache at `~/.cache/incusos-proxmox/iso-cache/` +- **VM creation HTTP 400** ("duplicate key"): Use JSON Content-Type, not form-encoded +- **Boot order swap HTTP 400**: Convert CLI flags to JSON with Python helper +- **API response leak**: Suppress JSON responses from mutating API calls + +## Verified deployment flow + +1. Seed generated as ISO 9660 (372K) with `genisoimage` +2. IncusOS ISO reused from Proxmox cache (already uploaded) +3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.) +4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3 +5. blockstat detects install completion at ~876 MiB written, ~50s elapsed +6. VM stopped, ide2 + ide3 deleted, boot order set to `order=scsi0` +7. VM started from disk — IncusOS boots successfully +8. Port 8443 open, Incus API responding +9. `incus remote add` works without auth prompt (certificate pre-trusted) +10. Storage pool (`local`, ZFS) and network bridge (`incusbr0`) confirmed + +## Existing VMs on Proxmox + +- 100: dev-vm-beelink +- 201: IncusOS-01 +- 202: IncusOS-02 +- 203: IncusOS-03 +- 300: IncusOS-OC +- 900: incus-test (deployed and running) + +## Next steps + +- Multi-VM deployment testing (full lab with ops-center + cluster) +- Cluster formation automation (`incus cluster add` / join tokens) +- SSH method testing (currently only API method verified)