149 lines
6.1 KiB
Markdown
149 lines
6.1 KiB
Markdown
# IncusOS Proxmox Deploy — Session Progress
|
|
|
|
_Last updated: 2026-02-19_
|
|
|
|
## Status: Complete — end-to-end automated deployment working
|
|
|
|
Full automated deployment of IncusOS to Proxmox VE via API is working.
|
|
Tested with a single standalone Incus VM (VMID 900) — seed generated,
|
|
ISO uploaded, VM created with correct settings, installation detected via
|
|
disk I/O monitoring, media ejected, booted from disk, Incus API accessible,
|
|
client certificate pre-trusted.
|
|
|
|
## Proxmox environment
|
|
|
|
| Item | Value |
|
|
|----------------------|-------------------------------------------|
|
|
| Host | 192.168.1.29 |
|
|
| Node name | pve |
|
|
| Proxmox VE version | 9.1.5 |
|
|
| API token ID | `automation@pve!deploy` |
|
|
| Token secret env var | `PROXMOX_TOKEN_SECRET` |
|
|
| ISO storage | `local` (directory, supports iso content) |
|
|
| VM disk storage | `local-zfs` (ZFS pool, images/rootdir) |
|
|
| Test config | `incusos/examples/my-test.yaml` |
|
|
| Test VM ID | 900 |
|
|
|
|
## Bugs found and fixed
|
|
|
|
### 1. FAT labels not detected on CD-ROM devices
|
|
|
|
**Symptom:** IncusOS reports "unable to begin install from read-only device
|
|
/dev/mapper/sr0 without seed configuration"
|
|
|
|
**Root cause:** Linux kernel's `sr_mod` driver does NOT expose FAT filesystem
|
|
labels for `/dev/sr*` devices. The `SEED_DATA` label is never created in
|
|
`/dev/disk/by-label/`.
|
|
|
|
**Fix:** Switched from FAT to ISO 9660 format. Added `--format iso` to
|
|
`incusos-seed` using `genisoimage -V SEED_DATA -J -r`. Changed
|
|
`incusos-proxmox` to use `--format iso` and `.iso` extension.
|
|
|
|
### 2. Disk target literal matching (not glob)
|
|
|
|
**Symptom:** "no target device matched 'scsi-*' (detected devices:
|
|
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)"
|
|
|
|
**Root cause:** IncusOS does literal string matching on disk target IDs,
|
|
not glob matching. `scsi-*` is not a valid device ID.
|
|
|
|
**Fix:** Removed `--disk-target "scsi-*"` from seed generation in
|
|
`incusos-proxmox`. IncusOS auto-detects when there's a single disk.
|
|
|
|
### 3. Preseed YAML structure (Go `yaml:",inline"`)
|
|
|
|
**Symptom:** "yaml: unmarshal errors: line 4 field server not found in
|
|
type api.initpreseed"
|
|
|
|
**Root cause:** The Incus `InitPreseed` struct has
|
|
`Server InitLocalPreseed` with `yaml:",inline"`, which promotes the
|
|
Server's fields (including `certificates`) to the parent level. The
|
|
correct path is `preseed.certificates[]`, NOT `preseed.server.certificates[]`.
|
|
|
|
**Fix:** Updated `generate_incus_yaml()` in `incusos-seed` and all
|
|
example files.
|
|
|
|
### 4. Install media detection blocking boot
|
|
|
|
**Symptom:** After installation and reboot, IncusOS refuses to boot with
|
|
"install media detected" error, regardless of boot order.
|
|
|
|
**Root cause:** IncusOS checks for the _presence_ of install media at
|
|
every boot. If IDE CD-ROMs are still attached, it blocks boot even when
|
|
disk is first in boot order.
|
|
|
|
**Fix:** After detecting install completion, the script now: stops the VM,
|
|
deletes ide2 and ide3 devices completely, sets boot order to `order=scsi0`,
|
|
then starts from disk.
|
|
|
|
### 5. Install completion detection
|
|
|
|
**Symptom:** Cannot reliably detect when IncusOS finishes installing.
|
|
Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start
|
|
does), making uptime-based detection impossible.
|
|
|
|
**Root cause:** `force_reboot: true` in the seed triggers a guest OS reboot,
|
|
but QEMU treats this as a guest-initiated reset — the VM process continues,
|
|
uptime keeps incrementing.
|
|
|
|
**Fix:** Poll `blockstat.scsi0.wr_bytes` via Proxmox API every 5 seconds.
|
|
During installation, disk writes increase (~876 MiB total). After install +
|
|
reboot, writes stop. Three consecutive stable readings (15 seconds of no
|
|
writes) = install done. Works reliably.
|
|
|
|
### 6. force_reboot is required for automation
|
|
|
|
**Symptom:** Without `force_reboot: true`, IncusOS sits at "please remove
|
|
installation media to complete the installation" and waits indefinitely.
|
|
|
|
**Root cause:** IncusOS does not auto-halt after installation. It expects
|
|
the user to remove media and manually reboot.
|
|
|
|
**Fix:** Always include `force_reboot: true` in the seed for automated
|
|
deployments.
|
|
|
|
### 7. IP detection without guest agent
|
|
|
|
**Symptom:** IncusOS is immutable with no QEMU guest agent. Standard
|
|
Proxmox guest agent IP detection returns nothing.
|
|
|
|
**Fix:** ARP-based detection: extract MAC from Proxmox VM config (net0
|
|
field), flush stale ARP entries, ping broadcast to populate ARP table,
|
|
look up MAC in ARP output. Verify with direct ping before returning.
|
|
|
|
### 8. Various API-related fixes (earlier session)
|
|
|
|
- **ISO upload HTTP 400**: Upload field must be named `filename` (not `file`)
|
|
- **ISO caching**: Added persistent cache at `~/.cache/incusos-proxmox/iso-cache/`
|
|
- **VM creation HTTP 400** ("duplicate key"): Use JSON Content-Type, not form-encoded
|
|
- **Boot order swap HTTP 400**: Convert CLI flags to JSON with Python helper
|
|
- **API response leak**: Suppress JSON responses from mutating API calls
|
|
|
|
## Verified deployment flow
|
|
|
|
1. Seed generated as ISO 9660 (372K) with `genisoimage`
|
|
2. IncusOS ISO reused from Proxmox cache (already uploaded)
|
|
3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.)
|
|
4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3
|
|
5. blockstat detects install completion at ~876 MiB written, ~50s elapsed
|
|
6. VM stopped, ide2 + ide3 deleted, boot order set to `order=scsi0`
|
|
7. VM started from disk — IncusOS boots successfully
|
|
8. Port 8443 open, Incus API responding
|
|
9. `incus remote add` works without auth prompt (certificate pre-trusted)
|
|
10. Storage pool (`local`, ZFS) and network bridge (`incusbr0`) confirmed
|
|
|
|
## Existing VMs on Proxmox
|
|
|
|
- 100: dev-vm-beelink
|
|
- 201: IncusOS-01
|
|
- 202: IncusOS-02
|
|
- 203: IncusOS-03
|
|
- 300: IncusOS-OC
|
|
- 900: incus-test (deployed and running)
|
|
|
|
## Next steps
|
|
|
|
- Multi-VM deployment testing (full lab with ops-center + cluster)
|
|
- Cluster formation automation (`incus cluster add` / join tokens)
|
|
- SSH method testing (currently only API method verified)
|