# IncusOS Proxmox Deploy — Session Progress

_Last updated: 2026-02-19_

## Status: Complete — end-to-end automated deployment working

Full automated deployment of IncusOS to Proxmox VE via API is working.
Tested with a single standalone Incus VM (VMID 900) — seed generated,
ISO uploaded, VM created with correct settings, installation detected via
disk I/O monitoring, media ejected, booted from disk, Incus API accessible,
client certificate pre-trusted.

## Proxmox environment

| Item                 | Value                                     |
|----------------------|-------------------------------------------|
| Host                 | 192.168.1.29                              |
| Node name            | pve                                       |
| Proxmox VE version   | 9.1.5                                     |
| API token ID         | `automation@pve!deploy`                   |
| Token secret env var | `PROXMOX_TOKEN_SECRET`                    |
| ISO storage          | `local` (directory, supports iso content) |
| VM disk storage      | `local-zfs` (ZFS pool, images/rootdir)    |
| Test config          | `incusos/examples/my-test.yaml`           |
| Test VM ID           | 900                                       |

## Bugs found and fixed

### 1. FAT labels not detected on CD-ROM devices

**Symptom:** IncusOS reports "unable to begin install from read-only device
/dev/mapper/sr0 without seed configuration"

**Root cause:** Linux kernel's `sr_mod` driver does NOT expose FAT filesystem
labels for `/dev/sr*` devices. The `SEED_DATA` label is never created in
`/dev/disk/by-label/`.

**Fix:** Switched from FAT to ISO 9660 format. Added `--format iso` to
`incusos-seed` using `genisoimage -V SEED_DATA -J -r`. Changed
`incusos-proxmox` to use `--format iso` and `.iso` extension.

### 2. Disk target literal matching (not glob)

**Symptom:** "no target device matched 'scsi-*' (detected devices:
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)"

**Root cause:** IncusOS does literal string matching on disk target IDs,
not glob matching. `scsi-*` is not a valid device ID.

**Fix:** Removed `--disk-target "scsi-*"` from seed generation in
`incusos-proxmox`. IncusOS auto-detects when there's a single disk.

### 3. Preseed YAML structure (Go `yaml:",inline"`)

**Symptom:** "yaml: unmarshal errors: line 4 field server not found in
type api.initpreseed"

**Root cause:** The Incus `InitPreseed` struct has
`Server InitLocalPreseed` with `yaml:",inline"`, which promotes the
Server's fields (including `certificates`) to the parent level. The
correct path is `preseed.certificates[]`, NOT `preseed.server.certificates[]`.

**Fix:** Updated `generate_incus_yaml()` in `incusos-seed` and all
example files.

### 4. Install media detection blocking boot

**Symptom:** After installation and reboot, IncusOS refuses to boot with
"install media detected" error, regardless of boot order.

**Root cause:** IncusOS checks for the _presence_ of install media at
every boot. If IDE CD-ROMs are still attached, it blocks boot even when
disk is first in boot order.

**Fix:** After detecting install completion, the script now: stops the VM,
deletes ide2 and ide3 devices completely, sets boot order to `order=scsi0`,
then starts from disk.

### 5. Install completion detection

**Symptom:** Cannot reliably detect when IncusOS finishes installing.
Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start
does), making uptime-based detection impossible.

**Root cause:** `force_reboot: true` in the seed triggers a guest OS reboot,
but QEMU treats this as a guest-initiated reset — the VM process continues,
uptime keeps incrementing.

**Fix:** Poll `blockstat.scsi0.wr_bytes` via Proxmox API every 5 seconds.
During installation, disk writes increase (~876 MiB total). After install +
reboot, writes stop. Three consecutive stable readings (15 seconds of no
writes) = install done. Works reliably.

### 6. force_reboot is required for automation

**Symptom:** Without `force_reboot: true`, IncusOS sits at "please remove
installation media to complete the installation" and waits indefinitely.

**Root cause:** IncusOS does not auto-halt after installation. It expects
the user to remove media and manually reboot.

**Fix:** Always include `force_reboot: true` in the seed for automated
deployments.

### 7. IP detection without guest agent

**Symptom:** IncusOS is immutable with no QEMU guest agent. Standard
Proxmox guest agent IP detection returns nothing.

**Fix:** ARP-based detection: extract MAC from Proxmox VM config (net0
field), flush stale ARP entries, ping broadcast to populate ARP table,
look up MAC in ARP output. Verify with direct ping before returning.

### 8. Various API-related fixes (earlier session)

- **ISO upload HTTP 400**: Upload field must be named `filename` (not `file`)
- **ISO caching**: Added persistent cache at `~/.cache/incusos-proxmox/iso-cache/`
- **VM creation HTTP 400** ("duplicate key"): Use JSON Content-Type, not form-encoded
- **Boot order swap HTTP 400**: Convert CLI flags to JSON with Python helper
- **API response leak**: Suppress JSON responses from mutating API calls

## Verified deployment flow

1. Seed generated as ISO 9660 (372K) with `genisoimage`
2. IncusOS ISO reused from Proxmox cache (already uploaded)
3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.)
4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3
5. blockstat detects install completion at ~876 MiB written, ~50s elapsed
6. VM stopped, ide2 + ide3 deleted, boot order set to `order=scsi0`
7. VM started from disk — IncusOS boots successfully
8. Port 8443 open, Incus API responding
9. `incus remote add` works without auth prompt (certificate pre-trusted)
10. Storage pool (`local`, ZFS) and network bridge (`incusbr0`) confirmed

## Existing VMs on Proxmox

- 100: dev-vm-beelink
- 201: IncusOS-01
- 202: IncusOS-02
- 203: IncusOS-03
- 300: IncusOS-OC
- 900: incus-test (deployed and running)

## Next steps

- Multi-VM deployment testing (full lab with ops-center + cluster)
- Cluster formation automation (`incus cluster add` / join tokens)
- SSH method testing (currently only API method verified)