maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/proxmox-deploy-progress.md

6.1 KiB
Raw Blame History

IncusOS Proxmox Deploy — Session Progress

Last updated: 2026-02-19

Status: Complete — end-to-end automated deployment working

Full automated deployment of IncusOS to Proxmox VE via API is working. Tested with a single standalone Incus VM (VMID 900) — seed generated, ISO uploaded, VM created with correct settings, installation detected via disk I/O monitoring, media ejected, booted from disk, Incus API accessible, client certificate pre-trusted.

Proxmox environment

Item Value
Host 192.168.1.29
Node name pve
Proxmox VE version 9.1.5
API token ID automation@pve!deploy
Token secret env var PROXMOX_TOKEN_SECRET
ISO storage local (directory, supports iso content)
VM disk storage local-zfs (ZFS pool, images/rootdir)
Test config incusos/examples/my-test.yaml
Test VM ID 900

Bugs found and fixed

1. FAT labels not detected on CD-ROM devices

Symptom: IncusOS reports "unable to begin install from read-only device /dev/mapper/sr0 without seed configuration"

Root cause: Linux kernel's sr_mod driver does NOT expose FAT filesystem labels for /dev/sr* devices. The SEED_DATA label is never created in /dev/disk/by-label/.

Fix: Switched from FAT to ISO 9660 format. Added --format iso to incusos-seed using genisoimage -V SEED_DATA -J -r. Changed incusos-proxmox to use --format iso and .iso extension.

2. Disk target literal matching (not glob)

Symptom: "no target device matched 'scsi-*' (detected devices: scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)"

Root cause: IncusOS does literal string matching on disk target IDs, not glob matching. scsi-* is not a valid device ID.

Fix: Removed --disk-target "scsi-*" from seed generation in incusos-proxmox. IncusOS auto-detects when there's a single disk.

3. Preseed YAML structure (Go yaml:",inline")

Symptom: "yaml: unmarshal errors: line 4 field server not found in type api.initpreseed"

Root cause: The Incus InitPreseed struct has Server InitLocalPreseed with yaml:",inline", which promotes the Server's fields (including certificates) to the parent level. The correct path is preseed.certificates[], NOT preseed.server.certificates[].

Fix: Updated generate_incus_yaml() in incusos-seed and all example files.

4. Install media detection blocking boot

Symptom: After installation and reboot, IncusOS refuses to boot with "install media detected" error, regardless of boot order.

Root cause: IncusOS checks for the presence of install media at every boot. If IDE CD-ROMs are still attached, it blocks boot even when disk is first in boot order.

Fix: After detecting install completion, the script now: stops the VM, deletes ide2 and ide3 devices completely, sets boot order to order=scsi0, then starts from disk.

5. Install completion detection

Symptom: Cannot reliably detect when IncusOS finishes installing. Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start does), making uptime-based detection impossible.

Root cause: force_reboot: true in the seed triggers a guest OS reboot, but QEMU treats this as a guest-initiated reset — the VM process continues, uptime keeps incrementing.

Fix: Poll blockstat.scsi0.wr_bytes via Proxmox API every 5 seconds. During installation, disk writes increase (~876 MiB total). After install + reboot, writes stop. Three consecutive stable readings (15 seconds of no writes) = install done. Works reliably.

6. force_reboot is required for automation

Symptom: Without force_reboot: true, IncusOS sits at "please remove installation media to complete the installation" and waits indefinitely.

Root cause: IncusOS does not auto-halt after installation. It expects the user to remove media and manually reboot.

Fix: Always include force_reboot: true in the seed for automated deployments.

7. IP detection without guest agent

Symptom: IncusOS is immutable with no QEMU guest agent. Standard Proxmox guest agent IP detection returns nothing.

Fix: ARP-based detection: extract MAC from Proxmox VM config (net0 field), flush stale ARP entries, ping broadcast to populate ARP table, look up MAC in ARP output. Verify with direct ping before returning.

8. Various API-related fixes (earlier session)

  • ISO upload HTTP 400: Upload field must be named filename (not file)
  • ISO caching: Added persistent cache at ~/.cache/incusos-proxmox/iso-cache/
  • VM creation HTTP 400 ("duplicate key"): Use JSON Content-Type, not form-encoded
  • Boot order swap HTTP 400: Convert CLI flags to JSON with Python helper
  • API response leak: Suppress JSON responses from mutating API calls

Verified deployment flow

  1. Seed generated as ISO 9660 (372K) with genisoimage
  2. IncusOS ISO reused from Proxmox cache (already uploaded)
  3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.)
  4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3
  5. blockstat detects install completion at ~876 MiB written, ~50s elapsed
  6. VM stopped, ide2 + ide3 deleted, boot order set to order=scsi0
  7. VM started from disk — IncusOS boots successfully
  8. Port 8443 open, Incus API responding
  9. incus remote add works without auth prompt (certificate pre-trusted)
  10. Storage pool (local, ZFS) and network bridge (incusbr0) confirmed

Existing VMs on Proxmox

  • 100: dev-vm-beelink
  • 201: IncusOS-01
  • 202: IncusOS-02
  • 203: IncusOS-03
  • 300: IncusOS-OC
  • 900: incus-test (deployed and running)

Next steps

  • Multi-VM deployment testing (full lab with ops-center + cluster)
  • Cluster formation automation (incus cluster add / join tokens)
  • SSH method testing (currently only API method verified)