maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/.claude/rules/proxmox-deployment.md

3.8 KiB
Raw Permalink Blame History

paths
bin/incusos-proxmox
bin/diagnostic/observe-deploy
bin/helpers/*
bin/examples/*
bin/TESTING.md

Proxmox VE Deployment

incusos-proxmox overview

  • Reads YAML config, generates per-VM SEED_DATA via incusos-seed --format iso, uploads ISO + seeds to Proxmox, creates VMs, boots through installation.
  • Config merge: proxmox.yaml base -> lab config proxmox: overlay -> CLI flags. Auto-discovers proxmox.yaml in script dir then cwd; override with --proxmox FILE.
  • API token: env file at repo root (gitignored). Auto-loaded on startup.
  • Methods: SSH (default) or API (curl -k https://host:8006/api2/json/...).

Required VM settings (wrong = install failure)

  • bios=ovmf, machine=q35 -- UEFI required
  • efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls own Secure Boot keys
  • tpmstate0: version=v2.0 -- required for disk encryption
  • cpu=host -- x86_64_v3 instruction set requirement
  • scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk broken with IncusOS
  • balloon=0 -- IncusOS manages memory internally
  • ide3 -- SEED_DATA ISO 9660 as second CD-ROM
  • Minimum 50 GiB disk, 4096 MiB RAM

Install flow (automated)

  1. Boot VM with ISO (ide2) + SEED_DATA (ide3), no force_reboot in seed
  2. IncusOS installs to scsi0 (~876 MiB), sits at "remove media" prompt
  3. Detect completion: poll blockstat.scsi0.wr_bytes -- writes start then stop
  4. Stop VM (Proxmox stop)
  5. Delete ide2 and ide3 -- IncusOS refuses to start if media present
  6. Set boot: order=scsi0, start from disk

IP detection

  • Static IP (preferred): ip: ADDR/PREFIX per VM + gateway: + dns:.
  • ARP-based (DHCP fallback): MAC from VM config -> ARP table. Same L2 only.

VLAN tagging

  • vlan: config key adds tag to NIC (tag=69). Current lab: VLAN 69 (192.168.100.0/22).

Disk target

  • Do NOT specify disk-target for Proxmox VMs. IncusOS does literal string matching. scsi-* does NOT match the actual device ID. Omit and let auto-detect.

Resource pool isolation

  • proxmox.pool scopes VM operations to a Proxmox pool. API token ACL on /pool/<name>.

Reconcile and idempotency

  • Re-runs with existing VMs: interactive menu (status/continue/destroy/abort).
  • --yes defaults to status checks (never auto-destroys).
  • phase_install checks each VM state before acting.

Doctor and cleanup

  • --doctor: environment check, no config needed.
  • --cleanup: destroys config VMs. --cleanup --deep: also ISOs + remotes.
  • --cleanup-all: pool-wide. --cleanup-all --deep: all IncusOS ISOs.
  • --verbose/-v, --quiet, --retries N (default 3).

Boot timeout

  • 180s (60s sleep + 120s poll). First boot downloads sysext (~30-120s). Do NOT reduce -- premature retries can corrupt TPM encryption key permanently.

Disk resize

  • Proxmox qm resize only grows. Partition 11 auto-expands via systemd-repart.
  • ZFS pool does NOT auto-expand. Manual expansion via privileged container needed. See notes/aether-guide.md for procedure.

First-boot sequence

  1. ISO boot + install (~60-85s)
  2. Transition: detect complete, stop, remove media, start (~15s)
  3. First disk boot: encryption key, sysext download, "System ready" (~50s)
  4. Total: ~130-215s to port 8443

Crontab bug (issue #843)

  • Eliminated by omitting force_reboot. Upstream bug only with force_reboot: true.
  • Empty ScrubSchedule crashes daemon. fix_scrub_schedule() auto-heals.
  • TPM corruption risk: hard-stop during first boot permanently corrupts. 180s timeout prevents this.

API privileges (role: IncusOSDeployer)

VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use Sys.Audit