---
paths:
  - "bin/incusos-proxmox"
  - "bin/diagnostic/observe-deploy"
  - "bin/helpers/*"
  - "bin/examples/*"
  - "bin/TESTING.md"
---

# Proxmox VE Deployment

## incusos-proxmox overview

- Reads YAML config, generates per-VM SEED_DATA via `incusos-seed --format iso`,
  uploads ISO + seeds to Proxmox, creates VMs, boots through installation.
- **Config merge**: `proxmox.yaml` base -> lab config `proxmox:` overlay -> CLI flags.
  Auto-discovers `proxmox.yaml` in script dir then cwd; override with `--proxmox FILE`.
- **API token**: `env` file at repo root (gitignored). Auto-loaded on startup.
- **Methods**: SSH (default) or API (`curl -k https://host:8006/api2/json/...`).

## Required VM settings (wrong = install failure)

- `bios=ovmf`, `machine=q35` -- UEFI required
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls own Secure Boot keys
- `tpmstate0`: `version=v2.0` -- required for disk encryption
- `cpu=host` -- x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally
- `ide3` -- SEED_DATA ISO 9660 as second CD-ROM
- Minimum 50 GiB disk, 4096 MiB RAM

## Install flow (automated)

1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
2. IncusOS installs to scsi0 (~876 MiB), sits at "remove media" prompt
3. Detect completion: poll `blockstat.scsi0.wr_bytes` -- writes start then stop
4. Stop VM (Proxmox stop)
5. **Delete ide2 and ide3** -- IncusOS refuses to start if media present
6. Set `boot: order=scsi0`, start from disk

## IP detection

- **Static IP** (preferred): `ip: ADDR/PREFIX` per VM + `gateway:` + `dns:`.
- **ARP-based** (DHCP fallback): MAC from VM config -> ARP table. Same L2 only.

## VLAN tagging

- `vlan:` config key adds tag to NIC (`tag=69`). Current lab: VLAN 69 (192.168.100.0/22).

## Disk target

- Do NOT specify `disk-target` for Proxmox VMs. IncusOS does **literal** string
  matching. `scsi-*` does NOT match the actual device ID. Omit and let auto-detect.

## Resource pool isolation

- `proxmox.pool` scopes VM operations to a Proxmox pool. API token ACL on `/pool/<name>`.

## Reconcile and idempotency

- Re-runs with existing VMs: interactive menu (status/continue/destroy/abort).
- `--yes` defaults to status checks (never auto-destroys).
- `phase_install` checks each VM state before acting.

## Doctor and cleanup

- `--doctor`: environment check, no config needed.
- `--cleanup`: destroys config VMs. `--cleanup --deep`: also ISOs + remotes.
- `--cleanup-all`: pool-wide. `--cleanup-all --deep`: all IncusOS ISOs.
- `--verbose`/`-v`, `--quiet`, `--retries N` (default 3).

## Boot timeout

- **180s** (60s sleep + 120s poll). First boot downloads sysext (~30-120s).
  Do NOT reduce -- premature retries can corrupt TPM encryption key permanently.

## Disk resize

- Proxmox `qm resize` only grows. Partition 11 auto-expands via `systemd-repart`.
- ZFS pool does NOT auto-expand. Manual expansion via privileged container needed.
  See `notes/aether-guide.md` for procedure.

## First-boot sequence

1. ISO boot + install (~60-85s)
2. Transition: detect complete, stop, remove media, start (~15s)
3. First disk boot: encryption key, sysext download, "System ready" (~50s)
4. **Total: ~130-215s** to port 8443

## Crontab bug (issue #843)

- **Eliminated** by omitting `force_reboot`. Upstream bug only with `force_reboot: true`.
- Empty `ScrubSchedule` crashes daemon. `fix_scrub_schedule()` auto-heals.
- **TPM corruption risk**: hard-stop during first boot permanently corrupts.
  180s timeout prevents this.

## API privileges (role: IncusOSDeployer)

```
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use Sys.Audit
```