105 lines
3.8 KiB
Markdown
105 lines
3.8 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/incusos-proxmox"
|
|
- "incusos/observe-deploy"
|
|
- "incusos/helpers/*"
|
|
- "incusos/examples/*"
|
|
- "incusos/TESTING.md"
|
|
---
|
|
|
|
# Proxmox VE Deployment
|
|
|
|
## incusos-proxmox overview
|
|
|
|
- Reads YAML config, generates per-VM SEED_DATA via `incusos-seed --format iso`,
|
|
uploads ISO + seeds to Proxmox, creates VMs, boots through installation.
|
|
- **Config merge**: `proxmox.yaml` base -> lab config `proxmox:` overlay -> CLI flags.
|
|
Auto-discovers `proxmox.yaml` in script dir then cwd; override with `--proxmox FILE`.
|
|
- **API token**: `env` file at repo root (gitignored). Auto-loaded on startup.
|
|
- **Methods**: SSH (default) or API (`curl -k https://host:8006/api2/json/...`).
|
|
|
|
## Required VM settings (wrong = install failure)
|
|
|
|
- `bios=ovmf`, `machine=q35` -- UEFI required
|
|
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls own Secure Boot keys
|
|
- `tpmstate0`: `version=v2.0` -- required for disk encryption
|
|
- `cpu=host` -- x86_64_v3 instruction set requirement
|
|
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk broken with IncusOS
|
|
- `balloon=0` -- IncusOS manages memory internally
|
|
- `ide3` -- SEED_DATA ISO 9660 as second CD-ROM
|
|
- Minimum 50 GiB disk, 4096 MiB RAM
|
|
|
|
## Install flow (automated)
|
|
|
|
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
|
|
2. IncusOS installs to scsi0 (~876 MiB), sits at "remove media" prompt
|
|
3. Detect completion: poll `blockstat.scsi0.wr_bytes` -- writes start then stop
|
|
4. Stop VM (Proxmox stop)
|
|
5. **Delete ide2 and ide3** -- IncusOS refuses to start if media present
|
|
6. Set `boot: order=scsi0`, start from disk
|
|
|
|
## IP detection
|
|
|
|
- **Static IP** (preferred): `ip: ADDR/PREFIX` per VM + `gateway:` + `dns:`.
|
|
- **ARP-based** (DHCP fallback): MAC from VM config -> ARP table. Same L2 only.
|
|
|
|
## VLAN tagging
|
|
|
|
- `vlan:` config key adds tag to NIC (`tag=69`). Current lab: VLAN 69 (192.168.100.0/22).
|
|
|
|
## Disk target
|
|
|
|
- Do NOT specify `disk-target` for Proxmox VMs. IncusOS does **literal** string
|
|
matching. `scsi-*` does NOT match the actual device ID. Omit and let auto-detect.
|
|
|
|
## Resource pool isolation
|
|
|
|
- `proxmox.pool` scopes VM operations to a Proxmox pool. API token ACL on `/pool/<name>`.
|
|
|
|
## Reconcile and idempotency
|
|
|
|
- Re-runs with existing VMs: interactive menu (status/continue/destroy/abort).
|
|
- `--yes` defaults to status checks (never auto-destroys).
|
|
- `phase_install` checks each VM state before acting.
|
|
|
|
## Doctor and cleanup
|
|
|
|
- `--doctor`: environment check, no config needed.
|
|
- `--cleanup`: destroys config VMs. `--cleanup --deep`: also ISOs + remotes.
|
|
- `--cleanup-all`: pool-wide. `--cleanup-all --deep`: all IncusOS ISOs.
|
|
- `--verbose`/`-v`, `--quiet`, `--retries N` (default 3).
|
|
|
|
## Boot timeout
|
|
|
|
- **180s** (60s sleep + 120s poll). First boot downloads sysext (~30-120s).
|
|
Do NOT reduce -- premature retries can corrupt TPM encryption key permanently.
|
|
|
|
## Disk resize
|
|
|
|
- Proxmox `qm resize` only grows. Partition 11 auto-expands via `systemd-repart`.
|
|
- ZFS pool does NOT auto-expand. Manual expansion via privileged container needed.
|
|
See `notes/aether-guide.md` for procedure.
|
|
|
|
## First-boot sequence
|
|
|
|
1. ISO boot + install (~60-85s)
|
|
2. Transition: detect complete, stop, remove media, start (~15s)
|
|
3. First disk boot: encryption key, sysext download, "System ready" (~50s)
|
|
4. **Total: ~130-215s** to port 8443
|
|
|
|
## Crontab bug (issue #843)
|
|
|
|
- **Eliminated** by omitting `force_reboot`. Upstream bug only with `force_reboot: true`.
|
|
- Empty `ScrubSchedule` crashes daemon. `fix_scrub_schedule()` auto-heals.
|
|
- **TPM corruption risk**: hard-stop during first boot permanently corrupts.
|
|
180s timeout prevents this.
|
|
|
|
## API privileges (role: IncusOSDeployer)
|
|
|
|
```
|
|
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
|
|
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
|
|
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
|
|
Datastore.Audit SDN.Use Sys.Audit
|
|
```
|