# CLAUDE.md - Project context for AI assistants ## What this repository is A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts. ## Repository structure ``` incu-contrib/ ├── CLAUDE.md # This file -- project context ├── README.md # Main overview ├── .gitignore ├── incusos/ # IncusOS installation tooling │ ├── README.md # Detailed usage docs │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with rapid console screenshots │ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) │ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material ├── clustering-guide.md # Detailed Incus clustering walkthrough ├── operations-center-guide.md # Operations Center provisioning & management ├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN) ├── migration-guide.md # Migration paths into Incus from other hypervisors └── utm-support.md # UTM support design document (future) ``` ## Key technical context ### Incus version differences - **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu. - **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package. - `incus remote get-client-certificate` was added in **Incus 6.3+** and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path. - Always prefer reading `~/.config/incus/client.crt` directly from disk. Fall back to the CLI command only as a secondary option. - See `notes/incus-version-compatibility.md` for full platform matrix and install instructions. ### IncusOS flasher-tool - Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest` - Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version` - There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`). - There is NO `--arch` flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via `--image`. - CDN index: `https://images.linuxcontainers.org/os/index.json` - CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz` ### Seed archives - Tar archives containing YAML files at the root level. - Written to byte offset 2148532224 (the seed partition) in the image. - Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM. - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on `/dev/sr*`. - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose FAT filesystem labels, so IncusOS cannot find the seed. - Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, `operations-center.yaml`, `network.yaml`, `update.yaml`. ### Client certificates - Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. - Running `incus remote list` triggers auto-generation if no keypair exists. - For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`). The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including `certificates`) are promoted to the top level of the `preseed` object. - For Operations Center seed: injected under `trusted_client_certificates[]`. - Operations Center **requires** at least one trusted certificate -- without it, you are locked out after installation. ### Proxmox VE deployment - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation. - **Config separation**: Proxmox connection settings (host, credentials, pool) live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`, etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`. - **API token secret**: stored in `env` file at the repo root (gitignored). Scripts auto-load it on startup by searching for `env` up the directory tree from the script location. No manual `source env` needed. If the file is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail with a clear error message. - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). - **Minimum API privileges** for token-based access (role: `IncusOSDeployer`): ``` VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use Sys.Audit ``` `Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via `/nodes//status`). It requires a separate ACL on `/nodes/` since the pool-scoped ACL doesn't cover node-level endpoints. - **Required VM settings** (getting any wrong causes IncusOS install failure): - `bios=ovmf`, `machine=q35` -- UEFI boot required - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys - `tpmstate0`: `version=v2.0` -- required for disk encryption - `cpu=host` -- needed for x86_64_v3 instruction set requirement - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - `balloon=0` -- IncusOS manages memory internally - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - Minimum 50 GiB disk, minimum 4096 MiB RAM - **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM on the tagged VLAN instead of the native/untagged network. The current lab uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the Proxmox VM level only — IncusOS and Incus instances inside the VM are unaware of it. The setting is optional: omit `vlan:` for untagged access. - **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. IncusOS does **literal** string matching (not glob) on disk device IDs. `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). - **Install flow** (automated by `incusos-proxmox`): 1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed 2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), then sits at "please remove installation media" prompt 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- when disk writes start then stop for 15s (3 stable polls), install is done 4. Stop the VM (Proxmox stop, not guest shutdown) 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order 6. Set boot order to `order=scsi0` and start from disk - **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's `network.yaml` configures the VM's interface at boot. No ARP scan or SSH needed — the IP is known at deploy time. Works across VLANs. (2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Only works on the same L2 domain (native LAN, not across VLANs). - **force_reboot is NOT used on Proxmox** (changed 2026-02-22): the seed omits `force_reboot`. Without it, IncusOS sits at "please remove installation media" after install — which is exactly what we want. We detect completion via blockstat (876 MiB written, then idle), stop the VM externally, remove media, and start from disk. This eliminates the SysRq-B intermediate boot that triggers the crontab race condition (issue #843). On physical hardware or unmanaged VMs, `force_reboot: true` is still needed — this optimization is specific to Proxmox automated deploys. `force_reboot` triggers SysRq-B (raw kernel reboot via `/proc/sysrq-trigger`). This does NOT reset QEMU VM uptime — only a Proxmox stop/start does. - **Resource pool isolation**: the optional `proxmox.pool` config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to `/pool/` instead of `/`. Setup: ``` pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0 ``` - **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL). - **Reconcile on re-runs**: when `--phase all` detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With `--yes`, defaults to option 1 (safe -- never auto-destroys). - **Install idempotency**: `phase_install` checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation. ### Multi-lab coexistence - **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop, VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk (refuses to start VMs with install media still attached). These are distinct from `--cleanup` (which destroys VMs permanently). - **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage usage, and per-pool allocation. Requires API method. - **Lab inventory**: `--labs` scans the pool for managed VMs (by `[incusos-lab:managed]` marker), groups by config file, and shows per-lab status, VM count, RAM, and disk. - **VMID range convention** (to avoid collisions between coexisting labs): | Range | Lab | |-------|-----| | 400-499 | OC-managed nodes | | 800-809 | Single-node labs | | 900-909 | Basic cluster | | 910-919 | OC combined (server + nodes) | | 920-929 | OC server standalone | | 930-939 | Advanced / heterogeneous | ### Resource constraints for multi-lab - **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource where you can actually run out. - **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. LZ4 compression provides ~1.5-1.8x ratio on OS data. - **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs can share CPUs without contention. - **Pre-deploy checks**: `incusos-proxmox` warns during preflight if requested RAM exceeds available host RAM (API method only). - **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to run the full deploy pipeline (auto-accepts with `--yes`). - **`--resources` actual disk**: shows actual vs allocated disk for pool VMs (via storage content API) and notes thin provisioning for ZFS/LVM-thin. ### Incus clustering via remotes - **Cluster formation** is done entirely through the `incus` CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access). - **No VIP needed**: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally. #### Pre-clustering: fix core.https_address - IncusOS nodes default to `core.https_address: :8443` (wildcard / all interfaces). Clustering requires a **specific routable IP** so nodes can address each other. - **Set the IP on every node BEFORE enabling clustering:** ```bash incus config set : core.https_address :8443 ``` - Get each node's routable IP via the API: ```bash incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ [print(a) for a in d['environment']['addresses'] \ if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" ``` - This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based. #### Cluster enable (init node) - ```bash incus cluster enable : ``` Note: this is TWO arguments: `:` (trailing colon) and ``. The help text shows `[:] ` — NOT `remote:name` as a single arg. - **TLS certificate regeneration**: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for `127.0.0.1` and `::1`, breaking the existing remote. - **Fix**: remove and re-add the remote to pin the new certificate: ```bash incus remote switch local # if init remote is current default incus remote remove incus remote add https://:8443 --accept-certificate ``` - The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address. #### Joining nodes: apply_defaults and the storage pool conflict - **Upstream recommendation**: use `apply_defaults: false` for nodes destined to join a cluster. The official IncusOS clustering tutorial states joining servers "cannot have preexisting networks or storage pools defined." With `apply_defaults: false`, the node still listens on port 8443, still trusts preseed certificates, and the underlying ZFS dataset (`local/incus`) still exists -- but no Incus storage pool or network metadata is created, so the join process works cleanly. - **`apply_defaults: true` on joining nodes** is also functional but requires an 8-command cleanup per node before join (delete pool, network, volumes, profile devices). This is automated in `lab-test` but adds complexity. - **Recommended seed pattern for clusters**: - Bootstrap/init node: `apply_defaults: true` (needs pool and network) - Joining nodes: `apply_defaults: false` (join process creates member-specific entries) - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional) - **If `apply_defaults: true` was used**, the cleanup before join is: ```bash # 1. Remove config references incus config unset : storage.backups_volume incus config unset : storage.images_volume # 2. Delete volumes incus storage volume delete :local backups incus storage volume delete :local images # 3. Clear default profile references (pool is "in use" otherwise) incus profile device remove :default root incus profile device remove :default eth0 # 4. Delete pool and network incus storage delete :local incus network delete :incusbr0 ``` #### Join workflow - **Generate token** (on init node, single argument `remote:member-name`): ```bash incus cluster add : ``` - **Join** (interactive -- prompts for 5 values): ```bash incus cluster join : : ``` Interactive prompts and correct answers: 1. IP address → accept default (node's IP, already set via core.https_address) 2. Member name → accept default (matches the token) 3. "All existing data is lost" → `yes` 4. `source` property for storage pool "local" → `local/incus` 5. `zfs.pool_name` property for storage pool "local" → `local/incus` - **Automated (non-interactive):** ```bash printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : ``` - **After join**: the joining node gets a new cluster certificate. Fix the remote (same as init node): ```bash incus remote remove incus remote add https://:8443 --accept-certificate ``` #### Command syntax gotchas - `incus cluster enable remote: member-name` -- TWO arguments (remote: + name) - `incus cluster add remote:member-name` -- ONE argument (no space) - `incus cluster remove remote:member-name --force` -- ONE argument; prompts "yes/no" even with `--force`, pipe `printf "yes\n"` for automation - `incus cluster evacuate remote:member-name` -- ONE argument (no space) - `incus cluster restore remote:member-name` -- ONE argument (no space) - `incus cluster join init-remote: joining-remote:` -- TWO arguments (space) - `incus storage show remote:pool` -- ONE argument (no space) - `incus storage show remote:pool --target member` -- target flag for member-specific config - `incus config set remote: key value` -- remote with trailing colon + space - General rule: `remote:resource` for targeting a resource, `remote:` (trailing colon) for targeting the server itself #### Post-join state - After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations. - **`lab-test`** automates cluster formation, workload testing, and migration. #### Workload placement and migration - **Targeted launch**: `incus launch images:debian/12 :name --target ` - **Cluster-wide visibility**: `incus list` on any member shows all instances. - **Container migration**: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not. ```bash incus stop : incus move : --target incus start : ``` - **VM live migration**: requires `migration.stateful=true` (must be set while VM is stopped). Preserves running state with no downtime. ```bash incus move : --target ``` - **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`), not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`). Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays differently, causing `Missing section footer for ICH9LPC` on restore. Using a range (pinning syntax) eliminates `maxcpus` entirely and uses fixed `sockets/cores/threads` topology — portable across all hosts. ```bash # WRONG: integer → maxcpus varies by host → migration fails incus config set limits.cpu=2 # RIGHT: range → fixed topology → migration works everywhere incus config set limits.cpu=0-1 ``` - **VM live migration works in nested virtualization** (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). - **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration. - **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also requires the `limits.cpu` range fix. Use `incus start --stateless` to discard a saved state file that cannot be restored. - **VM `size.state` config**: stateful operations require `size.state` on the root disk (`incus config device add root disk path=/ pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails. - **Cluster evacuation**: `incus cluster evacuate : --force` (ONE argument, like `cluster enable` and `cluster add`). Use `--action stop` if VMs lack the `limits.cpu` range fix. Restore with `incus cluster restore : --force`. - **VM agent reconnect**: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should `sleep 4` after migration before running `incus exec`. - **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs. - **Concurrent migrations**: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration. - **Active I/O during migration**: disk writes and network activity survive live migration transparently. File integrity verified after migration. - **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with `migration.stateful=true`. Containers are NOT auto-rebalanced. ```bash incus config set : cluster.rebalance.interval=1 # minutes incus config set : cluster.rebalance.threshold=10 # imbalance % incus config set : cluster.rebalance.batch=2 # max VMs/run incus config set : cluster.rebalance.cooldown=5m # wait between runs ``` - **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md` for step-by-step instructions. - **`incus cluster remove` requires confirmation**: even with `--force`, it prompts "Are you really sure?". Pipe `yes` for automation: ```bash printf "yes\n" | incus cluster remove : --force ``` - See `notes/clustering-guide.md` for full details and references. ### Lab validation (lab-test) - **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates on the VM names defined there (expects `incus` remotes to exist). - **Phases:** deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore). - **Test instances** use names starting with `test-` for easy cleanup. - The script reports PASS/FAIL/SKIP for each test and prints a summary. ### Operations Center - **CLI binary:** `operations-center` (installed from GitHub releases or built from source at `github.com/FuturFusion/operations-center`). - **Config directory:** `~/.config/operations-center/` (uses same cert format as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). - **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI. - **Browser access**: requires PKCS#12 client certificate (`client.pfx`) imported into the browser. Web UI is a React SPA at `/ui/`. - **OC is under active development** (v0.3.0). Commands and APIs may change. Use `operations-center --version` (not `version` subcommand) to check. - **v0.3.0 new commands**: `provisioning channel` (add/list/show), `cluster update/update-certificate/rename`, expanded `inventory` (network-acl, address-sets, load-balancers, peers, zones, integrations, forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource tree view with filters and Go templates), `system certificate set`. - The `--doctor` command on `incusos-proxmox` reports whether the CLI is installed. #### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster) - **No brownfield adoption**: nodes must boot from an OC-provisioned ISO. - **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML must use structured format with section keys (`install:`, not flat). ```bash operations-center provisioning token seed add proxmox-preseed \ /tmp/preseed.yaml --description "Force reboot for Proxmox" operations-center provisioning token seed get-image proxmox-preseed \ /tmp/IncusOS-oc.iso --type iso --architecture x86_64 ``` - **Hybrid deployment** (tested, recommended): `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml` combines OC auto-registration (from boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist. - **Self-registration**: nodes auto-register with OC within ~30s of first boot. Hostname from SEED_DATA is used as the server name. - **Cluster formation** (tested): ```bash operations-center provisioning cluster add oc-cluster \ https://:8443 \ --server-names oc-node-01,oc-node-02,oc-node-03 \ --server-type incus \ --application-seed-config /tmp/oc-app-config.yaml ``` OC handles: `core.https_address` → cluster enable → joins → seed application → Terraform config. Adds `meshbr0` network. - **apply_defaults: false is recommended** (tested): use `apply_defaults: false` for OC-managed nodes. OC's Terraform handles storage pool, network, and cert creation cleanly. With `apply_defaults: true`, nodes already have these resources and OC's Terraform fails with "already exists" errors (cluster still forms, but Terraform artifacts are empty). #### Tested limitations - **Inventory is NOT real-time** -- requires explicit `cluster resync` - **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). The failure is OC-specific: the OC agent pushes config via the IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid values (e.g., cron expression where Go duration is expected) crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. - **No cluster member state tracking** -- OC always shows `ready` even for EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat detects failure in ~40s; OC has no equivalent). - **Stale entries** from out-of-band cluster changes persist after resync - **Server removal blocked** if server is part of an OC cluster - **Node failure recovery**: Proxmox hard-stop simulates crash. After restart, node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`). - See `notes/operations-center-guide.md` for full tested OC reference. ### incusos-proxmox doctor and cleanup - **`--doctor`**: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file). - **`--cleanup`**: destroys VMs defined in the config file. - **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior). - **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab config required). Destroys all VMs with `[incusos-lab:managed]` marker. - **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso` and `seed-*.iso` from storage + remotes + cache. - **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default output is concise (step names + results). `--quiet` suppresses everything except warnings and errors. - **`--retries N`**: number of stop+start retries for VMs that fail to boot (port 8443 not reachable). Default: 3. `--retries 0` disables retries. Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for up to 120s. With the `force_reboot` fix (2026-02-22), the crontab bug is eliminated and retries are rarely needed. They remain as a safety net for other transient boot failures. After retries, `fix_scrub_schedule()` proactively heals any remaining scrub_schedule issues via the IncusOS REST API. - **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot downloads SecureBoot update + application sysext (Incus: ~30-120s depending on CDN speed and concurrent VMs). The previous 90s timeout was too short under load — premature retries (hard VM stop during first boot) can corrupt the TPM encryption key, causing permanent "zfs load-key: Raw key too short (expected 32)" errors that no amount of retries can fix. ### IncusOS first-boot sequence (observed via console screenshots) Observed using `observe-deploy` with 2-second screenshot intervals. The complete lifecycle from Proxmox VM start to port 8443 ready: **Phase 1: ISO boot and installation (~60-85s)** 1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s) 2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s) 3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s) - "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda" - "Cloning GPT partitions" → progress bar → complete 4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via `/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not a graceful systemd reboot. This is intentional — the install environment is minimal and may not have systemd running. 5. VM reboots but ISO still attached → boots from ISO again 6. "System check error: install media detected, but the system is already installed; please remove USB/CDROM and reboot the system" (~40s). IncusOS daemon sleeps **1 hour** at this error, then exits. The `state.txt` on the installed disk (scsi0 data partition) is **NOT touched** — this boot runs from the ISO rootfs (tmpfs). 7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle) **Phase 2: Transition (~15s)** 1. Proxmox stops VM 2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0` 3. Start VM from disk **Phase 3: First boot from disk (~50s to port 8443)** 1. UEFI → "IncusOS is starting..." (~8s) 2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`, `Channel="stable"` → `Save()` writes state.txt to data partition 3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s) 4. "System is starting up machine-id=... mode=production" (~10s after key) 5. "Bringing up the network" (immediate) 6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s) 7. "Downloading application update application=incus" + progress bar (~33-38s) 8. "Bringing up the local storage" (after download) 9. "Starting application" → "Initializing application" → TLS cert fingerprint 10. `registerJobs()` validates ScrubSchedule with gocron 11. "System is ready" — port 8443 now reachable **Total: ~130-215s from VM create to port 8443 ready.** The biggest variable is the application download (33-38s on first boot; skipped if already cached). **Important**: `DoInstall()` does NOT create `state.txt`. It writes the IncusOS image to disk, processes seed files (including deleting `install.yaml` from the target's seed partition via `CleanupPostInstall()`), and triggers the SysRq-B reboot. The `state.txt` is created exclusively on first boot from the installed disk by `state.LoadOrCreate()` → `initialize()`. ### IncusOS boot failure (crontab / update frequency error) - **RESOLVED** (2026-02-22): root cause identified and fixed in our pipeline. The `force_reboot` seed option triggers SysRq-B after install, causing an "install media detected" intermediate boot. This intermediate boot generates ~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs). Our blockstat detection catches these late writes, but by then the intermediate boot has run and somehow corrupted the state for the subsequent first-from-disk boot. **Fix: omit `force_reboot` from the seed.** Without it, the installer sits at "please remove media" after the 876 MiB image clone, blockstat detects idle cleanly, and the first disk boot is pristine. Results: **15/15 PASS (100%)** without force_reboot on 4 cores vs 50% with it. - **Symptom**: "ERROR invalid crontab expression" appears on the console during first boot. Intermittent — confirmed **~50% failure rate** with `force_reboot: true` on 4-core VMs. Error does NOT correlate with `update.yaml` presence/absence, ISO version, or version mismatch. - **Two distinct error paths in IncusOS**: - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire daemon exits — **this is the fatal one**. Port 8443 never opens (or opens briefly during the 15-second error sleep). - `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()` format like `"6h"`). If invalid, it logs an error but does **not** crash. - **Not always fatal** (confirmed by parallel investigation): - The REST API starts BEFORE `startup()` completes. Applications (Incus) start at step ~16 of `startup()`, while `registerJobs()` is at step ~19. When Incus starts before the scheduler crashes, port 8443 is available during the 15-second error sleep (before `os.Exit(1)`). - Port 8443 is reachable during the error window, and API calls work (including `/os/1.0/system/storage`). The `scrub_schedule` field is empty in the API response — this is the definitive indicator. - "System is ready" is NOT shown on console (only appears after `registerJobs()` succeeds). Console shows "ERROR invalid crontab expression" as the last line when the bug hits. - **Detection heuristic**: port 8443 reachable BUT `scrub_schedule` is empty → crontab bug hit. Port 8443 reachable AND `scrub_schedule` is `"0 4 * * 0"` → genuine success. - **Source code analysis** (root cause investigation): - `state.txt` is created FRESH on first boot from disk by `LoadOrCreate()` → `initialize()`. The `initialize()` function correctly sets `ScrubSchedule = "0 4 * * 0"`. - The "install media detected" boot (#2, ISO still attached) runs from the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed disk. The installed disk's state.txt is untouched. - `DoInstall()` does not create state.txt. It only writes the OS image and processes seed files. - **The encoder skips zero values**: `encodeHelper()` returns early on `v.IsZero()`. An empty ScrubSchedule string is NOT written to file. If state.txt is read with a missing ScrubSchedule, the field stays at Go's zero value (empty string) → `gocron.IsValid("")` fails. - **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which truncates then writes. Crash during write → partial file. - **Race window**: The REST API server starts BEFORE `startup()` completes. Applications start before `registerJobs()`. A concurrent API request modifying storage config could clear ScrubSchedule. - **Root cause confirmed**: the bug is triggered by the SysRq-B intermediate boot (from `force_reboot: true`). Controlled experiments on VLAN 69 with static IP (2026-02-22, ISO 202602210344): - Baseline (4 cores, force_reboot): 5/10 PASS (50%) - cache=writethrough: 3/5 PASS (60%) — falsified - agent=0: 3/5 PASS (60%) — falsified - cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race - **No force_reboot (4 cores): 15/15 PASS (100%)** — confirmed fix Correlation: runs with 876 MiB blockstat detection (before SysRq-B) had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%. - **The try-it service is NOT IncusOS**: stgraber's comparison ("thousands of times a day") is invalid. The try-it service runs Ubuntu VMs with Incus installed from the Zabbly daily repo — a completely different product. Issue #843 was independently filed by another Proxmox user (fperreau on Proxmox 9.1.4), confirming it's not unique to our setup. - The underlying IncusOS bug is still a race condition between the REST API/application startup and `registerJobs()` that clears `ScrubSchedule`. Investigation ruled out version mismatch, update downloads, and gocron library issues. The field is genuinely empty in the state struct. - **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel): Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%); Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG (67% failure rate). Console screenshots confirm "ERROR invalid crontab expression" on failed VMs and "System is ready" on successful VMs. - **Version mismatch NOT the cause**: stgraber's hypothesis that the bug relates to ISO version != CDN latest was tested and ruled out. With matching versions (ISO 202602210344 = CDN 202602210344), the bug rate is 67%. Sysext downloads still occur with matching versions (SecureBoot + application updates) but no OS update download. The bug occurs regardless of whether sysext downloads happen. - **Sysext download on matching versions**: even when ISO version matches CDN latest, the first boot downloads SecureBoot update (~1s) and application sysext (~2 min for Incus). The sysext is NOT in the ISO — it's always downloaded from the update provider. An OS update download (and reboot) only occurs when versions differ. - **Proposed upstream fix** (for `incus-os` project): ```go // In registerJobs(): validate and fall back to default func registerJobs(s *state.State) error { schedule := s.System.Storage.Config.ScrubSchedule if schedule == "" { schedule = "0 4 * * 0" s.System.Storage.Config.ScrubSchedule = schedule } err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools) if err != nil { // Fall back to default instead of crashing the daemon slog.Warn("Invalid scrub schedule, using default", "schedule", schedule) s.System.Storage.Config.ScrubSchedule = "0 4 * * 0" return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools) } return nil } ``` Additional recommendations: (1) use atomic writes in `Save()` (write to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even when empty (to preserve the default through encode/decode cycles), (3) add mutex protection on State fields shared between API handlers and startup(). - **Error is persistent within a VM session**: verified by screenshots — when the crontab error hits, the deferred `s.Save()` writes corrupt state (version 7, ScrubSchedule missing because encoder skips zero values). Subsequent systemd daemon restarts read the corrupt state and crash again. The error does NOT self-heal via daemon restart alone. - **Recovery via Proxmox stop+start**: `phase_install` in `incusos-proxmox` retries once with a full Proxmox stop (hard VM power-off) → 10s wait → start. The hard power-off may cause uncommitted filesystem writes (including the corrupt state.txt) to be lost, restoring the correct state from `initialize()`. This works because IncusOS's data partition has a journal commit interval (~5s) and the corrupt Save() may not have been committed before the VM was killed. - **Detection**: `phase_install` checks port 8443 after starting from disk (60s wait + up to 120s polling = 180s total). If first check fails, automatic retry adds ~180s. Failed VMs after retry are reported with remediation. - **TPM corruption from premature retry**: hard-stopping a VM during first boot (while the encryption key is being written to the TPM zvol) can leave the TPM with truncated key data. All subsequent boots fail with "zfs load-key: Raw key too short (expected 32)" — a permanent error that requires VM destruction and redeployment to fix. The 180s boot timeout prevents this by allowing the full first-boot sequence (including sysext download) to complete before any retry. - **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes `fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix proxies to the IncusOS daemon API through Incus. Safe to call on every node — returns early if schedule is already set. Called on both initial success and retry success paths. - **Filed upstream**: IncusOS issue #843. - **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`, `updateChecker()`, `startup()`, `firstBootActions()`), `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`, `ErrInvalidCronTab`), `incus-osd/internal/state/file.go` (`LoadOrCreate()`, `Save()`, `initialize()`), `incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip), `incus-osd/internal/install/install.go` (`DoInstall()`, `rebootUponDeviceRemoval()` SysRq-B). ### Proxmox SSH root access — strict rules Root SSH access to the Proxmox host is available for **diagnostics only**. The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`. **MANDATORY RULES — violation of any rule is unacceptable:** 1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM console screenshots via `qm monitor screendump`. No other use without explicit user instruction. 2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869) or VMs in the IncusLab pool that were created by our scripts. 3. **No modifications**: NEVER run any command that modifies, stops, starts, or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`, `pct` commands, `zfs` commands, `systemctl` commands, or ANY write operation on the host. 4. **Do not touch the dev VM**: the user's dev VM runs on the same host. Do not interact with it in any way — do not even `qm status` it. 5. **No config access**: do not read or modify `/etc/pve/`, storage configs, network configs, user/ACL settings, or any host-level configuration. 6. **Transparency**: every SSH command is visible in tool output. If the user rejects a command, do not retry it. 7. **Only during active tests**: only SSH during test runs where test VMs have been deployed, and only to screenshot those test VMs. **Permitted commands (exhaustive list):** ```bash # Take screenshot of a test VM (VMID in 850-869 range) sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ root@ "echo 'screendump /tmp/vm--screen.ppm' | qm monitor " # Retrieve screenshot sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \ root@:/tmp/vm--screen.ppm /tmp/ # Cleanup screenshot on remote sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ root@ "rm -f /tmp/vm--screen.ppm" ``` **Technical notes:** - QEMU's `screendump` is root-only in PVE 9's HMP permission model. API tokens cannot execute it regardless of privileges. - Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng, which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`). - PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`: `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"` PNG output is ~20 KB and can be read directly by Claude Code (multimodal). ## Coding conventions for scripts - **Shell**: bash with `set -euo pipefail` - **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid false exits under `set -e` when the value is 0. - **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern. - **Flags**: support both short (`-d`) and long (`--defaults`) options. - **Defaults**: sane defaults so the script does something useful with zero flags. - **Dry run**: all scripts should support `--dry-run` to preview actions. - **Cert detection order**: files on disk first, CLI command second. - **Error messages**: include actionable remediation steps, not just "failed". - **No hardcoded package managers**: say "install the Incus client" with a link, not "sudo apt install incus". ### Incus networking (OVN) - **Bridge networks are node-local**: each cluster member has its own independent bridge. Instances on the same bridge (same node) can communicate; cross-node instances CANNOT. Each bridge has the same subnet (e.g., 10.0.0.1/24) but they are separate L2 domains. - **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes. Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane + client services + physical uplink network. - **IncusOS OVN services are disabled by default**: must be enabled via the IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection` fails with `db.sock not found`. - **OVN service enable API call**: ```bash incus query :/os/1.0/services/ovn --request PUT --data '{ "config": { "database": "tcp::6642", "enabled": true, "tunnel_address": "", "tunnel_protocol": "geneve" }, "state": {} }' ``` `database` is the **southbound** DB (port 6642), NOT northbound (6641). - **OVN control plane as container**: deploy `ovn-central` package in a Debian container on the cluster. Use proxy devices to expose NB (6641) and SB (6642) ports on the host's LAN IP so all nodes can reach it. - **Setup sequence** (order matters): 1. Deploy OVN control plane container 2. Enable OVN services on ALL IncusOS nodes 3. `incus config set network.ovn.northbound_connection tcp::6641` 4. `incus cluster role add : ovn-chassis` (all nodes) 5. Create physical uplink network (two-step cluster pattern) 6. Create OVN network with `--type=ovn network=UPLINK` - **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC), `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, `ipv4.gateway` is the LAN gateway in CIDR format. - **OVN network isolation**: multiple OVN networks are fully isolated. Instances on different networks cannot communicate, even on the same node. Network peering (`incus network peer create`) enables cross-network routing. - **OVN features tested**: cross-node connectivity, network isolation, ACLs (per-source blocking), network peering, L4 load balancers (connection- based hashing, not round-robin), network forwards (port forwarding to LAN IPs), DNS resolution (per-network, hostname.incus domain). - See `notes/networking-guide.md` for full tutorial with test results. ### Migration into Incus - **`incus-migrate`**: official tool for importing disk images, running instances, or physical machines into Incus. - **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2, raw, vdi, vhd formats. Incus accepts raw and qcow2. - **Import workflow**: convert disk → `incus storage volume import` → `incus init --empty --vm` → attach disk → start. - **Container migration**: `docker export` → `incus import` for filesystem- level container migration. Docker volumes must be copied separately. - See `notes/migration-guide.md` for full procedures per source hypervisor. ### UTM support (future) - Design document at `notes/utm-support.md`. - UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation (requires AppleScript or `.utm` bundle generation). - No `blockstat` equivalent -- install detection must use timeout + port polling. - Seed generation already works cross-platform (Phase 3 macOS compatibility). ## Git workflow - Main branch: `main` - Development happens on feature branches - Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`