# CLAUDE.md - Project context for AI assistants ## What this repository is A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts. ## Repository structure ``` incu-contrib/ ├── CLAUDE.md # This file -- project context ├── README.md # Main overview ├── .gitignore ├── incusos/ # IncusOS installation tooling │ ├── README.md # Detailed usage docs │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with rapid console screenshots │ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) │ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material ├── clustering-guide.md # Detailed Incus clustering walkthrough ├── operations-center-guide.md # Operations Center provisioning & management ├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN) ├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested) ├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end) ├── migration-guide.md # Migration paths into Incus from other hypervisors ├── incus-version-compatibility.md # Incus versions across platforms ├── iso-download-methods.md # ISO download/customization research └── utm-support.md # UTM support design document (future) ``` ## Key technical context ### Incus version differences - **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu. - **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package. - `incus remote get-client-certificate` was added in **Incus 6.3+** and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path. - Always prefer reading `~/.config/incus/client.crt` directly from disk. Fall back to the CLI command only as a secondary option. - See `notes/incus-version-compatibility.md` for full platform matrix and install instructions. ### IncusOS flasher-tool - Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest` - Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version` - There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`). - There is NO `--arch` flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via `--image`. - CDN index: `https://images.linuxcontainers.org/os/index.json` - CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz` ### Seed archives - Tar archives containing YAML files at the root level. - Written to byte offset 2148532224 (the seed partition) in the image. - Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM. - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on `/dev/sr*`. - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose FAT filesystem labels, so IncusOS cannot find the seed. - Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, `operations-center.yaml`, `network.yaml`, `update.yaml`. ### Client certificates - Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. - Running `incus remote list` triggers auto-generation if no keypair exists. - For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`). The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including `certificates`) are promoted to the top level of the `preseed` object. - For Operations Center seed: injected under `trusted_client_certificates[]`. - Operations Center **requires** at least one trusted certificate -- without it, you are locked out after installation. ### Proxmox VE deployment - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation. - **Config separation**: Proxmox connection settings (host, credentials, pool) live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`, etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`. - **API token secret**: stored in `env` file at the repo root (gitignored). Scripts auto-load it on startup by searching for `env` up the directory tree from the script location. No manual `source env` needed. If the file is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail with a clear error message. - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). - **Minimum API privileges** for token-based access (role: `IncusOSDeployer`): ``` VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use Sys.Audit ``` `Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via `/nodes//status`). It requires a separate ACL on `/nodes/` since the pool-scoped ACL doesn't cover node-level endpoints. - **Required VM settings** (getting any wrong causes IncusOS install failure): - `bios=ovmf`, `machine=q35` -- UEFI boot required - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys - `tpmstate0`: `version=v2.0` -- required for disk encryption - `cpu=host` -- needed for x86_64_v3 instruction set requirement - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - `balloon=0` -- IncusOS manages memory internally - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - Minimum 50 GiB disk, minimum 4096 MiB RAM - **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM on the tagged VLAN instead of the native/untagged network. The current lab uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the Proxmox VM level only — IncusOS and Incus instances inside the VM are unaware of it. The setting is optional: omit `vlan:` for untagged access. - **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. IncusOS does **literal** string matching (not glob) on disk device IDs. `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). - **Install flow** (automated by `incusos-proxmox`): 1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed 2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), then sits at "please remove installation media" prompt 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- when disk writes start then stop for 15s (3 stable polls), install is done 4. Stop the VM (Proxmox stop, not guest shutdown) 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order 6. Set boot order to `order=scsi0` and start from disk - **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's `network.yaml` configures the VM's interface at boot. No ARP scan or SSH needed — the IP is known at deploy time. Works across VLANs. (2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Only works on the same L2 domain (native LAN, not across VLANs). - **force_reboot is NOT used on Proxmox**: the seed omits `force_reboot`. IncusOS sits at "please remove installation media" after install. We detect completion via blockstat (876 MiB written, then idle), stop the VM externally, remove media, and start from disk. This avoids the crontab race condition (issue #843). On physical hardware, `force_reboot: true` is still needed (no external orchestrator). - **Resource pool isolation**: the optional `proxmox.pool` config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to `/pool/` instead of `/`. Setup: ``` pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0 ``` - **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL). - **Reconcile on re-runs**: when `--phase all` detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With `--yes`, defaults to option 1 (safe -- never auto-destroys). - **Install idempotency**: `phase_install` checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation. ### Multi-lab coexistence - **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop, VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk (refuses to start VMs with install media still attached). These are distinct from `--cleanup` (which destroys VMs permanently). - **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage usage, and per-pool allocation. Requires API method. - **Lab inventory**: `--labs` scans the pool for managed VMs (by `[incusos-lab:managed]` marker), groups by config file, and shows per-lab status, VM count, RAM, and disk. - **VMID range convention** (to avoid collisions between coexisting labs): | Range | Lab | |-------|-----| | 400-499 | OC-managed nodes | | 800-809 | Single-node labs | | 900-909 | Basic cluster | | 910-919 | OC combined (server + nodes) | | 920-929 | OC server standalone | | 930-939 | Advanced / heterogeneous | ### Resource constraints for multi-lab - **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource where you can actually run out. - **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. LZ4 compression provides ~1.5-1.8x ratio on OS data. - **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs can share CPUs without contention. - **Pre-deploy checks**: `incusos-proxmox` warns during preflight if requested RAM exceeds available host RAM (API method only). - **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to run the full deploy pipeline (auto-accepts with `--yes`). - **`--resources` actual disk**: shows actual vs allocated disk for pool VMs (via storage content API) and notes thin provisioning for ZFS/LVM-thin. ### Incus clustering via remotes - **Cluster formation** is done entirely through the `incus` CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access). - **No VIP needed**: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally. #### Pre-clustering: fix core.https_address - IncusOS nodes default to `core.https_address: :8443` (wildcard / all interfaces). Clustering requires a **specific routable IP** so nodes can address each other. - **Set the IP on every node BEFORE enabling clustering:** ```bash incus config set : core.https_address :8443 ``` - Get each node's routable IP via the API: ```bash incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ [print(a) for a in d['environment']['addresses'] \ if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" ``` - This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based. #### Cluster enable (init node) - ```bash incus cluster enable : ``` Note: this is TWO arguments: `:` (trailing colon) and ``. The help text shows `[:] ` — NOT `remote:name` as a single arg. - **TLS certificate regeneration**: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for `127.0.0.1` and `::1`, breaking the existing remote. - **Fix**: remove and re-add the remote to pin the new certificate: ```bash incus remote switch local # if init remote is current default incus remote remove incus remote add https://:8443 --accept-certificate ``` - The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address. #### Joining nodes: apply_defaults and the storage pool conflict - **Upstream recommendation**: use `apply_defaults: false` for nodes destined to join a cluster. The official IncusOS clustering tutorial states joining servers "cannot have preexisting networks or storage pools defined." With `apply_defaults: false`, the node still listens on port 8443, still trusts preseed certificates, and the underlying ZFS dataset (`local/incus`) still exists -- but no Incus storage pool or network metadata is created, so the join process works cleanly. - **`apply_defaults: true` on joining nodes** is also functional but requires an 8-command cleanup per node before join (delete pool, network, volumes, profile devices). This is automated in `lab-test` but adds complexity. - **Recommended seed pattern for clusters**: - Bootstrap/init node: `apply_defaults: true` (needs pool and network) - Joining nodes: `apply_defaults: false` (join process creates member-specific entries) - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional) - **If `apply_defaults: true` was used**, the cleanup before join is: ```bash # 1. Remove config references incus config unset : storage.backups_volume incus config unset : storage.images_volume # 2. Delete volumes incus storage volume delete :local backups incus storage volume delete :local images # 3. Clear default profile references (pool is "in use" otherwise) incus profile device remove :default root incus profile device remove :default eth0 # 4. Delete pool and network incus storage delete :local incus network delete :incusbr0 ``` #### Join workflow - **Generate token** (on init node, single argument `remote:member-name`): ```bash incus cluster add : ``` - **Join** (interactive -- prompts for 5 values): ```bash incus cluster join : : ``` Interactive prompts and correct answers: 1. IP address → accept default (node's IP, already set via core.https_address) 2. Member name → accept default (matches the token) 3. "All existing data is lost" → `yes` 4. `source` property for storage pool "local" → `local/incus` 5. `zfs.pool_name` property for storage pool "local" → `local/incus` - **Automated (non-interactive):** ```bash printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : ``` - **After join**: the joining node gets a new cluster certificate. Fix the remote (same as init node): ```bash incus remote remove incus remote add https://:8443 --accept-certificate ``` #### Command syntax gotchas - `incus cluster enable remote: member-name` -- TWO arguments (remote: + name) - `incus cluster add remote:member-name` -- ONE argument (no space) - `incus cluster remove remote:member-name --force` -- ONE argument; prompts "yes/no" even with `--force`, pipe `printf "yes\n"` for automation - `incus cluster evacuate remote:member-name` -- ONE argument (no space) - `incus cluster restore remote:member-name` -- ONE argument (no space) - `incus cluster join init-remote: joining-remote:` -- TWO arguments (space) - `incus storage show remote:pool` -- ONE argument (no space) - `incus storage show remote:pool --target member` -- target flag for member-specific config - `incus config set remote: key value` -- remote with trailing colon + space - General rule: `remote:resource` for targeting a resource, `remote:` (trailing colon) for targeting the server itself #### Post-join state - After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations. - **`lab-test`** automates cluster formation, workload testing, and migration. #### Workload placement and migration - **Targeted launch**: `incus launch images:debian/12 :name --target ` - **Cluster-wide visibility**: `incus list` on any member shows all instances. - **Container migration**: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not. ```bash incus stop : incus move : --target incus start : ``` - **VM live migration**: requires `migration.stateful=true` (must be set while VM is stopped). Preserves running state with no downtime. ```bash incus move : --target ``` - **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`), not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`). Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays differently, causing `Missing section footer for ICH9LPC` on restore. Using a range (pinning syntax) eliminates `maxcpus` entirely and uses fixed `sockets/cores/threads` topology — portable across all hosts. ```bash # WRONG: integer → maxcpus varies by host → migration fails incus config set limits.cpu=2 # RIGHT: range → fixed topology → migration works everywhere incus config set limits.cpu=0-1 ``` - **VM live migration works in nested virtualization** (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). - **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration. - **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also requires the `limits.cpu` range fix. Use `incus start --stateless` to discard a saved state file that cannot be restored. - **VM `size.state` config**: stateful operations require `size.state` on the root disk (`incus config device add root disk path=/ pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails. - **Cluster evacuation**: `incus cluster evacuate : --force` (ONE argument, like `cluster enable` and `cluster add`). Use `--action stop` if VMs lack the `limits.cpu` range fix. Restore with `incus cluster restore : --force`. - **VM agent reconnect**: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should `sleep 4` after migration before running `incus exec`. - **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs. - **Concurrent migrations**: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration. - **Active I/O during migration**: disk writes and network activity survive live migration transparently. File integrity verified after migration. - **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with `migration.stateful=true`. Containers are NOT auto-rebalanced. ```bash incus config set : cluster.rebalance.interval=1 # minutes incus config set : cluster.rebalance.threshold=10 # imbalance % incus config set : cluster.rebalance.batch=2 # max VMs/run incus config set : cluster.rebalance.cooldown=5m # wait between runs ``` - **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md` for step-by-step instructions. - **`incus cluster remove` requires confirmation**: even with `--force`, it prompts "Are you really sure?". Pipe `yes` for automation: ```bash printf "yes\n" | incus cluster remove : --force ``` - See `notes/clustering-guide.md` for full details and references. ### Lab validation (lab-test) - **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates on the VM names defined there (expects `incus` remotes to exist). - **Phases:** deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore). - **Test instances** use names starting with `test-` for easy cleanup. - The script reports PASS/FAIL/SKIP for each test and prints a summary. ### Operations Center - **CLI binary:** `operations-center` (installed from GitHub releases or built from source at `github.com/FuturFusion/operations-center`). - **Config directory:** `~/.config/operations-center/` (uses same cert format as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). - **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI. - **Browser access**: requires PKCS#12 client certificate (`client.pfx`) imported into the browser. Web UI is a React SPA at `/ui/`. - **OC is under active development** (v0.3.0). Commands and APIs may change. Use `operations-center --version` (not `version` subcommand) to check. - **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI, the OC CLI requires `operations-center remote switch NAME` before running commands. `operations-center admin os show oc-lab:` fails with "Invalid number of arguments". - **v0.3.0 new commands**: `provisioning channel` (add/list/show), `cluster update/update-certificate/rename`, expanded `inventory` (network-acl, address-sets, load-balancers, peers, zones, integrations, forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource tree view with filters and Go templates), `system certificate set`. - The `--doctor` command on `incusos-proxmox` reports whether the CLI is installed. #### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster) - **No brownfield adoption**: nodes must boot from an OC-provisioned ISO. - **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML must use structured format with section keys (`install:`, not flat). **No force_reboot** in token seeds for Proxmox (same fix as standard deploys). ```bash operations-center provisioning token seed add proxmox-preseed \ /tmp/preseed.yaml --description "No force_reboot for Proxmox" operations-center provisioning token seed get-image proxmox-preseed \ /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable ``` - **Hybrid deployment** (tested, recommended): `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml` combines OC auto-registration (from boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist. - **Self-registration**: nodes auto-register with OC within ~30s of first boot. Hostname from SEED_DATA is used as the server name. - **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires all nodes to have `needs_update: false` before `provisioning cluster add` succeeds. Nodes deployed from an ISO matching the latest OC update version are tracked as `needs_update: true` because the OS was never delivered through OC's update pipeline. The `needs_update` flag is server-side computed (not a simple version comparison) and cannot be overridden via REST API PUT. **Solution**: generate the ISO from an older channel (`--channel old-stable`) so nodes start with an older version. OC then pushes the latest update through its pipeline, clearing the flag. Use `provisioning update assign-channels` to control which versions are in which channels. - **Cluster formation** (tested, 2026-02-23): ```bash # Use empty app config if cert already injected via SEED_DATA echo '{}' > /tmp/oc-app-config.yaml operations-center provisioning cluster add oc-cluster \ https://:8443 \ --server-names oc-node-01,oc-node-02,oc-node-03 \ --server-type incus \ --application-seed-config /tmp/oc-app-config.yaml ``` OC handles: `core.https_address` → cluster enable → joins → storage pool + network creation → Terraform config. Adds `meshbr0` network. If the client cert is already in the trust store (from SEED_DATA), use an empty `{}` app config to avoid "Certificate already in trust store" Terraform error. The cluster forms successfully either way. - **apply_defaults: false is recommended** (tested): use `apply_defaults: false` for OC-managed nodes. OC's Terraform handles storage pool, network, and cert creation cleanly. With `apply_defaults: true`, nodes already have these resources and OC's Terraform fails with "already exists" errors (cluster still forms, but Terraform artifacts are empty). - **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster formation via OC, OVN overlay networking works identically to manual clusters. Deploy ovn-central container, enable OVN services, create UPLINK + OVN network, add ovn-chassis roles. HA nginx workload with OVN load balancer tested and working. See `notes/operations-center-guide.md` for full guide. - **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously skipped uploading an ISO if one with the same filename existed on Proxmox. This caused stale ISOs from previous deployments to be silently reused. Fixed: when `--iso` is explicitly provided, the script now deletes the existing ISO and re-uploads the new one. #### Tested limitations - **Inventory is NOT real-time** -- requires explicit `cluster resync` - **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). The failure is OC-specific: the OC agent pushes config via the IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid values (e.g., cron expression where Go duration is expected) crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. - **No cluster member state tracking** -- OC always shows `ready` even for EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat detects failure in ~40s; OC has no equivalent). - **Stale entries** from out-of-band cluster changes persist after resync - **Server removal blocked** if server is part of an OC cluster - **Node failure recovery**: Proxmox hard-stop simulates crash. After restart, node auto-rejoins cluster in ~60s. Containers auto-start. - **`needs_update` tracking is pipeline-based**: OC tracks whether an update was delivered through its pipeline, not just whether versions match. Nodes deployed from the latest ISO are tracked as needing updates even when `version == available_version`. Must deploy from older ISO to work around. - **OVN LB has no health checks**: connection-based hashing distributes traffic to dead backends. Requests to stopped instances return empty. - See `notes/operations-center-guide.md` for full tested OC reference. ### incusos-proxmox doctor and cleanup - **`--doctor`**: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file). - **`--cleanup`**: destroys VMs defined in the config file. - **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior). - **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab config required). Destroys all VMs with `[incusos-lab:managed]` marker. - **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso` and `seed-*.iso` from storage + remotes + cache. - **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default output is concise (step names + results). `--quiet` suppresses everything except warnings and errors. - **`--retries N`**: number of stop+start retries for VMs that fail to boot (port 8443 not reachable). Default: 3. `--retries 0` disables retries. Retries are rarely needed since omitting `force_reboot`. After boot, `fix_scrub_schedule()` proactively heals any scrub_schedule issues. - **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot downloads application sysext (~30-120s depending on CDN speed). Do not reduce — premature retries can corrupt the TPM encryption key permanently. ### IncusOS first-boot sequence Complete lifecycle from Proxmox VM start to port 8443 ready: **Phase 1: ISO boot and installation (~60-85s)** 1. UEFI firmware → IncusOS boot menu → "Starting install of IncusOS to local disk" 2. "Cloning GPT partitions" → progress bar → complete (~876 MiB written) 3. Without `force_reboot`, installer sits at "please remove installation media" **Phase 2: Transition (~15s)** 1. `incusos-proxmox` detects install complete via blockstat (876 MiB written, then idle) 2. Proxmox stops VM, removes ide2 (ISO) and ide3 (seed), sets `boot: order=scsi0` 3. Start VM from disk **Phase 3: First boot from disk (~50s to port 8443)** 1. UEFI → "IncusOS is starting..." 2. `state.LoadOrCreate()` creates `state.txt` with defaults 3. "Auto-generating encryption recovery key" (~10s) 4. "Downloading SecureBoot update" + "Downloading application update" (~33-38s) 5. "Starting application" → "System is ready" — port 8443 now reachable **Total: ~130-215s from VM create to port 8443 ready.** The biggest variable is the application sysext download (~33-38s on first boot; skipped if cached). ### IncusOS crontab bug (upstream issue #843) **Status:** Eliminated in our pipeline by omitting `force_reboot` from seeds. The upstream bug still exists but only triggers with `force_reboot: true`. - **What it is**: a race condition in IncusOS where `ScrubSchedule` ends up empty in `state.txt`, causing `registerJobs()` → `gocron.IsValid("")` to fail and the daemon to exit. Port 8443 never opens (or opens briefly before the daemon crashes). - **Our fix**: omit `force_reboot` from the seed. Without it, there is no SysRq-B intermediate boot, and the race condition does not trigger. Result: **100% success rate** (vs ~50% with `force_reboot` on 4-core VMs). - **Detection**: port 8443 reachable but `scrub_schedule` is empty in `GET /os/1.0/system/storage` → bug hit. `incusos-proxmox` checks this automatically. - **Auto-heal**: `incusos-proxmox` includes `fix_scrub_schedule()` which sets `scrub_schedule` to `"0 4 * * 0"` via `PUT /os/1.0/system/storage` on every deployed node as a safety net. - **TPM corruption risk**: hard-stopping a VM during first boot (while the encryption key is being written) can permanently corrupt the TPM. Error: "zfs load-key: Raw key too short (expected 32)". Only fix is VM destruction and redeployment. The 180s boot timeout in `incusos-proxmox` prevents this. - **On physical hardware**: `force_reboot: true` is still needed (no external orchestrator to remove install media). The bug may occur; recovery is a manual power cycle. ### Proxmox SSH root access — strict rules Root SSH access to the Proxmox host is available for **diagnostics only**. The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`. **MANDATORY RULES — violation of any rule is unacceptable:** 1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM console screenshots via `qm monitor screendump`. No other use without explicit user instruction. 2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869) or VMs in the IncusLab pool that were created by our scripts. 3. **No modifications**: NEVER run any command that modifies, stops, starts, or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`, `pct` commands, `zfs` commands, `systemctl` commands, or ANY write operation on the host. 4. **Do not touch the dev VM**: the user's dev VM runs on the same host. Do not interact with it in any way — do not even `qm status` it. 5. **No config access**: do not read or modify `/etc/pve/`, storage configs, network configs, user/ACL settings, or any host-level configuration. 6. **Transparency**: every SSH command is visible in tool output. If the user rejects a command, do not retry it. 7. **Only during active tests**: only SSH during test runs where test VMs have been deployed, and only to screenshot those test VMs. **Permitted commands (exhaustive list):** ```bash # Take screenshot of a test VM (VMID in 850-869 range) sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ root@ "echo 'screendump /tmp/vm--screen.ppm' | qm monitor " # Retrieve screenshot sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \ root@:/tmp/vm--screen.ppm /tmp/ # Cleanup screenshot on remote sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ root@ "rm -f /tmp/vm--screen.ppm" ``` **Technical notes:** - QEMU's `screendump` is root-only in PVE 9's HMP permission model. API tokens cannot execute it regardless of privileges. - Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng, which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`). - PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`: `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"` PNG output is ~20 KB and can be read directly by Claude Code (multimodal). ## Coding conventions for scripts - **Shell**: bash with `set -euo pipefail` - **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid false exits under `set -e` when the value is 0. - **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern. - **Flags**: support both short (`-d`) and long (`--defaults`) options. - **Defaults**: sane defaults so the script does something useful with zero flags. - **Dry run**: all scripts should support `--dry-run` to preview actions. - **Cert detection order**: files on disk first, CLI command second. - **Error messages**: include actionable remediation steps, not just "failed". - **No hardcoded package managers**: say "install the Incus client" with a link, not "sudo apt install incus". ### Incus networking (OVN) - **Bridge networks are node-local**: each cluster member has its own independent bridge. Instances on the same bridge (same node) can communicate; cross-node instances CANNOT. Each bridge has the same subnet (e.g., 10.0.0.1/24) but they are separate L2 domains. - **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes. Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane + client services + physical uplink network. - **IncusOS OVN services are disabled by default**: must be enabled via the IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection` fails with `db.sock not found`. - **OVN service enable API call**: ```bash incus query :/os/1.0/services/ovn --request PUT --data '{ "config": { "database": "tcp::6642", "enabled": true, "tunnel_address": "", "tunnel_protocol": "geneve" }, "state": {} }' ``` `database` is the **southbound** DB (port 6642), NOT northbound (6641). - **OVN control plane as container**: deploy `ovn-central` package in a Debian container on the cluster. Use proxy devices to expose NB (6641) and SB (6642) ports on the host's LAN IP so all nodes can reach it. - **Setup sequence** (order matters): 1. Deploy OVN control plane container 2. Enable OVN services on ALL IncusOS nodes 3. `incus config set network.ovn.northbound_connection tcp::6641` 4. `incus cluster role add : ovn-chassis` (all nodes) 5. Create physical uplink network (two-step cluster pattern) 6. Create OVN network with `--type=ovn network=UPLINK` - **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC — NOT `ens18`, which is the underlying device name but not exposed to Incus). `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, `ipv4.gateway` is the LAN gateway in CIDR format. - **OVN network isolation**: multiple OVN networks are fully isolated. Instances on different networks cannot communicate, even on the same node. Network peering (`incus network peer create`) enables cross-network routing. - **OVN features tested**: cross-node connectivity, network isolation, ACLs (per-source blocking), network peering, L4 load balancers (connection- based hashing, not round-robin), network forwards (port forwarding to LAN IPs), DNS resolution (per-network, hostname.incus domain). - **LB/forward backends require IP addresses**: `incus network load-balancer backend add` and `incus network forward port add` require the target instance's IP address, NOT its name. Using instance names fails with "Invalid target address". - See `notes/networking-guide.md` for full tutorial with test results. ### Shared storage (iSCSI + lvmcluster) - **iSCSI + lvmcluster** is the IncusOS-native path to shared storage. All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS and enabled via the REST API — no packages to install. - **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN field is `"target"` (NOT `"iqn"` — using the wrong field silently fails). The config lives under `config.targets[]`, and `state` returns the auto-generated `initiator_name`. - **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique `system_id` (1-2000) per node for sanlock host identification. Using 0 or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool creation. - **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately. ~256 MiB overhead for LVM metadata + sanlock lease area. - **Pool creation**: two-step cluster pattern (`--target` per member, then finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`, and `vgchange --lock-start` automatically. - **Migration performance** (tested on 1GbE, 2026-02-23): - Container stop/move/start: **0.12-0.15s** (metadata only) - VM non-live (stop/move/start): **1.8s** (LVM metadata update) - VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer) - Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data) - **Proxy devices don't work for iSCSI**: SendTargets discovery returns container IP in TargetAddress, causing portal mismatch. Use direct network paths (bridge for same-node, macvlan for cross-node). - **Lab target container**: Debian container with `tgt` (userspace iSCSI target) on the cluster — no external hardware needed. Uses dual network: bridge IP for same-node access, macvlan on `mgmt` for cross-node access. - **First live migration after stop/start may fail**: QEMU on destination fails to start (transient sanlock lease issue). Retry succeeds. Non-live migration always works as fallback. - **Hybrid architecture recommended**: local ZFS (`local` pool) for general workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant migration. - See `notes/shared-storage-guide.md` for the full tested walkthrough. ### Migration into Incus - **`incus-migrate`**: official tool for importing disk images, running instances, or physical machines into Incus. - **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2, raw, vdi, vhd formats. Incus accepts raw and qcow2. - **Import workflow**: convert disk → `incus storage volume import` → `incus init --empty --vm` → attach disk → start. - **Container migration**: `docker export` → `incus import` for filesystem- level container migration. Docker volumes must be copied separately. - See `notes/migration-guide.md` for full procedures per source hypervisor. ### UTM support (future) - Design document at `notes/utm-support.md`. - UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation (requires AppleScript or `.utm` bundle generation). - No `blockstat` equivalent -- install detection must use timeout + port polling. - Seed generation already works cross-platform (Phase 3 macOS compatibility). ## Git workflow - Main branch: `main` - Development happens on feature branches - Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`