# CLAUDE.md - Project context for AI assistants ## What this repository is A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts. ## Repository structure ``` incu-contrib/ ├── CLAUDE.md # This file -- project context ├── README.md # Main overview ├── .gitignore ├── incusos/ # IncusOS installation tooling │ ├── README.md # Detailed usage docs │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator │ ├── incusos-proxmox # Declarative Proxmox VM deployment │ ├── lab-test # Guided lab validation (workloads, clustering, migration) │ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material └── clustering-guide.md # Detailed Incus clustering walkthrough ``` ## Key technical context ### Incus version differences - **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu. - **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package. - `incus remote get-client-certificate` was added in **Incus 6.3+** and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path. - Always prefer reading `~/.config/incus/client.crt` directly from disk. Fall back to the CLI command only as a secondary option. - See `notes/incus-version-compatibility.md` for full platform matrix and install instructions. ### IncusOS flasher-tool - Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest` - Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version` - There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`). - There is NO `--arch` flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via `--image`. - CDN index: `https://images.linuxcontainers.org/os/index.json` - CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz` ### Seed archives - Tar archives containing YAML files at the root level. - Written to byte offset 2148532224 (the seed partition) in the image. - Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM. - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on `/dev/sr*`. - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose FAT filesystem labels, so IncusOS cannot find the seed. - Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, `operations-center.yaml`, `network.yaml`, `update.yaml`. ### Client certificates - Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. - Running `incus remote list` triggers auto-generation if no keypair exists. - For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`). The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including `certificates`) are promoted to the top level of the `preseed` object. - For Operations Center seed: injected under `trusted_client_certificates[]`. - Operations Center **requires** at least one trusted certificate -- without it, you are locked out after installation. ### Proxmox VE deployment - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation. - **Config separation**: Proxmox connection settings (host, credentials, pool) live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`, etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`. - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). - **Minimum API privileges** for token-based access: ``` VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use ``` - **Required VM settings** (getting any wrong causes IncusOS install failure): - `bios=ovmf`, `machine=q35` -- UEFI boot required - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys - `tpmstate0`: `version=v2.0` -- required for disk encryption - `cpu=host` -- needed for x86_64_v3 instruction set requirement - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - `balloon=0` -- IncusOS manages memory internally - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - Minimum 50 GiB disk, minimum 4096 MiB RAM - **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. IncusOS does **literal** string matching (not glob) on disk device IDs. `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). - **Install flow** (automated by `incusos-proxmox`): 1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed 2. IncusOS reads seed, installs to disk (scsi0), auto-reboots 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- when disk writes start then stop for 15s (3 stable polls), install is done 4. Stop the VM (Proxmox stop, not guest shutdown) 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order 6. Set boot order to `order=scsi0` and start from disk - **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Verify with direct ping before trusting the result. - **force_reboot is required**: without `force_reboot: true` in the seed, IncusOS sits at "please remove installation media" and waits indefinitely. It does NOT halt automatically. `force_reboot` triggers a guest-level reboot (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does). - **Resource pool isolation**: the optional `proxmox.pool` config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to `/pool/` instead of `/`. Setup: ``` pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy ``` - **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL). - **Reconcile on re-runs**: when `--phase all` detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With `--yes`, defaults to option 1 (safe -- never auto-destroys). - **Install idempotency**: `phase_install` checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation. ### Incus clustering via remotes - **Cluster formation** is done entirely through the `incus` CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access). - **No VIP needed**: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally. #### Pre-clustering: fix core.https_address - IncusOS nodes default to `core.https_address: :8443` (wildcard / all interfaces). Clustering requires a **specific routable IP** so nodes can address each other. - **Set the IP on every node BEFORE enabling clustering:** ```bash incus config set : core.https_address :8443 ``` - Get each node's routable IP via the API: ```bash incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ [print(a) for a in d['environment']['addresses'] \ if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" ``` - This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based. #### Cluster enable (init node) - ```bash incus cluster enable : ``` Note: this is TWO arguments: `:` (trailing colon) and ``. The help text shows `[:] ` — NOT `remote:name` as a single arg. - **TLS certificate regeneration**: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for `127.0.0.1` and `::1`, breaking the existing remote. - **Fix**: remove and re-add the remote to pin the new certificate: ```bash incus remote switch local # if init remote is current default incus remote remove incus remote add https://:8443 --accept-certificate ``` - The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address. #### Joining nodes: the apply_defaults storage pool conflict - **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0` network bridge on each node at install time. When joining a cluster that already has a `local` pool defined, the join wizard asks for `source` and `zfs.pool_name` properties but then tries to set them as cluster-wide config. Incus rejects this because they are **member-specific** keys. - **The fix**: delete the storage pool and network on the joining node before joining. The join process then creates the member-specific pool entry correctly. ```bash # 1. Remove config references incus config unset : storage.backups_volume incus config unset : storage.images_volume # 2. Delete volumes incus storage volume delete :local backups incus storage volume delete :local images # 3. Clear default profile references (pool is "in use" otherwise) incus profile device remove :default root incus profile device remove :default eth0 # 4. Delete pool and network incus storage delete :local incus network delete :incusbr0 ``` #### Join workflow - **Generate token** (on init node, single argument `remote:member-name`): ```bash incus cluster add : ``` - **Join** (interactive -- prompts for 5 values): ```bash incus cluster join : : ``` Interactive prompts and correct answers: 1. IP address → accept default (node's IP, already set via core.https_address) 2. Member name → accept default (matches the token) 3. "All existing data is lost" → `yes` 4. `source` property for storage pool "local" → `local/incus` 5. `zfs.pool_name` property for storage pool "local" → `local/incus` - **Automated (non-interactive):** ```bash printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : ``` - **After join**: the joining node gets a new cluster certificate. Fix the remote (same as init node): ```bash incus remote remove incus remote add https://:8443 --accept-certificate ``` #### Command syntax gotchas - `incus cluster enable remote: member-name` -- TWO arguments (remote: + name) - `incus cluster add remote:member-name` -- ONE argument (no space) - `incus cluster remove remote:member-name --force` -- ONE argument; prompts "yes/no" even with `--force`, pipe `printf "yes\n"` for automation - `incus cluster evacuate remote:member-name` -- ONE argument (no space) - `incus cluster restore remote:member-name` -- ONE argument (no space) - `incus cluster join init-remote: joining-remote:` -- TWO arguments (space) - `incus storage show remote:pool` -- ONE argument (no space) - `incus storage show remote:pool --target member` -- target flag for member-specific config - `incus config set remote: key value` -- remote with trailing colon + space - General rule: `remote:resource` for targeting a resource, `remote:` (trailing colon) for targeting the server itself #### Post-join state - After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations. - **`lab-test`** automates cluster formation, workload testing, and migration. #### Workload placement and migration - **Targeted launch**: `incus launch images:debian/12 :name --target ` - **Cluster-wide visibility**: `incus list` on any member shows all instances. - **Container migration**: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not. ```bash incus stop : incus move : --target incus start : ``` - **VM live migration**: requires `migration.stateful=true` (must be set while VM is stopped). Preserves running state with no downtime. ```bash incus move : --target ``` - **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`), not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`). Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays differently, causing `Missing section footer for ICH9LPC` on restore. Using a range (pinning syntax) eliminates `maxcpus` entirely and uses fixed `sockets/cores/threads` topology — portable across all hosts. ```bash # WRONG: integer → maxcpus varies by host → migration fails incus config set limits.cpu=2 # RIGHT: range → fixed topology → migration works everywhere incus config set limits.cpu=0-1 ``` - **VM live migration works in nested virtualization** (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). - **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration. - **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also requires the `limits.cpu` range fix. Use `incus start --stateless` to discard a saved state file that cannot be restored. - **VM `size.state` config**: stateful operations require `size.state` on the root disk (`incus config device add root disk path=/ pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails. - **Cluster evacuation**: `incus cluster evacuate : --force` (ONE argument, like `cluster enable` and `cluster add`). Use `--action stop` if VMs lack the `limits.cpu` range fix. Restore with `incus cluster restore : --force`. - **VM agent reconnect**: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should `sleep 4` after migration before running `incus exec`. - **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs. - **Concurrent migrations**: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration. - **Active I/O during migration**: disk writes and network activity survive live migration transparently. File integrity verified after migration. - **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with `migration.stateful=true`. Containers are NOT auto-rebalanced. ```bash incus config set : cluster.rebalance.interval=1 # minutes incus config set : cluster.rebalance.threshold=10 # imbalance % incus config set : cluster.rebalance.batch=2 # max VMs/run incus config set : cluster.rebalance.cooldown=5m # wait between runs ``` - **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md` for step-by-step instructions. - **`incus cluster remove` requires confirmation**: even with `--force`, it prompts "Are you really sure?". Pipe `yes` for automation: ```bash printf "yes\n" | incus cluster remove : --force ``` - See `notes/clustering-guide.md` for full details and references. ### Lab validation (lab-test) - **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates on the VM names defined there (expects `incus` remotes to exist). - **Phases:** deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore). - **Test instances** use names starting with `test-` for easy cleanup. - The script reports PASS/FAIL/SKIP for each test and prints a summary. ### Operations Center - **CLI binary:** `operations-center` (installed from GitHub releases or built from source at `github.com/FuturFusion/operations-center`). - **Config directory:** `~/.config/operations-center/` (uses same cert format as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). - **Provisioning workflow:** create token → get provisioned ISO → deploy nodes with that ISO → nodes auto-register with OC → form cluster via OC. - **OC is under active development** (v0.2.2+). Commands and APIs may change. The `--doctor` command on `incusos-proxmox` reports whether the CLI is installed. ### incusos-proxmox doctor and cleanup - **`--doctor`**: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file). - **`--cleanup`**: destroys VMs defined in the config file. - **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior). - **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab config required). Destroys all VMs with `[incusos-lab:managed]` marker. - **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso` and `seed-*.iso` from storage + remotes + cache. - **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default output is concise (step names + results). `--quiet` suppresses everything except warnings and errors. ## Coding conventions for scripts - **Shell**: bash with `set -euo pipefail` - **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid false exits under `set -e` when the value is 0. - **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern. - **Flags**: support both short (`-d`) and long (`--defaults`) options. - **Defaults**: sane defaults so the script does something useful with zero flags. - **Dry run**: all scripts should support `--dry-run` to preview actions. - **Cert detection order**: files on disk first, CLI command second. - **Error messages**: include actionable remediation steps, not just "failed". - **No hardcoded package managers**: say "install the Incus client" with a link, not "sudo apt install incus". ## Git workflow - Main branch: `main` - Development happens on feature branches - Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`