27 KiB

Raw Blame History

CLAUDE.md - Project context for AI assistants

What this repository is

A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts.

Repository structure

incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator (cross-platform: Linux + macOS)
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
│   ├── lab-test           # Guided lab validation (12 test phases)
│   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
│   ├── ../env             # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
    ├── clustering-guide.md          # Detailed Incus clustering walkthrough
    ├── operations-center-guide.md   # Operations Center provisioning & management
    ├── migration-guide.md           # Migration paths into Incus from other hypervisors
    └── utm-support.md               # UTM support design document (future)

Key technical context

Incus version differences

Debian stable ships Incus 6.0 LTS which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
macOS (Homebrew) and Arch Linux both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package.
incus remote get-client-certificate was added in Incus 6.3+ and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
Always prefer reading ~/.config/incus/client.crt directly from disk. Fall back to the CLI command only as a secondary option.
See notes/incus-version-compatibility.md for full platform matrix and install instructions.

IncusOS flasher-tool

Install: go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
Actual CLI flags: -f/--format, -s/--seed, -c/--channel, -i/--image, -v/--version
There is NO --seed-tar flag -- it's just --seed (or -s).
There is NO --arch flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via --image.
CDN index: https://images.linuxcontainers.org/os/index.json
CDN images: https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz

Seed archives

Tar archives containing YAML files at the root level.
Written to byte offset 2148532224 (the seed partition) in the image.
Alternative: external boot media labeled SEED_DATA attached as CD-ROM.
- ISO 9660 (genisoimage -V SEED_DATA -J -r): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on /dev/sr*.
- FAT image (mkfs.fat -n SEED_DATA): works for USB/block devices but NOT for CD-ROM -- the Linux kernel's sr_mod driver does not expose FAT filesystem labels, so IncusOS cannot find the seed.
Key files: install.yaml, applications.yaml, incus.yaml, operations-center.yaml, network.yaml, update.yaml.

Client certificates

Stored at ~/.config/incus/client.crt and ~/.config/incus/client.key.
Running incus remote list triggers auto-generation if no keypair exists.
For Incus seed: injected under preseed.certificates[] (NOT preseed.server.certificates[]). The InitPreseed.Server field uses yaml:",inline" so its fields (including certificates) are promoted to the top level of the preseed object.
For Operations Center seed: injected under trusted_client_certificates[].
Operations Center requires at least one trusted certificate -- without it, you are locked out after installation.

Proxmox VE deployment

incusos-proxmox reads a YAML config, generates per-VM SEED_DATA images via incusos-seed --format iso, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation.
Config separation: Proxmox connection settings (host, credentials, pool) live in incusos/proxmox.yaml (gitignored). Lab configs (lab-cluster.yaml, etc.) only define VM specs. Merge priority: proxmox.yaml base → lab config proxmox: overlay → CLI flags (--host, --method). Auto-discovery looks for proxmox.yaml in script directory then cwd; override with --proxmox FILE.
API token secret: stored in env file at the repo root (gitignored). Scripts auto-load it on startup by searching for env up the directory tree from the script location. No manual source env needed. If the file is missing and PROXMOX_TOKEN_SECRET is not exported, API commands fail with a clear error message.
Connection methods: SSH (default, ssh root@host qm ...) or API (curl -k https://host:8006/api2/json/... with PVEAPIToken header).
Minimum API privileges for token-based access (role: IncusOSDeployer):
```
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use Sys.Audit
```
Sys.Audit is needed for --resources (host RAM/CPU/uptime via /nodes/<node>/status). It requires a separate ACL on /nodes/<node> since the pool-scoped ACL doesn't cover node-level endpoints.
Required VM settings (getting any wrong causes IncusOS install failure):
- bios=ovmf, machine=q35 -- UEFI boot required
- efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
- tpmstate0: version=v2.0 -- required for disk encryption
- cpu=host -- needed for x86_64_v3 instruction set requirement
- scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
- balloon=0 -- IncusOS manages memory internally
- ide3 -- SEED_DATA ISO 9660 image attached as second CD-ROM
- Minimum 50 GiB disk, minimum 4096 MiB RAM
Disk target: do NOT specify disk-target in the seed for Proxmox VMs. IncusOS does literal string matching (not glob) on disk device IDs. scsi-* does NOT match scsi-0QEMU_QEMU_HARDDISK_drive-scsi0. Omit disk-target entirely and let IncusOS auto-detect (works for single-disk VMs).
Install flow (automated by incusos-proxmox):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + force_reboot: true in seed
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
3. Detect install completion by polling blockstat.scsi0.wr_bytes via API -- when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. Delete ide2 and ide3 -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order
6. Set boot order to order=scsi0 and start from disk
IP detection: IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Verify with direct ping before trusting the result.
force_reboot is required: without force_reboot: true in the seed, IncusOS sits at "please remove installation media" and waits indefinitely. It does NOT halt automatically. force_reboot triggers a guest-level reboot (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
Resource pool isolation: the optional proxmox.pool config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to /pool/<name> instead of /. Setup:
```
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
```
--status command: incusos-proxmox --status config.yaml shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL).
Reconcile on re-runs: when --phase all detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With --yes, defaults to option 1 (safe -- never auto-destroys).
Install idempotency: phase_install checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation.

Multi-lab coexistence

Lab lifecycle: --lab-down config.yaml stops all VMs (Proxmox stop, VMs stay on disk). --lab-up config.yaml starts stopped VMs from disk (refuses to start VMs with install media still attached). These are distinct from --cleanup (which destroys VMs permanently).
Resource awareness: --resources shows Proxmox host RAM, CPU, storage usage, and per-pool allocation. Requires API method.
Lab inventory: --labs scans the pool for managed VMs (by [incusos-lab:managed] marker), groups by config file, and shows per-lab status, VM count, RAM, and disk.
VMID range convention (to avoid collisions between coexisting labs):

Range Lab

400-499 OC-managed nodes

800-809 Single-node labs

900-909 Basic cluster

910-919 OC combined (server + nodes)

920-929 OC server standalone

930-939 Advanced / heterogeneous

Range	Lab
400-499	OC-managed nodes
800-809	Single-node labs
900-909	Basic cluster
910-919	OC combined (server + nodes)
920-929	OC server standalone
930-939	Advanced / heterogeneous

Incus clustering via remotes

Cluster formation is done entirely through the incus CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
No VIP needed: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally.

Pre-clustering: fix core.https_address

IncusOS nodes default to core.https_address: :8443 (wildcard / all interfaces). Clustering requires a specific routable IP so nodes can address each other.

Set the IP on every node BEFORE enabling clustering:

incus config set <remote>: core.https_address <NODE_IP>:8443

Get each node's routable IP via the API:

incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(a) for a in d['environment']['addresses'] \
   if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"

This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based.

Cluster enable (init node)

```
incus cluster enable <remote>: <member-name>
```
Note: this is TWO arguments: <remote>: (trailing colon) and <member-name>. The help text shows [<remote>:] <name> — NOT remote:name as a single arg.
TLS certificate regeneration: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for 127.0.0.1 and ::1, breaking the existing remote.

Fix: remove and re-add the remote to pin the new certificate:

incus remote switch local         # if init remote is current default
incus remote remove <remote>
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate

The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address.

Joining nodes: apply_defaults and the storage pool conflict

Upstream recommendation: use apply_defaults: false for nodes destined to join a cluster. The official IncusOS clustering tutorial states joining servers "cannot have preexisting networks or storage pools defined." With apply_defaults: false, the node still listens on port 8443, still trusts preseed certificates, and the underlying ZFS dataset (local/incus) still exists -- but no Incus storage pool or network metadata is created, so the join process works cleanly.
apply_defaults: true on joining nodes is also functional but requires an 8-command cleanup per node before join (delete pool, network, volumes, profile devices). This is automated in lab-test but adds complexity.
Recommended seed pattern for clusters:
- Bootstrap/init node: apply_defaults: true (needs pool and network)
- Joining nodes: apply_defaults: false (join process creates member-specific entries)
- Standalone nodes: apply_defaults: true (needs pool and network to be functional)

If apply_defaults: true was used, the cleanup before join is:

# 1. Remove config references
incus config unset <remote>: storage.backups_volume
incus config unset <remote>: storage.images_volume
# 2. Delete volumes
incus storage volume delete <remote>:local backups
incus storage volume delete <remote>:local images
# 3. Clear default profile references (pool is "in use" otherwise)
incus profile device remove <remote>:default root
incus profile device remove <remote>:default eth0
# 4. Delete pool and network
incus storage delete <remote>:local
incus network delete <remote>:incusbr0

Join workflow

Generate token (on init node, single argument remote:member-name):
```
incus cluster add <init-remote>:<new-member-name>
```
Join (interactive -- prompts for 5 values):
```
incus cluster join <init-remote>: <joining-remote>:
```
Interactive prompts and correct answers:
1. IP address → accept default (node's IP, already set via core.https_address)
2. Member name → accept default (matches the token)
3. "All existing data is lost" → yes
4. source property for storage pool "local" → local/incus
5. zfs.pool_name property for storage pool "local" → local/incus

Automated (non-interactive):

printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:

After join: the joining node gets a new cluster certificate. Fix the remote (same as init node):

incus remote remove <joining-remote>
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate

Command syntax gotchas

incus cluster enable remote: member-name -- TWO arguments (remote: + name)
incus cluster add remote:member-name -- ONE argument (no space)
incus cluster remove remote:member-name --force -- ONE argument; prompts "yes/no" even with --force, pipe printf "yes\n" for automation
incus cluster evacuate remote:member-name -- ONE argument (no space)
incus cluster restore remote:member-name -- ONE argument (no space)
incus cluster join init-remote: joining-remote: -- TWO arguments (space)
incus storage show remote:pool -- ONE argument (no space)
incus storage show remote:pool --target member -- target flag for member-specific config
incus config set remote: key value -- remote with trailing colon + space
General rule: remote:resource for targeting a resource, remote: (trailing colon) for targeting the server itself

Post-join state

After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations.
lab-test automates cluster formation, workload testing, and migration.

Workload placement and migration

Targeted launch: incus launch images:debian/12 <cluster-remote>:name --target <member>
Cluster-wide visibility: incus list on any member shows all instances.

Container migration: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not.

incus stop <remote>:<instance>
incus move <remote>:<instance> --target <destination-member>
incus start <remote>:<instance>

VM live migration: requires migration.stateful=true (must be set while VM is stopped). Preserves running state with no downtime.
```
incus move <remote>:<instance> --target <destination-member>
```
VM live migration requires limits.cpu as a range (e.g., 0-1), not an integer. Without this, Incus sets QEMU's maxcpus to the host's CPU count (driver_qemu_templates.go: maxcpus = min(cpu.Total, 64)). Different maxcpus values size the ICH9 ACPI CPU hotplug state arrays differently, causing Missing section footer for ICH9LPC on restore. Using a range (pinning syntax) eliminates maxcpus entirely and uses fixed sockets/cores/threads topology — portable across all hosts.
```
# WRONG: integer → maxcpus varies by host → migration fails
incus config set <instance> limits.cpu=2
# RIGHT: range → fixed topology → migration works everywhere
incus config set <instance> limits.cpu=0-1
```
VM live migration works in nested virtualization (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
The vnmi CPUID warning (CPUID[eax=8000000Ah].EDX.vnmi) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration.
Stateful stop/restore (incus stop --stateful + incus start) also requires the limits.cpu range fix. Use incus start --stateless to discard a saved state file that cannot be restored.
VM size.state config: stateful operations require size.state on the root disk (incus config device add <instance> root disk path=/ pool=local size.state=2GiB). Without it, incus stop --stateful fails.
Cluster evacuation: incus cluster evacuate <remote>:<member> --force (ONE argument, like cluster enable and cluster add). Use --action stop if VMs lack the limits.cpu range fix. Restore with incus cluster restore <remote>:<member> --force.
VM agent reconnect: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. incus exec commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should sleep 4 after migration before running incus exec.
Multi-vCPU migration: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., limits.cpu=0-2) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. size.state=4GiB recommended for 3-4 vCPU VMs.
Concurrent migrations: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration.
Active I/O during migration: disk writes and network activity survive live migration transparently. File integrity verified after migration.

Cluster rebalancing: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with migration.stateful=true. Containers are NOT auto-rebalanced.

incus config set <remote>: cluster.rebalance.interval=1     # minutes
incus config set <remote>: cluster.rebalance.threshold=10    # imbalance %
incus config set <remote>: cluster.rebalance.batch=2         # max VMs/run
incus config set <remote>: cluster.rebalance.cooldown=5m     # wait between runs

Node replacement lifecycle: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See notes/clustering-guide.md for step-by-step instructions.
incus cluster remove requires confirmation: even with --force, it prompts "Are you really sure?". Pipe yes for automation:
```
printf "yes\n" | incus cluster remove <remote>:<member> --force
```
See notes/clustering-guide.md for full details and references.

Lab validation (lab-test)

lab-test reads the same YAML config as incusos-proxmox and operates on the VM names defined there (expects incus remotes to exist).
Phases: deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore).
Test instances use names starting with test- for easy cleanup.
The script reports PASS/FAIL/SKIP for each test and prints a summary.

Operations Center

CLI binary: operations-center (installed from GitHub releases or built from source at github.com/FuturFusion/operations-center).
Config directory: ~/.config/operations-center/ (uses same cert format as Incus: copy client.crt and client.key from ~/.config/incus/).
Port: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
Browser access: requires PKCS#12 client certificate (client.pfx) imported into the browser.
OC is under active development (v0.2.2). Commands and APIs may change.
The --doctor command on incusos-proxmox reports whether the CLI is installed.

Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)

No brownfield adoption: nodes must boot from an OC-provisioned ISO.

Token seeds: named, reusable pre-seed configs attached to tokens. YAML must use structured format with section keys (install:, not flat).

operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/preseed.yaml --description "Force reboot for Proxmox"
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

Hybrid deployment (tested, recommended): incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml combines OC auto-registration (from boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
Self-registration: nodes auto-register with OC within ~30s of first boot. Hostname from SEED_DATA is used as the server name.

Cluster formation (tested):

operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC handles: core.https_address → cluster enable → joins → seed application → Terraform config. Adds meshbr0 network.

apply_defaults conflict (tested): if SEED_DATA has apply_defaults: true, nodes already have storage pool/network/cert. OC's Terraform fails with "already exists" errors but the cluster forms successfully. Use apply_defaults: false for OC-managed nodes to avoid this.

Tested limitations

Inventory is NOT real-time -- requires explicit cluster resync
OC reboot breaks OC-managed nodes on Proxmox -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). The failure is OC-specific: the OC agent runs on boot and fails with errors like "invalid crontab expression" (bad update config), "constraint violation" (re-registration of existing server), or Incus stuck at "starting application" (waiting for peers in boot loops). Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
No cluster member state tracking -- OC always shows ready even for EVACUATED/OFFLINE nodes
Stale entries from out-of-band cluster changes persist after resync
Server removal blocked if server is part of an OC cluster
See notes/operations-center-guide.md for full tested OC reference.

incusos-proxmox doctor and cleanup

--doctor: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file).
--cleanup: destroys VMs defined in the config file.
--cleanup --deep: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior).
--cleanup-all: pool-wide cleanup. Only needs proxmox.yaml (no lab config required). Destroys all VMs with [incusos-lab:managed] marker.
--cleanup-all --deep: aggressive blanket delete of ALL IncusOS_*.iso and seed-*.iso from storage + remotes + cache.
--verbose / -v: shows detailed output (tool paths, API calls). Default output is concise (step names + results). --quiet suppresses everything except warnings and errors.

Coding conventions for scripts

Shell: bash with set -euo pipefail
Arithmetic: use var=$((var + 1)) instead of ((var++)) to avoid false exits under set -e when the value is 0.
Colors: support NO_COLOR=1 and TERM=dumb; use setup_colors() pattern.
Flags: support both short (-d) and long (--defaults) options.
Defaults: sane defaults so the script does something useful with zero flags.
Dry run: all scripts should support --dry-run to preview actions.
Cert detection order: files on disk first, CLI command second.
Error messages: include actionable remediation steps, not just "failed".
No hardcoded package managers: say "install the Incus client" with a link, not "sudo apt install incus".

Migration into Incus

incus-migrate: official tool for importing disk images, running instances, or physical machines into Incus.
Disk format conversion: use qemu-img convert between vmdk, qcow2, raw, vdi, vhd formats. Incus accepts raw and qcow2.
Import workflow: convert disk → incus storage volume import → incus init --empty --vm → attach disk → start.
Container migration: docker export → incus import for filesystem- level container migration. Docker volumes must be copied separately.
See notes/migration-guide.md for full procedures per source hypervisor.

UTM support (future)

Design document at notes/utm-support.md.
UTM provides utmctl CLI for start/stop/status but not for VM creation (requires AppleScript or .utm bundle generation).
No blockstat equivalent -- install detection must use timeout + port polling.
Seed generation already works cross-platform (Phase 3 macOS compatibility).

Git workflow

Main branch: main
Development happens on feature branches
Remote: private Gitea at ssh://git@192.168.1.200:2222/maarten/incu-contrib.git

27 KiB Raw Blame History