11 KiB

Raw Blame History

CLAUDE.md - Project context for AI assistants

What this repository is

A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts.

Repository structure

incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment
│   ├── lab-test           # Guided lab validation (workloads, clustering, migration)
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material

Key technical context

Incus version differences

Debian stable ships Incus 6.0 LTS which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
macOS (Homebrew) and Arch Linux both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package.
incus remote get-client-certificate was added in Incus 6.3+ and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
Always prefer reading ~/.config/incus/client.crt directly from disk. Fall back to the CLI command only as a secondary option.
See notes/incus-version-compatibility.md for full platform matrix and install instructions.

IncusOS flasher-tool

Install: go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
Actual CLI flags: -f/--format, -s/--seed, -c/--channel, -i/--image, -v/--version
There is NO --seed-tar flag -- it's just --seed (or -s).
There is NO --arch flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via --image.
CDN index: https://images.linuxcontainers.org/os/index.json
CDN images: https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz

Seed archives

Tar archives containing YAML files at the root level.
Written to byte offset 2148532224 (the seed partition) in the image.
Alternative: external boot media labeled SEED_DATA attached as CD-ROM.
- ISO 9660 (genisoimage -V SEED_DATA -J -r): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on /dev/sr*.
- FAT image (mkfs.fat -n SEED_DATA): works for USB/block devices but NOT for CD-ROM -- the Linux kernel's sr_mod driver does not expose FAT filesystem labels, so IncusOS cannot find the seed.
Key files: install.yaml, applications.yaml, incus.yaml, operations-center.yaml, network.yaml, update.yaml.

Client certificates

Stored at ~/.config/incus/client.crt and ~/.config/incus/client.key.
Running incus remote list triggers auto-generation if no keypair exists.
For Incus seed: injected under preseed.certificates[] (NOT preseed.server.certificates[]). The InitPreseed.Server field uses yaml:",inline" so its fields (including certificates) are promoted to the top level of the preseed object.
For Operations Center seed: injected under trusted_client_certificates[].
Operations Center requires at least one trusted certificate -- without it, you are locked out after installation.

Proxmox VE deployment

incusos-proxmox reads a YAML config, generates per-VM SEED_DATA images via incusos-seed --format iso, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation.
Connection methods: SSH (default, ssh root@host qm ...) or API (curl -k https://host:8006/api2/json/... with PVEAPIToken header).

Minimum API privileges for token-based access:

VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use

Required VM settings (getting any wrong causes IncusOS install failure):
- bios=ovmf, machine=q35 -- UEFI boot required
- efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
- tpmstate0: version=v2.0 -- required for disk encryption
- cpu=host -- needed for x86_64_v3 instruction set requirement
- scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
- balloon=0 -- IncusOS manages memory internally
- ide3 -- SEED_DATA ISO 9660 image attached as second CD-ROM
- Minimum 50 GiB disk, minimum 4096 MiB RAM
Disk target: do NOT specify disk-target in the seed for Proxmox VMs. IncusOS does literal string matching (not glob) on disk device IDs. scsi-* does NOT match scsi-0QEMU_QEMU_HARDDISK_drive-scsi0. Omit disk-target entirely and let IncusOS auto-detect (works for single-disk VMs).
Install flow (automated by incusos-proxmox):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + force_reboot: true in seed
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
3. Detect install completion by polling blockstat.scsi0.wr_bytes via API -- when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. Delete ide2 and ide3 -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order
6. Set boot order to order=scsi0 and start from disk
IP detection: IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Verify with direct ping before trusting the result.
force_reboot is required: without force_reboot: true in the seed, IncusOS sits at "please remove installation media" and waits indefinitely. It does NOT halt automatically. force_reboot triggers a guest-level reboot (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
Resource pool isolation: the optional proxmox.pool config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to /pool/<name> instead of /. Setup:
```
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```
--status command: incusos-proxmox --status config.yaml shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL).
Reconcile on re-runs: when --phase all detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With --yes, defaults to option 1 (safe -- never auto-destroys).
Install idempotency: phase_install checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation.

Incus clustering via remotes

Cluster formation is done entirely through the incus CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).

Workflow:

# Enable clustering on first node
incus cluster enable <remote1>: <member-name>

# Generate join token (on cluster, for new member name)
incus cluster add <remote1>: <new-member-name>

# Join second node to cluster
incus cluster join <remote1>: <remote2>:

# Verify
incus cluster list <remote1>:

apply_defaults: true must be set on all nodes that will be clustered -- this ensures matching storage pool (local) and network bridge names, which are required for migration.
After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations.
lab-test automates cluster formation, workload testing, and migration.

Lab validation (lab-test)

lab-test reads the same YAML config as incusos-proxmox and operates on the VM names defined there (expects incus remotes to exist).
Phases: deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore).
Test instances use names starting with test- for easy cleanup.
The script reports PASS/FAIL/SKIP for each test and prints a summary.

Operations Center

CLI binary: operations-center (installed from GitHub releases or built from source at github.com/FuturFusion/operations-center).
Config directory: ~/.config/operations-center/ (uses same cert format as Incus: copy client.crt and client.key from ~/.config/incus/).
Provisioning workflow: create token → get provisioned ISO → deploy nodes with that ISO → nodes auto-register with OC → form cluster via OC.
OC is under active development (v0.2.2+). Commands and APIs may change. The --doctor command on incusos-proxmox reports whether the CLI is installed.

incusos-proxmox doctor and cleanup

--doctor: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
--cleanup --deep: destroys VMs + deletes ISOs/seeds from Proxmox + removes incus remotes + clears local ISO cache.
--cleanup-all: pool-wide cleanup. Destroys all VMs with the [incusos-lab:managed] marker in the configured resource pool.

Coding conventions for scripts

Shell: bash with set -euo pipefail
Arithmetic: use var=$((var + 1)) instead of ((var++)) to avoid false exits under set -e when the value is 0.
Colors: support NO_COLOR=1 and TERM=dumb; use setup_colors() pattern.
Flags: support both short (-d) and long (--defaults) options.
Defaults: sane defaults so the script does something useful with zero flags.
Dry run: all scripts should support --dry-run to preview actions.
Cert detection order: files on disk first, CLI command second.
Error messages: include actionable remediation steps, not just "failed".
No hardcoded package managers: say "install the Incus client" with a link, not "sudo apt install incus".

Git workflow

Main branch: main
Development happens on feature branches
Remote: private Gitea at ssh://git@192.168.1.200:2222/maarten/incu-contrib.git

11 KiB Raw Blame History