48 KiB

Raw Blame History

CLAUDE.md - Project context for AI assistants

What this repository is

A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts.

Repository structure

incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── ansible/               # Ansible playbooks for AWX/Aether lifecycle hooks
│   ├── ansible.cfg        # Project-level Ansible config
│   └── playbooks/
│       ├── post-deploy.yml    # Runs after Aether creates an instance (Incus API)
│       └── decommission.yml   # Runs before Aether deletes an instance
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator (cross-platform: Linux + macOS)
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
│   ├── deploy-awx         # AWX deployment + management on Incus cluster
│   ├── awx-manifests/     # K8s manifests for AWX Operator + instance
│   ├── lab-test           # Guided lab validation (12 test phases)
│   ├── observe-deploy     # Single-VM deploy with rapid console screenshots
│   ├── observe-runs/      # Screenshot output from observe-deploy (gitignored)
│   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
│   ├── ../env             # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
    ├── clustering-guide.md          # Detailed Incus clustering walkthrough
    ├── operations-center-guide.md   # Operations Center provisioning & management
    ├── networking-guide.md          # OVN overlay networking tutorial (bridge + OVN + LAN)
    ├── shared-storage-guide.md      # iSCSI + lvmcluster shared storage (tested)
    ├── production-lab-guide.md      # Manual cluster + OVN + HA (validated end-to-end)
    ├── migration-guide.md           # Migration paths into Incus from other hypervisors
    ├── aether-guide.md              # Aether management platform (deploy, blueprints, API)
    ├── awx-guide.md                 # AWX + Aether Ansible automation
    ├── incus-version-compatibility.md  # Incus versions across platforms
    ├── iso-download-methods.md      # ISO download/customization research
    └── utm-support.md               # UTM support design document (future)

Key technical context

Incus version differences

Debian stable ships Incus 6.0 LTS which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
macOS (Homebrew) and Arch Linux both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package.
incus remote get-client-certificate was added in Incus 6.3+ and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
Always prefer reading ~/.config/incus/client.crt directly from disk. Fall back to the CLI command only as a secondary option.
See notes/incus-version-compatibility.md for full platform matrix and install instructions.

IncusOS flasher-tool

Install: go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
Actual CLI flags: -f/--format, -s/--seed, -c/--channel, -i/--image, -v/--version
There is NO --seed-tar flag -- it's just --seed (or -s).
There is NO --arch flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via --image.
CDN index: https://images.linuxcontainers.org/os/index.json
CDN images: https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz

Seed archives

Tar archives containing YAML files at the root level.
Written to byte offset 2148532224 (the seed partition) in the image.
Alternative: external boot media labeled SEED_DATA attached as CD-ROM.
- ISO 9660 (genisoimage -V SEED_DATA -J -r): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on /dev/sr*.
- FAT image (mkfs.fat -n SEED_DATA): works for USB/block devices but NOT for CD-ROM -- the Linux kernel's sr_mod driver does not expose FAT filesystem labels, so IncusOS cannot find the seed.
Key files: install.yaml, applications.yaml, incus.yaml, operations-center.yaml, network.yaml, update.yaml.

Client certificates

Stored at ~/.config/incus/client.crt and ~/.config/incus/client.key.
Running incus remote list triggers auto-generation if no keypair exists.
For Incus seed: injected under preseed.certificates[] (NOT preseed.server.certificates[]). The InitPreseed.Server field uses yaml:",inline" so its fields (including certificates) are promoted to the top level of the preseed object.
For Operations Center seed: injected under trusted_client_certificates[].
Operations Center requires at least one trusted certificate -- without it, you are locked out after installation.

Proxmox VE deployment

incusos-proxmox reads a YAML config, generates per-VM SEED_DATA images via incusos-seed --format iso, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation.
Config separation: Proxmox connection settings (host, credentials, pool) live in incusos/proxmox.yaml (gitignored). Lab configs (lab-cluster.yaml, etc.) only define VM specs. Merge priority: proxmox.yaml base → lab config proxmox: overlay → CLI flags (--host, --method). Auto-discovery looks for proxmox.yaml in script directory then cwd; override with --proxmox FILE.
API token secret: stored in env file at the repo root (gitignored). Scripts auto-load it on startup by searching for env up the directory tree from the script location. No manual source env needed. If the file is missing and PROXMOX_TOKEN_SECRET is not exported, API commands fail with a clear error message.
Connection methods: SSH (default, ssh root@host qm ...) or API (curl -k https://host:8006/api2/json/... with PVEAPIToken header).
Minimum API privileges for token-based access (role: IncusOSDeployer):
```
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use Sys.Audit
```
Sys.Audit is needed for --resources (host RAM/CPU/uptime via /nodes/<node>/status). It requires a separate ACL on /nodes/<node> since the pool-scoped ACL doesn't cover node-level endpoints.
Required VM settings (getting any wrong causes IncusOS install failure):
- bios=ovmf, machine=q35 -- UEFI boot required
- efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
- tpmstate0: version=v2.0 -- required for disk encryption
- cpu=host -- needed for x86_64_v3 instruction set requirement
- scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
- balloon=0 -- IncusOS manages memory internally
- ide3 -- SEED_DATA ISO 9660 image attached as second CD-ROM
- Minimum 50 GiB disk, minimum 4096 MiB RAM
VLAN tagging: the vlan config key in proxmox.yaml adds a VLAN tag to the VM's NIC (net0: virtio,bridge=vmbr0,tag=69). This places the VM on the tagged VLAN instead of the native/untagged network. The current lab uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the Proxmox VM level only — IncusOS and Incus instances inside the VM are unaware of it. The setting is optional: omit vlan: for untagged access.
Disk target: do NOT specify disk-target in the seed for Proxmox VMs. IncusOS does literal string matching (not glob) on disk device IDs. scsi-* does NOT match scsi-0QEMU_QEMU_HARDDISK_drive-scsi0. Omit disk-target entirely and let IncusOS auto-detect (works for single-disk VMs).
Install flow (automated by incusos-proxmox):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), no force_reboot in seed
2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), then sits at "please remove installation media" prompt
3. Detect install completion by polling blockstat.scsi0.wr_bytes via API -- when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. Delete ide2 and ide3 -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order
6. Set boot order to order=scsi0 and start from disk
IP detection: IncusOS is immutable and has no QEMU guest agent. Two strategies: (1) Static IP (preferred): set ip: ADDR/PREFIX per VM in the lab config + gateway: and dns: in proxmox.yaml. The seed's network.yaml configures the VM's interface at boot. No ARP scan or SSH needed — the IP is known at deploy time. Works across VLANs. (2) ARP-based lookup (fallback for DHCP): get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Only works on the same L2 domain (native LAN, not across VLANs).
force_reboot is NOT used on Proxmox: the seed omits force_reboot. IncusOS sits at "please remove installation media" after install. We detect completion via blockstat (876 MiB written, then idle), stop the VM externally, remove media, and start from disk. This avoids the crontab race condition (issue #843). On physical hardware, force_reboot: true is still needed (no external orchestrator).
Resource pool isolation: the optional proxmox.pool config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to /pool/<name> instead of /. Setup:
```
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
```
--status command: incusos-proxmox --status config.yaml shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL).
Reconcile on re-runs: when --phase all detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With --yes, defaults to option 1 (safe -- never auto-destroys).
Install idempotency: phase_install checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation.

Multi-lab coexistence

Lab lifecycle: --lab-down config.yaml stops all VMs (Proxmox stop, VMs stay on disk). --lab-up config.yaml starts stopped VMs from disk (refuses to start VMs with install media still attached). These are distinct from --cleanup (which destroys VMs permanently).
Resource awareness: --resources shows Proxmox host RAM, CPU, storage usage, and per-pool allocation. Requires API method.
Lab inventory: --labs scans the pool for managed VMs (by [incusos-lab:managed] marker), groups by config file, and shows per-lab status, VM count, RAM, and disk.
VMID range convention (to avoid collisions between coexisting labs):

Range Lab

400-499 OC-managed nodes

800-809 Single-node labs

900-909 Basic cluster

910-919 OC combined (server + nodes)

920-929 OC server standalone

930-939 Advanced / heterogeneous

Range	Lab
400-499	OC-managed nodes
800-809	Single-node labs
900-909	Basic cluster
910-919	OC combined (server + nodes)
920-929	OC server standalone
930-939	Advanced / heterogeneous

Resource constraints for multi-lab

RAM is the bottleneck: each IncusOS VM needs 4 GiB minimum. A 3-node cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource where you can actually run out.
Storage is not a concern: Proxmox ZFS uses thin provisioning by default (sparse in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. LZ4 compression provides ~1.5-1.8x ratio on OS data.
CPU is plentiful: 4 cores per VM, 20 cores on host. Multiple labs can share CPUs without contention.
Pre-deploy checks: incusos-proxmox warns during preflight if requested RAM exceeds available host RAM (API method only).
--lab-up auto-deploy: if no VMs exist yet, --lab-up offers to run the full deploy pipeline (auto-accepts with --yes).
--resources actual disk: shows actual vs allocated disk for pool VMs (via storage content API) and notes thin provisioning for ZFS/LVM-thin.

Incus clustering via remotes

Cluster formation is done entirely through the incus CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
No VIP needed: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally.

Pre-clustering: fix core.https_address

IncusOS nodes default to core.https_address: :8443 (wildcard / all interfaces). Clustering requires a specific routable IP so nodes can address each other.

Set the IP on every node BEFORE enabling clustering:

incus config set <remote>: core.https_address <NODE_IP>:8443

Get each node's routable IP via the API:

incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(a) for a in d['environment']['addresses'] \
   if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"

This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based.

Cluster enable (init node)

```
incus cluster enable <remote>: <member-name>
```
Note: this is TWO arguments: <remote>: (trailing colon) and <member-name>. The help text shows [<remote>:] <name> — NOT remote:name as a single arg.
TLS certificate regeneration: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for 127.0.0.1 and ::1, breaking the existing remote.

Fix: remove and re-add the remote to pin the new certificate:

incus remote switch local         # if init remote is current default
incus remote remove <remote>
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate

The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address.

Joining nodes: apply_defaults and the storage pool conflict

Upstream recommendation: use apply_defaults: false for nodes destined to join a cluster. The official IncusOS clustering tutorial states joining servers "cannot have preexisting networks or storage pools defined." With apply_defaults: false, the node still listens on port 8443, still trusts preseed certificates, and the underlying ZFS dataset (local/incus) still exists -- but no Incus storage pool or network metadata is created, so the join process works cleanly.
apply_defaults: true on joining nodes is also functional but requires an 8-command cleanup per node before join (delete pool, network, volumes, profile devices). This is automated in lab-test but adds complexity.
Recommended seed pattern for clusters:
- Bootstrap/init node: apply_defaults: true (needs pool and network)
- Joining nodes: apply_defaults: false (join process creates member-specific entries)
- Standalone nodes: apply_defaults: true (needs pool and network to be functional)

If apply_defaults: true was used, the cleanup before join is:

# 1. Remove config references
incus config unset <remote>: storage.backups_volume
incus config unset <remote>: storage.images_volume
# 2. Delete volumes
incus storage volume delete <remote>:local backups
incus storage volume delete <remote>:local images
# 3. Clear default profile references (pool is "in use" otherwise)
incus profile device remove <remote>:default root
incus profile device remove <remote>:default eth0
# 4. Delete pool and network
incus storage delete <remote>:local
incus network delete <remote>:incusbr0

Join workflow

Generate token (on init node, single argument remote:member-name):
```
incus cluster add <init-remote>:<new-member-name>
```
Join (interactive -- prompts for 5 values):
```
incus cluster join <init-remote>: <joining-remote>:
```
Interactive prompts and correct answers:
1. IP address → accept default (node's IP, already set via core.https_address)
2. Member name → accept default (matches the token)
3. "All existing data is lost" → yes
4. source property for storage pool "local" → local/incus
5. zfs.pool_name property for storage pool "local" → local/incus

Automated (non-interactive):

printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:

After join: the joining node gets a new cluster certificate. Fix the remote (same as init node):

incus remote remove <joining-remote>
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate

Command syntax gotchas

incus cluster enable remote: member-name -- TWO arguments (remote: + name)
incus cluster add remote:member-name -- ONE argument (no space)
incus cluster remove remote:member-name --force -- ONE argument; prompts "yes/no" even with --force, pipe printf "yes\n" for automation
incus cluster evacuate remote:member-name -- ONE argument (no space)
incus cluster restore remote:member-name -- ONE argument (no space)
incus cluster join init-remote: joining-remote: -- TWO arguments (space)
incus storage show remote:pool -- ONE argument (no space)
incus storage show remote:pool --target member -- target flag for member-specific config
incus config set remote: key value -- remote with trailing colon + space
General rule: remote:resource for targeting a resource, remote: (trailing colon) for targeting the server itself

Post-join state

After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations.
lab-test automates cluster formation, workload testing, and migration.

Workload placement and migration

Targeted launch: incus launch images:debian/12 <cluster-remote>:name --target <member>
Cluster-wide visibility: incus list on any member shows all instances.

Container migration: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not.

incus stop <remote>:<instance>
incus move <remote>:<instance> --target <destination-member>
incus start <remote>:<instance>

VM live migration: requires migration.stateful=true (must be set while VM is stopped). Preserves running state with no downtime.
```
incus move <remote>:<instance> --target <destination-member>
```
VM live migration requires limits.cpu as a range (e.g., 0-1), not an integer. Without this, Incus sets QEMU's maxcpus to the host's CPU count (driver_qemu_templates.go: maxcpus = min(cpu.Total, 64)). Different maxcpus values size the ICH9 ACPI CPU hotplug state arrays differently, causing Missing section footer for ICH9LPC on restore. Using a range (pinning syntax) eliminates maxcpus entirely and uses fixed sockets/cores/threads topology — portable across all hosts.
```
# WRONG: integer → maxcpus varies by host → migration fails
incus config set <instance> limits.cpu=2
# RIGHT: range → fixed topology → migration works everywhere
incus config set <instance> limits.cpu=0-1
```
VM live migration works in nested virtualization (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
The vnmi CPUID warning (CPUID[eax=8000000Ah].EDX.vnmi) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration.
Stateful stop/restore (incus stop --stateful + incus start) also requires the limits.cpu range fix. Use incus start --stateless to discard a saved state file that cannot be restored.
VM size.state config: stateful operations require size.state on the root disk (incus config device add <instance> root disk path=/ pool=local size.state=2GiB). Without it, incus stop --stateful fails.
Cluster evacuation: incus cluster evacuate <remote>:<member> --force (ONE argument, like cluster enable and cluster add). Use --action stop if VMs lack the limits.cpu range fix. Restore with incus cluster restore <remote>:<member> --force.
VM agent reconnect: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. incus exec commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should sleep 4 after migration before running incus exec.
Multi-vCPU migration: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., limits.cpu=0-2) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. size.state=4GiB recommended for 3-4 vCPU VMs.
Concurrent migrations: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration.
Active I/O during migration: disk writes and network activity survive live migration transparently. File integrity verified after migration.

Cluster rebalancing: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with migration.stateful=true. Containers are NOT auto-rebalanced.

incus config set <remote>: cluster.rebalance.interval=1     # minutes
incus config set <remote>: cluster.rebalance.threshold=10    # imbalance %
incus config set <remote>: cluster.rebalance.batch=2         # max VMs/run
incus config set <remote>: cluster.rebalance.cooldown=5m     # wait between runs

Node replacement lifecycle: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See notes/clustering-guide.md for step-by-step instructions.
incus cluster remove requires confirmation: even with --force, it prompts "Are you really sure?". Pipe yes for automation:
```
printf "yes\n" | incus cluster remove <remote>:<member> --force
```
See notes/clustering-guide.md for full details and references.

Lab validation (lab-test)

lab-test reads the same YAML config as incusos-proxmox and operates on the VM names defined there (expects incus remotes to exist).
Phases: deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore).
Test instances use names starting with test- for easy cleanup.
The script reports PASS/FAIL/SKIP for each test and prints a summary.

Operations Center

CLI binary: operations-center (installed from GitHub releases or built from source at github.com/FuturFusion/operations-center).
Config directory: ~/.config/operations-center/ (uses same cert format as Incus: copy client.crt and client.key from ~/.config/incus/).
Port: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
Browser access: requires PKCS#12 client certificate (client.pfx) imported into the browser. Web UI is a React SPA at /ui/.
OC is under active development (v0.3.0). Commands and APIs may change. Use operations-center --version (not version subcommand) to check.
OC CLI does NOT support remote: suffix syntax: unlike the Incus CLI, the OC CLI requires operations-center remote switch NAME before running commands. operations-center admin os show oc-lab: fails with "Invalid number of arguments".
v0.3.0 new commands: provisioning channel (add/list/show), cluster update/update-certificate/rename, expanded inventory (network-acl, address-sets, load-balancers, peers, zones, integrations, forwards, storage-buckets, storage-volumes), inventory query (cross-resource tree view with filters and Go templates), system certificate set.
The --doctor command on incusos-proxmox reports whether the CLI is installed.

Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)

No brownfield adoption: nodes must boot from an OC-provisioned ISO.

Token seeds: named, reusable pre-seed configs attached to tokens. YAML must use structured format with section keys (install:, not flat). No force_reboot in token seeds for Proxmox (same fix as standard deploys).

operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/preseed.yaml --description "No force_reboot for Proxmox"
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable

Hybrid deployment (tested, recommended): incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml combines OC auto-registration (from boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
Self-registration: nodes auto-register with OC within ~30s of first boot. Hostname from SEED_DATA is used as the server name.
needs_update blocker (critical discovery, 2026-02-23): OC requires all nodes to have needs_update: false before provisioning cluster add succeeds. Nodes deployed from an ISO matching the latest OC update version are tracked as needs_update: true because the OS was never delivered through OC's update pipeline. The needs_update flag is server-side computed (not a simple version comparison) and cannot be overridden via REST API PUT. Solution: generate the ISO from an older channel (--channel old-stable) so nodes start with an older version. OC then pushes the latest update through its pipeline, clearing the flag. Use provisioning update assign-channels to control which versions are in which channels.
Cluster formation (tested, 2026-02-23):
```
# Use empty app config if cert already injected via SEED_DATA
echo '{}' > /tmp/oc-app-config.yaml
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```
OC handles: core.https_address → cluster enable → joins → storage pool
- network creation → Terraform config. Adds meshbr0 network. If the client cert is already in the trust store (from SEED_DATA), use an empty {} app config to avoid "Certificate already in trust store" Terraform error. The cluster forms successfully either way.
apply_defaults: false is recommended (tested): use apply_defaults: false for OC-managed nodes. OC's Terraform handles storage pool, network, and cert creation cleanly. With apply_defaults: true, nodes already have these resources and OC's Terraform fails with "already exists" errors (cluster still forms, but Terraform artifacts are empty).
OC-managed cluster with OVN: fully tested (2026-02-23). After cluster formation via OC, OVN overlay networking works identically to manual clusters. Deploy ovn-central container, enable OVN services, create UPLINK + OVN network, add ovn-chassis roles. HA nginx workload with OVN load balancer tested and working. See notes/operations-center-guide.md for full guide.
ISO upload skip bug (fixed, 2026-02-23): incusos-proxmox previously skipped uploading an ISO if one with the same filename existed on Proxmox. This caused stale ISOs from previous deployments to be silently reused. Fixed: when --iso is explicitly provided, the script now deletes the existing ISO and re-uploads the new one.

Tested limitations

Inventory is NOT real-time -- requires explicit cluster resync
OC reboot breaks OC-managed nodes on Proxmox -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). The failure is OC-specific: the OC agent pushes config via the IncusOS REST API that gets persisted to state.txt. On reboot, invalid values (e.g., cron expression where Go duration is expected) crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
No cluster member state tracking -- OC always shows ready even for EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat detects failure in ~40s; OC has no equivalent).
Stale entries from out-of-band cluster changes persist after resync
Server removal blocked if server is part of an OC cluster
Node failure recovery: Proxmox hard-stop simulates crash. After restart, node auto-rejoins cluster in ~60s. Containers auto-start.
needs_update tracking is pipeline-based: OC tracks whether an update was delivered through its pipeline, not just whether versions match. Nodes deployed from the latest ISO are tracked as needing updates even when version == available_version. Must deploy from older ISO to work around.
OVN LB has no health checks: connection-based hashing distributes traffic to dead backends. Requests to stopped instances return empty.
See notes/operations-center-guide.md for full tested OC reference.

incusos-proxmox doctor and cleanup

--doctor: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file).
--cleanup: destroys VMs defined in the config file.
--cleanup --deep: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior).
--cleanup-all: pool-wide cleanup. Only needs proxmox.yaml (no lab config required). Destroys all VMs with [incusos-lab:managed] marker.
--cleanup-all --deep: aggressive blanket delete of ALL IncusOS_*.iso and seed-*.iso from storage + remotes + cache.
--verbose / -v: shows detailed output (tool paths, API calls). Default output is concise (step names + results). --quiet suppresses everything except warnings and errors.
--retries N: number of stop+start retries for VMs that fail to boot (port 8443 not reachable). Default: 3. --retries 0 disables retries. Retries are rarely needed since omitting force_reboot. After boot, fix_scrub_schedule() proactively heals any scrub_schedule issues.
Boot timeout is 180s (60s initial sleep + 120s polling): first boot downloads application sysext (~30-120s depending on CDN speed). Do not reduce — premature retries can corrupt the TPM encryption key permanently.

IncusOS disk resize (Proxmox VMs)

Disk grow is non-destructive: Proxmox qm resize only grows disks.
Partition 11 auto-expands: systemd-repart runs in initrd on every boot. The local-data partition has no SizeMaxBytes, so it fills all remaining space automatically.
ZFS pool does NOT auto-expand: IncusOS creates the pool without autoexpand=on. After partition grow, zpool list shows the correct EXPANDSZ but the pool stays at its original size.
Manual expansion via privileged container: create a container with security.privileged=true, pass /dev/zfs (unix-char) and /dev/sda11 (unix-block), install zfsutils-linux, create a symlink at /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11 → /dev/sda11, then run zpool online -e local scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11. Clean up the container after expansion. Full procedure in notes/aether-guide.md.
Warning: running zpool online -e with a non-existent device path inside the container causes pool SUSPENSION. Recovery: Proxmox stop/start (ZFS re-imports cleanly, no data loss).

IncusOS first-boot sequence

Complete lifecycle from Proxmox VM start to port 8443 ready:

Phase 1: ISO boot and installation (~60-85s)

UEFI firmware → IncusOS boot menu → "Starting install of IncusOS to local disk"
"Cloning GPT partitions" → progress bar → complete (~876 MiB written)
Without force_reboot, installer sits at "please remove installation media"

Phase 2: Transition (~15s)

incusos-proxmox detects install complete via blockstat (876 MiB written, then idle)
Proxmox stops VM, removes ide2 (ISO) and ide3 (seed), sets boot: order=scsi0
Start VM from disk

Phase 3: First boot from disk (~50s to port 8443)

UEFI → "IncusOS is starting..."
state.LoadOrCreate() creates state.txt with defaults
"Auto-generating encryption recovery key" (~10s)
"Downloading SecureBoot update" + "Downloading application update" (~33-38s)
"Starting application" → "System is ready" — port 8443 now reachable

Total: ~130-215s from VM create to port 8443 ready. The biggest variable is the application sysext download (~33-38s on first boot; skipped if cached).

IncusOS crontab bug (upstream issue #843)

Status: Eliminated in our pipeline by omitting force_reboot from seeds. The upstream bug still exists but only triggers with force_reboot: true.

What it is: a race condition in IncusOS where ScrubSchedule ends up empty in state.txt, causing registerJobs() → gocron.IsValid("") to fail and the daemon to exit. Port 8443 never opens (or opens briefly before the daemon crashes).
Our fix: omit force_reboot from the seed. Without it, there is no SysRq-B intermediate boot, and the race condition does not trigger. Result: 100% success rate (vs ~50% with force_reboot on 4-core VMs).
Detection: port 8443 reachable but scrub_schedule is empty in GET /os/1.0/system/storage → bug hit. incusos-proxmox checks this automatically.
Auto-heal: incusos-proxmox includes fix_scrub_schedule() which sets scrub_schedule to "0 4 * * 0" via PUT /os/1.0/system/storage on every deployed node as a safety net.
TPM corruption risk: hard-stopping a VM during first boot (while the encryption key is being written) can permanently corrupt the TPM. Error: "zfs load-key: Raw key too short (expected 32)". Only fix is VM destruction and redeployment. The 180s boot timeout in incusos-proxmox prevents this.
On physical hardware: force_reboot: true is still needed (no external orchestrator to remove install media). The bug may occur; recovery is a manual power cycle.

Proxmox SSH root access — strict rules

Root SSH access to the Proxmox host is available for diagnostics only. The password is stored in the env file as PROXMOX_ROOT_PASSWORD.

MANDATORY RULES — violation of any rule is unacceptable:

Screenshots only: the ONLY permitted use of root SSH is taking VM console screenshots via qm monitor <vmid> screendump. No other use without explicit user instruction.
Test VMs only: only screenshot VMs in the test VMID range (850-869) or VMs in the IncusLab pool that were created by our scripts.
No modifications: NEVER run any command that modifies, stops, starts, or deletes any VM. No qm set, qm stop, qm start, qm destroy, pct commands, zfs commands, systemctl commands, or ANY write operation on the host.
Do not touch the dev VM: the user's dev VM runs on the same host. Do not interact with it in any way — do not even qm status it.
No config access: do not read or modify /etc/pve/, storage configs, network configs, user/ACL settings, or any host-level configuration.
Transparency: every SSH command is visible in tool output. If the user rejects a command, do not retry it.
Only during active tests: only SSH during test runs where test VMs have been deployed, and only to screenshot those test VMs.

Permitted commands (exhaustive list):

# Take screenshot of a test VM (VMID in 850-869 range)
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"

# Retrieve screenshot
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
    root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/

# Cleanup screenshot on remote
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"

Technical notes:

QEMU's screendump is root-only in PVE 9's HMP permission model. API tokens cannot execute it regardless of privileges.
Use .ppm format — PNG (-f png) requires QEMU compiled with libpng, which PVE 9.1 does not have (Error: Enable PNG support with libpng).
PPM files are ~3 MB (1280x800). Convert to PNG with python3-pil: python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')" PNG output is ~20 KB and can be read directly by Claude Code (multimodal).

Coding conventions for scripts

Shell: bash with set -euo pipefail
Arithmetic: use var=$((var + 1)) instead of ((var++)) to avoid false exits under set -e when the value is 0.
Colors: support NO_COLOR=1 and TERM=dumb; use setup_colors() pattern.
Flags: support both short (-d) and long (--defaults) options.
Defaults: sane defaults so the script does something useful with zero flags.
Dry run: all scripts should support --dry-run to preview actions.
Cert detection order: files on disk first, CLI command second.
Error messages: include actionable remediation steps, not just "failed".
No hardcoded package managers: say "install the Incus client" with a link, not "sudo apt install incus".

Incus networking (OVN)

Bridge networks are node-local: each cluster member has its own independent bridge. Instances on the same bridge (same node) can communicate; cross-node instances CANNOT. Each bridge has the same subnet (e.g., 10.0.0.1/24) but they are separate L2 domains.
OVN provides cross-node L2 overlay: uses Geneve tunnels between nodes. Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
- client services + physical uplink network.
IncusOS OVN services are disabled by default: must be enabled via the IncusOS REST API (/os/1.0/services/ovn) on EVERY node before configuring Incus OVN settings. Without this, incus config set network.ovn.northbound_connection fails with db.sock not found.

OVN service enable API call:

incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:<SB_HOST>:6642",
    "enabled": true,
    "tunnel_address": "<THIS_NODE_LAN_IP>",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

database is the southbound DB (port 6642), NOT northbound (6641).

OVN control plane as container: deploy ovn-central package in a Debian container on the cluster. Use proxy devices to expose NB (6641) and SB (6642) ports on the host's LAN IP so all nodes can reach it.
Setup sequence (order matters):
1. Deploy OVN control plane container
2. Enable OVN services on ALL IncusOS nodes
3. incus config set network.ovn.northbound_connection tcp:<host>:6641
4. incus cluster role add <remote>:<member> ovn-chassis (all nodes)
5. Create physical uplink network (two-step cluster pattern)
6. Create OVN network with --type=ovn network=UPLINK
Physical uplink network: uses parent=mgmt (IncusOS management NIC — NOT ens18, which is the underlying device name but not exposed to Incus). ipv4.ovn.ranges reserves LAN IPs for OVN router external addresses, ipv4.gateway is the LAN gateway in CIDR format.
OVN network isolation: multiple OVN networks are fully isolated. Instances on different networks cannot communicate, even on the same node. Network peering (incus network peer create) enables cross-network routing.
OVN features tested: cross-node connectivity, network isolation, ACLs (per-source blocking), network peering, L4 load balancers (connection- based hashing, not round-robin), network forwards (port forwarding to LAN IPs), DNS resolution (per-network, hostname.incus domain).
LB/forward backends require IP addresses: incus network load-balancer backend add and incus network forward port add require the target instance's IP address, NOT its name. Using instance names fails with "Invalid target address".
See notes/networking-guide.md for full tutorial with test results.

Shared storage (iSCSI + lvmcluster)

iSCSI + lvmcluster is the IncusOS-native path to shared storage. All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS and enabled via the REST API — no packages to install.
IncusOS iSCSI service API: /os/1.0/services/iscsi. The target IQN field is "target" (NOT "iqn" — using the wrong field silently fails). The config lives under config.targets[], and state returns the auto-generated initiator_name.
IncusOS LVM service API: /os/1.0/services/lvm. Requires unique system_id (1-2000) per node for sanlock host identification. Using 0 or omitting it causes "Invalid host_id 0, use 1-2000" during pool creation.
lvmcluster driver: uses thick provisioning (no thin, no snapshots on custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately. ~256 MiB overhead for LVM metadata + sanlock lease area.
Pool creation: two-step cluster pattern (--target per member, then finalize without --target). Incus handles pvcreate, vgcreate --shared, and vgchange --lock-start automatically.
Migration performance (tested on 1GbE, 2026-02-23):
- Container stop/move/start: 0.12-0.15s (metadata only)
- VM non-live (stop/move/start): 1.8s (LVM metadata update)
- VM live migration: ~6s (1 GiB RAM at ~141 MB/s, no disk transfer)
- Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data)
Proxy devices don't work for iSCSI: SendTargets discovery returns container IP in TargetAddress, causing portal mismatch. Use direct network paths (bridge for same-node, macvlan for cross-node).
Lab target container: Debian container with tgt (userspace iSCSI target) on the cluster — no external hardware needed. Uses dual network: bridge IP for same-node access, macvlan on mgmt for cross-node access.
First live migration after stop/start may fail: QEMU on destination fails to start (transient sanlock lease issue). Retry succeeds. Non-live migration always works as fallback.
Hybrid architecture recommended: local ZFS (local pool) for general workloads + shared lvmcluster (shared pool) for HA VMs needing instant migration.
See notes/shared-storage-guide.md for the full tested walkthrough.

Migration into Incus

incus-migrate: official tool for importing disk images, running instances, or physical machines into Incus.
Disk format conversion: use qemu-img convert between vmdk, qcow2, raw, vdi, vhd formats. Incus accepts raw and qcow2.
Import workflow: convert disk → incus storage volume import → incus init --empty --vm → attach disk → start.
Container migration: docker export → incus import for filesystem- level container migration. Docker volumes must be copied separately.
See notes/migration-guide.md for full procedures per source hypervisor.

UTM support (future)

Design document at notes/utm-support.md.
UTM provides utmctl CLI for start/stop/status but not for VM creation (requires AppleScript or .utm bundle generation).
No blockstat equivalent -- install detection must use timeout + port polling.
Seed generation already works cross-platform (Phase 3 macOS compatibility).

AWX integration (Ansible automation for Aether)

AWX is the open-source Ansible automation platform (upstream of Ansible Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
Lab VM: awx on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
AWX URL: http://192.168.102.161:30080 — exposed via K3s NodePort. Traefik ingress returns 404 for IP-based access; use NodePort directly.
deploy-awx script manages the full lifecycle: --deploy, --status, --heal, --configure, --join-aether, --cleanup, --doctor.
K8s manifests in incusos/awx-manifests/ (operator + AWX CR via kustomize).
Ansible playbooks in ansible/ directory:
- playbooks/post-deploy.yml -- runs after Aether creates an instance
- playbooks/decommission.yml -- runs before Aether deletes an instance
Aether extra vars use ffsdn_ prefix: Aether passes ffsdn_instance_name, ffsdn_instance_ip, ffsdn_cluster_id, ffsdn_cluster_name, ffsdn_deployed_by, ffsdn_image_os, ffsdn_image_release, ffsdn_image_alias. It does NOT pass vm_name, vm_ip, environment, owner, or cost_center (the original plan assumed these).
Playbook pattern — Incus REST API, not SSH: AWX cannot SSH to containers on incusbr0 (bridge subnet not routable from management VLAN, IncusOS nftables blocks inbound forwarding). Playbooks use the Incus REST API (uri module with client cert) for file push + exec. The cert is at /runner/project/incus-client.crt during job execution (AWX EE mounts projects at /runner/project/, NOT /var/lib/awx/projects/).
Manual project (local_path): AWX EE containers cannot reach the private git repo. Use scm_type: "" with local_path: "incus-contrib" and push playbooks directly to the AWX task pod at /var/lib/awx/projects/incus-contrib/playbooks/.
Lifecycle hooks: post-deploy failure triggers auto-rollback (instance deleted). Decommission failure does NOT block deletion.
Aether cluster AWX config API bug: PUT /api/clusters/{id}/awx-config returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL UPDATE on the clusters table from within the Aether container.
Self-referencing vars cause infinite recursion in Ansible: patterns like vm_ip: "{{ vm_ip | default('') }}" cause AnsibleUndefinedVariable recursive loop when the variable is not provided. Use ffsdn_* vars directly without redefining them.
Ansible environment is a reserved keyword: using it as an extra var resolves to [] instead of the string value.
See notes/awx-guide.md for full deployment guide and troubleshooting.

Git workflow

Main branch: main
Development happens on feature branches
Remote: private Gitea at ssh://git@192.168.1.200:2222/maarten/incu-contrib.git

48 KiB Raw Blame History