49 KiB

Raw Blame History

CLAUDE.md - Project context for AI assistants

What this repository is

A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts.

Repository structure

incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator (cross-platform: Linux + macOS)
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
│   ├── lab-test           # Guided lab validation (12 test phases)
│   ├── observe-deploy     # Single-VM deploy with rapid console screenshots
│   ├── observe-runs/      # Screenshot output from observe-deploy (gitignored)
│   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
│   ├── ../env             # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
    ├── clustering-guide.md          # Detailed Incus clustering walkthrough
    ├── operations-center-guide.md   # Operations Center provisioning & management
    ├── networking-guide.md            # OVN overlay networking tutorial (bridge + OVN + LAN)
    ├── migration-guide.md           # Migration paths into Incus from other hypervisors
    └── utm-support.md               # UTM support design document (future)

Key technical context

Incus version differences

Debian stable ships Incus 6.0 LTS which is significantly behind upstream. The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
macOS (Homebrew) and Arch Linux both track latest upstream (currently 6.21). macOS is client-only by design; Arch has no client-only split package.
incus remote get-client-certificate was added in Incus 6.3+ and does not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
Always prefer reading ~/.config/incus/client.crt directly from disk. Fall back to the CLI command only as a secondary option.
See notes/incus-version-compatibility.md for full platform matrix and install instructions.

IncusOS flasher-tool

Install: go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
Actual CLI flags: -f/--format, -s/--seed, -c/--channel, -i/--image, -v/--version
There is NO --seed-tar flag -- it's just --seed (or -s).
There is NO --arch flag -- architecture is determined by the downloaded image. For cross-arch builds, download the image manually and pass via --image.
CDN index: https://images.linuxcontainers.org/os/index.json
CDN images: https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz

Seed archives

Tar archives containing YAML files at the root level.
Written to byte offset 2148532224 (the seed partition) in the image.
Alternative: external boot media labeled SEED_DATA attached as CD-ROM.
- ISO 9660 (genisoimage -V SEED_DATA -J -r): preferred for CD-ROM devices. Volume labels are correctly detected by the kernel on /dev/sr*.
- FAT image (mkfs.fat -n SEED_DATA): works for USB/block devices but NOT for CD-ROM -- the Linux kernel's sr_mod driver does not expose FAT filesystem labels, so IncusOS cannot find the seed.
Key files: install.yaml, applications.yaml, incus.yaml, operations-center.yaml, network.yaml, update.yaml.

Client certificates

Stored at ~/.config/incus/client.crt and ~/.config/incus/client.key.
Running incus remote list triggers auto-generation if no keypair exists.
For Incus seed: injected under preseed.certificates[] (NOT preseed.server.certificates[]). The InitPreseed.Server field uses yaml:",inline" so its fields (including certificates) are promoted to the top level of the preseed object.
For Operations Center seed: injected under trusted_client_certificates[].
Operations Center requires at least one trusted certificate -- without it, you are locked out after installation.

Proxmox VE deployment

incusos-proxmox reads a YAML config, generates per-VM SEED_DATA images via incusos-seed --format iso, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation.
Config separation: Proxmox connection settings (host, credentials, pool) live in incusos/proxmox.yaml (gitignored). Lab configs (lab-cluster.yaml, etc.) only define VM specs. Merge priority: proxmox.yaml base → lab config proxmox: overlay → CLI flags (--host, --method). Auto-discovery looks for proxmox.yaml in script directory then cwd; override with --proxmox FILE.
API token secret: stored in env file at the repo root (gitignored). Scripts auto-load it on startup by searching for env up the directory tree from the script location. No manual source env needed. If the file is missing and PROXMOX_TOKEN_SECRET is not exported, API commands fail with a clear error message.
Connection methods: SSH (default, ssh root@host qm ...) or API (curl -k https://host:8006/api2/json/... with PVEAPIToken header).
Minimum API privileges for token-based access (role: IncusOSDeployer):
```
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
Datastore.Audit SDN.Use Sys.Audit
```
Sys.Audit is needed for --resources (host RAM/CPU/uptime via /nodes/<node>/status). It requires a separate ACL on /nodes/<node> since the pool-scoped ACL doesn't cover node-level endpoints.
Required VM settings (getting any wrong causes IncusOS install failure):
- bios=ovmf, machine=q35 -- UEFI boot required
- efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
- tpmstate0: version=v2.0 -- required for disk encryption
- cpu=host -- needed for x86_64_v3 instruction set requirement
- scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
- balloon=0 -- IncusOS manages memory internally
- ide3 -- SEED_DATA ISO 9660 image attached as second CD-ROM
- Minimum 50 GiB disk, minimum 4096 MiB RAM
VLAN tagging: the vlan config key in proxmox.yaml adds a VLAN tag to the VM's NIC (net0: virtio,bridge=vmbr0,tag=69). This places the VM on the tagged VLAN instead of the native/untagged network. The current lab uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the Proxmox VM level only — IncusOS and Incus instances inside the VM are unaware of it. The setting is optional: omit vlan: for untagged access.
Disk target: do NOT specify disk-target in the seed for Proxmox VMs. IncusOS does literal string matching (not glob) on disk device IDs. scsi-* does NOT match scsi-0QEMU_QEMU_HARDDISK_drive-scsi0. Omit disk-target entirely and let IncusOS auto-detect (works for single-disk VMs).
Install flow (automated by incusos-proxmox):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), no force_reboot in seed
2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), then sits at "please remove installation media" prompt
3. Detect install completion by polling blockstat.scsi0.wr_bytes via API -- when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. Delete ide2 and ide3 -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order
6. Set boot order to order=scsi0 and start from disk
IP detection: IncusOS is immutable and has no QEMU guest agent. Two strategies: (1) Static IP (preferred): set ip: ADDR/PREFIX per VM in the lab config + gateway: and dns: in proxmox.yaml. The seed's network.yaml configures the VM's interface at boot. No ARP scan or SSH needed — the IP is known at deploy time. Works across VLANs. (2) ARP-based lookup (fallback for DHCP): get MAC from Proxmox VM config → flush stale ARP → ping broadcast → look up MAC in ARP table. Only works on the same L2 domain (native LAN, not across VLANs).
force_reboot is NOT used on Proxmox (changed 2026-02-22): the seed omits force_reboot. Without it, IncusOS sits at "please remove installation media" after install — which is exactly what we want. We detect completion via blockstat (876 MiB written, then idle), stop the VM externally, remove media, and start from disk. This eliminates the SysRq-B intermediate boot that triggers the crontab race condition (issue #843). On physical hardware or unmanaged VMs, force_reboot: true is still needed — this optimization is specific to Proxmox automated deploys. force_reboot triggers SysRq-B (raw kernel reboot via /proc/sysrq-trigger). This does NOT reset QEMU VM uptime — only a Proxmox stop/start does.
Resource pool isolation: the optional proxmox.pool config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API token ACL can be scoped to /pool/<name> instead of /. Setup:
```
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
```
--status command: incusos-proxmox --status config.yaml shows per-VM deployment status (Proxmox state, install status, IP, port 8443, incus remote). Also runs post-deployment checks (Incus connectivity, Operations Center URL).
Reconcile on re-runs: when --phase all detects existing VMs from config, an interactive menu offers: (1) run status checks, (2) continue install for incomplete VMs, (3) destroy and redeploy, (4) abort. With --yes, defaults to option 1 (safe -- never auto-destroys).
Install idempotency: phase_install checks each VM's state before acting -- already-running VMs are skipped, stopped-but-installed VMs are started from disk, only VMs with install media proceed through installation.

Multi-lab coexistence

Lab lifecycle: --lab-down config.yaml stops all VMs (Proxmox stop, VMs stay on disk). --lab-up config.yaml starts stopped VMs from disk (refuses to start VMs with install media still attached). These are distinct from --cleanup (which destroys VMs permanently).
Resource awareness: --resources shows Proxmox host RAM, CPU, storage usage, and per-pool allocation. Requires API method.
Lab inventory: --labs scans the pool for managed VMs (by [incusos-lab:managed] marker), groups by config file, and shows per-lab status, VM count, RAM, and disk.
VMID range convention (to avoid collisions between coexisting labs):

Range Lab

400-499 OC-managed nodes

800-809 Single-node labs

900-909 Basic cluster

910-919 OC combined (server + nodes)

920-929 OC server standalone

930-939 Advanced / heterogeneous

Range	Lab
400-499	OC-managed nodes
800-809	Single-node labs
900-909	Basic cluster
910-919	OC combined (server + nodes)
920-929	OC server standalone
930-939	Advanced / heterogeneous

Resource constraints for multi-lab

RAM is the bottleneck: each IncusOS VM needs 4 GiB minimum. A 3-node cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource where you can actually run out.
Storage is not a concern: Proxmox ZFS uses thin provisioning by default (sparse in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. LZ4 compression provides ~1.5-1.8x ratio on OS data.
CPU is plentiful: 4 cores per VM, 20 cores on host. Multiple labs can share CPUs without contention.
Pre-deploy checks: incusos-proxmox warns during preflight if requested RAM exceeds available host RAM (API method only).
--lab-up auto-deploy: if no VMs exist yet, --lab-up offers to run the full deploy pipeline (auto-accepts with --yes).
--resources actual disk: shows actual vs allocated disk for pool VMs (via storage content API) and notes thin provisioning for ZFS/LVM-thin.

Incus clustering via remotes

Cluster formation is done entirely through the incus CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
No VIP needed: each node advertises its own IP as its cluster address. Clients can connect to any cluster member; requests are forwarded internally.

Pre-clustering: fix core.https_address

IncusOS nodes default to core.https_address: :8443 (wildcard / all interfaces). Clustering requires a specific routable IP so nodes can address each other.

Set the IP on every node BEFORE enabling clustering:

incus config set <remote>: core.https_address <NODE_IP>:8443

Get each node's routable IP via the API:

incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(a) for a in d['environment']['addresses'] \
   if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"

This is safe to do while remotes are connected -- the remote already points to the specific IP; we're just narrowing the bind address. Certificate trust is fingerprint-based, not address-based.

Cluster enable (init node)

```
incus cluster enable <remote>: <member-name>
```
Note: this is TWO arguments: <remote>: (trailing colon) and <member-name>. The help text shows [<remote>:] <name> — NOT remote:name as a single arg.
TLS certificate regeneration: enabling clustering causes the server to generate a new TLS certificate (cluster cert). The new cert may only have SANs for 127.0.0.1 and ::1, breaking the existing remote.

Fix: remove and re-add the remote to pin the new certificate:

incus remote switch local         # if init remote is current default
incus remote remove <remote>
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate

The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address.

Joining nodes: apply_defaults and the storage pool conflict

Upstream recommendation: use apply_defaults: false for nodes destined to join a cluster. The official IncusOS clustering tutorial states joining servers "cannot have preexisting networks or storage pools defined." With apply_defaults: false, the node still listens on port 8443, still trusts preseed certificates, and the underlying ZFS dataset (local/incus) still exists -- but no Incus storage pool or network metadata is created, so the join process works cleanly.
apply_defaults: true on joining nodes is also functional but requires an 8-command cleanup per node before join (delete pool, network, volumes, profile devices). This is automated in lab-test but adds complexity.
Recommended seed pattern for clusters:
- Bootstrap/init node: apply_defaults: true (needs pool and network)
- Joining nodes: apply_defaults: false (join process creates member-specific entries)
- Standalone nodes: apply_defaults: true (needs pool and network to be functional)

If apply_defaults: true was used, the cleanup before join is:

# 1. Remove config references
incus config unset <remote>: storage.backups_volume
incus config unset <remote>: storage.images_volume
# 2. Delete volumes
incus storage volume delete <remote>:local backups
incus storage volume delete <remote>:local images
# 3. Clear default profile references (pool is "in use" otherwise)
incus profile device remove <remote>:default root
incus profile device remove <remote>:default eth0
# 4. Delete pool and network
incus storage delete <remote>:local
incus network delete <remote>:incusbr0

Join workflow

Generate token (on init node, single argument remote:member-name):
```
incus cluster add <init-remote>:<new-member-name>
```
Join (interactive -- prompts for 5 values):
```
incus cluster join <init-remote>: <joining-remote>:
```
Interactive prompts and correct answers:
1. IP address → accept default (node's IP, already set via core.https_address)
2. Member name → accept default (matches the token)
3. "All existing data is lost" → yes
4. source property for storage pool "local" → local/incus
5. zfs.pool_name property for storage pool "local" → local/incus

Automated (non-interactive):

printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:

After join: the joining node gets a new cluster certificate. Fix the remote (same as init node):

incus remote remove <joining-remote>
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate

Command syntax gotchas

incus cluster enable remote: member-name -- TWO arguments (remote: + name)
incus cluster add remote:member-name -- ONE argument (no space)
incus cluster remove remote:member-name --force -- ONE argument; prompts "yes/no" even with --force, pipe printf "yes\n" for automation
incus cluster evacuate remote:member-name -- ONE argument (no space)
incus cluster restore remote:member-name -- ONE argument (no space)
incus cluster join init-remote: joining-remote: -- TWO arguments (space)
incus storage show remote:pool -- ONE argument (no space)
incus storage show remote:pool --target member -- target flag for member-specific config
incus config set remote: key value -- remote with trailing colon + space
General rule: remote:resource for targeting a resource, remote: (trailing colon) for targeting the server itself

Post-join state

After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations.
lab-test automates cluster formation, workload testing, and migration.

Workload placement and migration

Targeted launch: incus launch images:debian/12 <cluster-remote>:name --target <member>
Cluster-wide visibility: incus list on any member shows all instances.

Container migration: stop/move/start only (CRIU live migration is unreliable). Data persists, processes do not.

incus stop <remote>:<instance>
incus move <remote>:<instance> --target <destination-member>
incus start <remote>:<instance>

VM live migration: requires migration.stateful=true (must be set while VM is stopped). Preserves running state with no downtime.
```
incus move <remote>:<instance> --target <destination-member>
```
VM live migration requires limits.cpu as a range (e.g., 0-1), not an integer. Without this, Incus sets QEMU's maxcpus to the host's CPU count (driver_qemu_templates.go: maxcpus = min(cpu.Total, 64)). Different maxcpus values size the ICH9 ACPI CPU hotplug state arrays differently, causing Missing section footer for ICH9LPC on restore. Using a range (pinning syntax) eliminates maxcpus entirely and uses fixed sockets/cores/threads topology — portable across all hosts.
```
# WRONG: integer → maxcpus varies by host → migration fails
incus config set <instance> limits.cpu=2
# RIGHT: range → fixed topology → migration works everywhere
incus config set <instance> limits.cpu=0-1
```
VM live migration works in nested virtualization (IncusOS inside Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
The vnmi CPUID warning (CPUID[eax=8000000Ah].EDX.vnmi) that appears during migration is cosmetic. It fires from QEMU's feature dependency checker before KVM filters out unsupported features and does not affect migration.
Stateful stop/restore (incus stop --stateful + incus start) also requires the limits.cpu range fix. Use incus start --stateless to discard a saved state file that cannot be restored.
VM size.state config: stateful operations require size.state on the root disk (incus config device add <instance> root disk path=/ pool=local size.state=2GiB). Without it, incus stop --stateful fails.
Cluster evacuation: incus cluster evacuate <remote>:<member> --force (ONE argument, like cluster enable and cluster add). Use --action stop if VMs lack the limits.cpu range fix. Restore with incus cluster restore <remote>:<member> --force.
VM agent reconnect: after live migration, the incus agent inside the VM needs ~3-4 seconds to reconnect. incus exec commands issued immediately after migration may fail with "VM agent isn't currently running". Scripts should sleep 4 after migration before running incus exec.
Multi-vCPU migration: tested with 2, 3, and 4 vCPU VMs across heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., limits.cpu=0-2) work identically to even counts. A 4-vCPU VM on a 4-core host (100% core usage) migrates without issues. size.state=4GiB recommended for 3-4 vCPU VMs.
Concurrent migrations: migrating multiple VMs simultaneously from different source nodes works without interference. ~140 MB/s per migration.
Active I/O during migration: disk writes and network activity survive live migration transparently. File integrity verified after migration.

Cluster rebalancing: Incus can auto-redistribute VMs when a new node joins. Only moves VMs with migration.stateful=true. Containers are NOT auto-rebalanced.

incus config set <remote>: cluster.rebalance.interval=1     # minutes
incus config set <remote>: cluster.rebalance.threshold=10    # imbalance %
incus config set <remote>: cluster.rebalance.batch=2         # max VMs/run
incus config set <remote>: cluster.rebalance.cooldown=5m     # wait between runs

Node replacement lifecycle: evacuate → remove → destroy → deploy fresh → join → auto-rebalance. Full procedure tested. See notes/clustering-guide.md for step-by-step instructions.
incus cluster remove requires confirmation: even with --force, it prompts "Are you really sure?". Pipe yes for automation:
```
printf "yes\n" | incus cluster remove <remote>:<member> --force
```
See notes/clustering-guide.md for full details and references.

Lab validation (lab-test)

lab-test reads the same YAML config as incusos-proxmox and operates on the VM names defined there (expects incus remotes to exist).
Phases: deploy, single (workloads), cluster, workload (on cluster), migrate (stop/move + evacuate/restore).
Test instances use names starting with test- for easy cleanup.
The script reports PASS/FAIL/SKIP for each test and prints a summary.

Operations Center

CLI binary: operations-center (installed from GitHub releases or built from source at github.com/FuturFusion/operations-center).
Config directory: ~/.config/operations-center/ (uses same cert format as Incus: copy client.crt and client.key from ~/.config/incus/).
Port: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
Browser access: requires PKCS#12 client certificate (client.pfx) imported into the browser. Web UI is a React SPA at /ui/.
OC is under active development (v0.3.0). Commands and APIs may change. Use operations-center --version (not version subcommand) to check.
OC CLI does NOT support remote: suffix syntax: unlike the Incus CLI, the OC CLI requires operations-center remote switch NAME before running commands. operations-center admin os show oc-lab: fails with "Invalid number of arguments".
v0.3.0 new commands: provisioning channel (add/list/show), cluster update/update-certificate/rename, expanded inventory (network-acl, address-sets, load-balancers, peers, zones, integrations, forwards, storage-buckets, storage-volumes), inventory query (cross-resource tree view with filters and Go templates), system certificate set.
The --doctor command on incusos-proxmox reports whether the CLI is installed.

Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)

No brownfield adoption: nodes must boot from an OC-provisioned ISO.

Token seeds: named, reusable pre-seed configs attached to tokens. YAML must use structured format with section keys (install:, not flat).

operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/preseed.yaml --description "Force reboot for Proxmox"
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

Hybrid deployment (tested, recommended): incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml combines OC auto-registration (from boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
Self-registration: nodes auto-register with OC within ~30s of first boot. Hostname from SEED_DATA is used as the server name.

Cluster formation (tested):

operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC handles: core.https_address → cluster enable → joins → seed application → Terraform config. Adds meshbr0 network.

apply_defaults: false is recommended (tested): use apply_defaults: false for OC-managed nodes. OC's Terraform handles storage pool, network, and cert creation cleanly. With apply_defaults: true, nodes already have these resources and OC's Terraform fails with "already exists" errors (cluster still forms, but Terraform artifacts are empty).

Tested limitations

Inventory is NOT real-time -- requires explicit cluster resync
OC reboot breaks OC-managed nodes on Proxmox -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). The failure is OC-specific: the OC agent pushes config via the IncusOS REST API that gets persisted to state.txt. On reboot, invalid values (e.g., cron expression where Go duration is expected) crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
No cluster member state tracking -- OC always shows ready even for EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat detects failure in ~40s; OC has no equivalent).
Stale entries from out-of-band cluster changes persist after resync
Server removal blocked if server is part of an OC cluster
Node failure recovery: Proxmox hard-stop simulates crash. After restart, node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly if crontab bug doesn't hit (auto-healed by fix_scrub_schedule()).
See notes/operations-center-guide.md for full tested OC reference.

incusos-proxmox doctor and cleanup

--doctor: standalone environment check. No config file required. Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally Proxmox connectivity (from proxmox.yaml or config file).
--cleanup: destroys VMs defined in the config file.
--cleanup --deep: also deletes the specific IncusOS ISO used by this deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete all IncusOS ISOs (unlike the old behavior).
--cleanup-all: pool-wide cleanup. Only needs proxmox.yaml (no lab config required). Destroys all VMs with [incusos-lab:managed] marker.
--cleanup-all --deep: aggressive blanket delete of ALL IncusOS_*.iso and seed-*.iso from storage + remotes + cache.
--verbose / -v: shows detailed output (tool paths, API calls). Default output is concise (step names + results). --quiet suppresses everything except warnings and errors.
--retries N: number of stop+start retries for VMs that fail to boot (port 8443 not reachable). Default: 3. --retries 0 disables retries. Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for up to 120s. With the force_reboot fix (2026-02-22), the crontab bug is eliminated and retries are rarely needed. They remain as a safety net for other transient boot failures. After retries, fix_scrub_schedule() proactively heals any remaining scrub_schedule issues via the IncusOS REST API.
Boot timeout is 180s (60s initial sleep + 120s polling): first boot downloads SecureBoot update + application sysext (Incus: ~30-120s depending on CDN speed and concurrent VMs). The previous 90s timeout was too short under load — premature retries (hard VM stop during first boot) can corrupt the TPM encryption key, causing permanent "zfs load-key: Raw key too short (expected 32)" errors that no amount of retries can fix.

IncusOS first-boot sequence (observed via console screenshots)

Observed using observe-deploy with 2-second screenshot intervals. The complete lifecycle from Proxmox VM start to port 8443 ready:

Phase 1: ISO boot and installation (~60-85s)

UEFI firmware → "Guest has not initialized the display (yet)" (~2s)
IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s)
IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s)
- "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda"
- "Cloning GPT partitions" → progress bar → complete
force_reboot: true triggers SysRq-B (raw kernel reboot via /proc/sysrq-trigger, preceded by unix.Sync() + 5s sleep). Not a graceful systemd reboot. This is intentional — the install environment is minimal and may not have systemd running.
VM reboots but ISO still attached → boots from ISO again
"System check error: install media detected, but the system is already installed; please remove USB/CDROM and reboot the system" (~40s). IncusOS daemon sleeps 1 hour at this error, then exits. The state.txt on the installed disk (scsi0 data partition) is NOT touched — this boot runs from the ISO rootfs (tmpfs).
Stays at this error until blockstat detects stable writes (3×5s = 15s idle)

Phase 2: Transition (~15s)

Proxmox stops VM
Remove ide2 (ISO) and ide3 (seed), set boot: order=scsi0
Start VM from disk

Phase 3: First boot from disk (~50s to port 8443)

UEFI → "IncusOS is starting..." (~8s)
state.LoadOrCreate() → state.txt doesn't exist → initialize() sets defaults: ScrubSchedule="0 4 * * 0", CheckFrequency="6h", Channel="stable" → Save() writes state.txt to data partition
"Auto-generating encryption recovery key, this may take a few seconds" (~10s)
"System is starting up machine-id=... mode=production" (~10s after key)
"Bringing up the network" (immediate)
"Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s)
"Downloading application update application=incus" + progress bar (~33-38s)
"Bringing up the local storage" (after download)
"Starting application" → "Initializing application" → TLS cert fingerprint
registerJobs() validates ScrubSchedule with gocron
"System is ready" — port 8443 now reachable

Total: ~130-215s from VM create to port 8443 ready. The biggest variable is the application download (33-38s on first boot; skipped if already cached).

Important: DoInstall() does NOT create state.txt. It writes the IncusOS image to disk, processes seed files (including deleting install.yaml from the target's seed partition via CleanupPostInstall()), and triggers the SysRq-B reboot. The state.txt is created exclusively on first boot from the installed disk by state.LoadOrCreate() → initialize().

IncusOS boot failure (crontab / update frequency error)

RESOLVED (2026-02-22): root cause identified and fixed in our pipeline. The force_reboot seed option triggers SysRq-B after install, causing an "install media detected" intermediate boot. This intermediate boot generates ~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs). Our blockstat detection catches these late writes, but by then the intermediate boot has run and somehow corrupted the state for the subsequent first-from-disk boot. Fix: omit force_reboot from the seed. Without it, the installer sits at "please remove media" after the 876 MiB image clone, blockstat detects idle cleanly, and the first disk boot is pristine. Results: 15/15 PASS (100%) without force_reboot on 4 cores vs 50% with it.
Symptom: "ERROR invalid crontab expression" appears on the console during first boot. Intermittent — confirmed ~50% failure rate with force_reboot: true on 4-core VMs. Error does NOT correlate with update.yaml presence/absence, ISO version, or version mismatch.
Two distinct error paths in IncusOS:
- registerJobs() validates ScrubSchedule (5-field crontab expression like "0 4 * * 0") using gocron.IsValid(). If invalid, the entire daemon exits — this is the fatal one. Port 8443 never opens (or opens briefly during the 15-second error sleep).
- updateChecker() validates CheckFrequency (Go time.ParseDuration() format like "6h"). If invalid, it logs an error but does not crash.
Not always fatal (confirmed by parallel investigation):
- The REST API starts BEFORE startup() completes. Applications (Incus) start at step ~16 of startup(), while registerJobs() is at step ~19. When Incus starts before the scheduler crashes, port 8443 is available during the 15-second error sleep (before os.Exit(1)).
- Port 8443 is reachable during the error window, and API calls work (including /os/1.0/system/storage). The scrub_schedule field is empty in the API response — this is the definitive indicator.
- "System is ready" is NOT shown on console (only appears after registerJobs() succeeds). Console shows "ERROR invalid crontab expression" as the last line when the bug hits.
- Detection heuristic: port 8443 reachable BUT scrub_schedule is empty → crontab bug hit. Port 8443 reachable AND scrub_schedule is "0 4 * * 0" → genuine success.
Source code analysis (root cause investigation):
- state.txt is created FRESH on first boot from disk by LoadOrCreate() → initialize(). The initialize() function correctly sets ScrubSchedule = "0 4 * * 0".
- The "install media detected" boot (#2, ISO still attached) runs from the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed disk. The installed disk's state.txt is untouched.
- DoInstall() does not create state.txt. It only writes the OS image and processes seed files.
- The encoder skips zero values: encodeHelper() returns early on v.IsZero(). An empty ScrubSchedule string is NOT written to file. If state.txt is read with a missing ScrubSchedule, the field stays at Go's zero value (empty string) → gocron.IsValid("") fails.
- Non-atomic state writes: Save() uses os.WriteFile() which truncates then writes. Crash during write → partial file.
- Race window: The REST API server starts BEFORE startup() completes. Applications start before registerJobs(). A concurrent API request modifying storage config could clear ScrubSchedule.
- Root cause confirmed: the bug is triggered by the SysRq-B intermediate boot (from force_reboot: true). Controlled experiments on VLAN 69 with static IP (2026-02-22, ISO 202602210344):
  - Baseline (4 cores, force_reboot): 5/10 PASS (50%)
  - cache=writethrough: 3/5 PASS (60%) — falsified
  - agent=0: 3/5 PASS (60%) — falsified
  - cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race
  - No force_reboot (4 cores): 15/15 PASS (100%) — confirmed fix Correlation: runs with 876 MiB blockstat detection (before SysRq-B) had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%.
- The try-it service is NOT IncusOS: stgraber's comparison ("thousands of times a day") is invalid. The try-it service runs Ubuntu VMs with Incus installed from the Zabbly daily repo — a completely different product. Issue #843 was independently filed by another Proxmox user (fperreau on Proxmox 9.1.4), confirming it's not unique to our setup.
- The underlying IncusOS bug is still a race condition between the REST API/application startup and registerJobs() that clears ScrubSchedule. Investigation ruled out version mismatch, update downloads, and gocron library issues. The field is genuinely empty in the state struct.
- Investigation data (2026-02-21, ISO 202602210344, 4 VMs parallel): Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%); Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG (67% failure rate). Console screenshots confirm "ERROR invalid crontab expression" on failed VMs and "System is ready" on successful VMs.
- Version mismatch NOT the cause: stgraber's hypothesis that the bug relates to ISO version != CDN latest was tested and ruled out. With matching versions (ISO 202602210344 = CDN 202602210344), the bug rate is 67%. Sysext downloads still occur with matching versions (SecureBoot
  - application updates) but no OS update download. The bug occurs regardless of whether sysext downloads happen.
- Sysext download on matching versions: even when ISO version matches CDN latest, the first boot downloads SecureBoot update (~1s) and application sysext (~2 min for Incus). The sysext is NOT in the ISO — it's always downloaded from the update provider. An OS update download (and reboot) only occurs when versions differ.

Proposed upstream fix (for incus-os project):

// In registerJobs(): validate and fall back to default
func registerJobs(s *state.State) error {
    schedule := s.System.Storage.Config.ScrubSchedule
    if schedule == "" {
        schedule = "0 4 * * 0"
        s.System.Storage.Config.ScrubSchedule = schedule
    }
    err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools)
    if err != nil {
        // Fall back to default instead of crashing the daemon
        slog.Warn("Invalid scrub schedule, using default", "schedule", schedule)
        s.System.Storage.Config.ScrubSchedule = "0 4 * * 0"
        return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools)
    }
    return nil
}

Additional recommendations: (1) use atomic writes in Save() (write to temp file + os.Rename()), (2) always encode ScrubSchedule even when empty (to preserve the default through encode/decode cycles), (3) add mutex protection on State fields shared between API handlers and startup().

Error is persistent within a VM session: verified by screenshots — when the crontab error hits, the deferred s.Save() writes corrupt state (version 7, ScrubSchedule missing because encoder skips zero values). Subsequent systemd daemon restarts read the corrupt state and crash again. The error does NOT self-heal via daemon restart alone.
Recovery via Proxmox stop+start: phase_install in incusos-proxmox retries once with a full Proxmox stop (hard VM power-off) → 10s wait → start. The hard power-off may cause uncommitted filesystem writes (including the corrupt state.txt) to be lost, restoring the correct state from initialize(). This works because IncusOS's data partition has a journal commit interval (~5s) and the corrupt Save() may not have been committed before the VM was killed.
Detection: phase_install checks port 8443 after starting from disk (60s wait + up to 120s polling = 180s total). If first check fails, automatic retry adds ~180s. Failed VMs after retry are reported with remediation.
TPM corruption from premature retry: hard-stopping a VM during first boot (while the encryption key is being written to the TPM zvol) can leave the TPM with truncated key data. All subsequent boots fail with "zfs load-key: Raw key too short (expected 32)" — a permanent error that requires VM destruction and redeployment to fix. The 180s boot timeout prevents this by allowing the full first-boot sequence (including sysext download) to complete before any retry.
Auto-heal via IncusOS REST API: incusos-proxmox includes fix_scrub_schedule() which proactively fixes empty scrub_schedule on every deployed node via PUT /os/1.0/system/storage. The /os/ prefix proxies to the IncusOS daemon API through Incus. Safe to call on every node — returns early if schedule is already set. Called on both initial success and retry success paths.
Filed upstream: IncusOS issue #843.
Source files: incus-osd/cmd/incus-osd/main.go (registerJobs(), updateChecker(), startup(), firstBootActions()), incus-osd/internal/scheduling/scheduling.go (RegisterJob(), ErrInvalidCronTab), incus-osd/internal/state/file.go (LoadOrCreate(), Save(), initialize()), incus-osd/internal/state/encode.go (encodeHelper() zero-value skip), incus-osd/internal/install/install.go (DoInstall(), rebootUponDeviceRemoval() SysRq-B).

Proxmox SSH root access — strict rules

Root SSH access to the Proxmox host is available for diagnostics only. The password is stored in the env file as PROXMOX_ROOT_PASSWORD.

MANDATORY RULES — violation of any rule is unacceptable:

Screenshots only: the ONLY permitted use of root SSH is taking VM console screenshots via qm monitor <vmid> screendump. No other use without explicit user instruction.
Test VMs only: only screenshot VMs in the test VMID range (850-869) or VMs in the IncusLab pool that were created by our scripts.
No modifications: NEVER run any command that modifies, stops, starts, or deletes any VM. No qm set, qm stop, qm start, qm destroy, pct commands, zfs commands, systemctl commands, or ANY write operation on the host.
Do not touch the dev VM: the user's dev VM runs on the same host. Do not interact with it in any way — do not even qm status it.
No config access: do not read or modify /etc/pve/, storage configs, network configs, user/ACL settings, or any host-level configuration.
Transparency: every SSH command is visible in tool output. If the user rejects a command, do not retry it.
Only during active tests: only SSH during test runs where test VMs have been deployed, and only to screenshot those test VMs.

Permitted commands (exhaustive list):

# Take screenshot of a test VM (VMID in 850-869 range)
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"

# Retrieve screenshot
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
    root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/

# Cleanup screenshot on remote
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"

Technical notes:

QEMU's screendump is root-only in PVE 9's HMP permission model. API tokens cannot execute it regardless of privileges.
Use .ppm format — PNG (-f png) requires QEMU compiled with libpng, which PVE 9.1 does not have (Error: Enable PNG support with libpng).
PPM files are ~3 MB (1280x800). Convert to PNG with python3-pil: python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')" PNG output is ~20 KB and can be read directly by Claude Code (multimodal).

Coding conventions for scripts

Shell: bash with set -euo pipefail
Arithmetic: use var=$((var + 1)) instead of ((var++)) to avoid false exits under set -e when the value is 0.
Colors: support NO_COLOR=1 and TERM=dumb; use setup_colors() pattern.
Flags: support both short (-d) and long (--defaults) options.
Defaults: sane defaults so the script does something useful with zero flags.
Dry run: all scripts should support --dry-run to preview actions.
Cert detection order: files on disk first, CLI command second.
Error messages: include actionable remediation steps, not just "failed".
No hardcoded package managers: say "install the Incus client" with a link, not "sudo apt install incus".

Incus networking (OVN)

Bridge networks are node-local: each cluster member has its own independent bridge. Instances on the same bridge (same node) can communicate; cross-node instances CANNOT. Each bridge has the same subnet (e.g., 10.0.0.1/24) but they are separate L2 domains.
OVN provides cross-node L2 overlay: uses Geneve tunnels between nodes. Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
- client services + physical uplink network.
IncusOS OVN services are disabled by default: must be enabled via the IncusOS REST API (/os/1.0/services/ovn) on EVERY node before configuring Incus OVN settings. Without this, incus config set network.ovn.northbound_connection fails with db.sock not found.

OVN service enable API call:

incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:<SB_HOST>:6642",
    "enabled": true,
    "tunnel_address": "<THIS_NODE_LAN_IP>",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

database is the southbound DB (port 6642), NOT northbound (6641).

OVN control plane as container: deploy ovn-central package in a Debian container on the cluster. Use proxy devices to expose NB (6641) and SB (6642) ports on the host's LAN IP so all nodes can reach it.
Setup sequence (order matters):
1. Deploy OVN control plane container
2. Enable OVN services on ALL IncusOS nodes
3. incus config set network.ovn.northbound_connection tcp:<host>:6641
4. incus cluster role add <remote>:<member> ovn-chassis (all nodes)
5. Create physical uplink network (two-step cluster pattern)
6. Create OVN network with --type=ovn network=UPLINK
Physical uplink network: uses parent=mgmt (IncusOS management NIC — NOT ens18, which is the underlying device name but not exposed to Incus). ipv4.ovn.ranges reserves LAN IPs for OVN router external addresses, ipv4.gateway is the LAN gateway in CIDR format.
OVN network isolation: multiple OVN networks are fully isolated. Instances on different networks cannot communicate, even on the same node. Network peering (incus network peer create) enables cross-network routing.
OVN features tested: cross-node connectivity, network isolation, ACLs (per-source blocking), network peering, L4 load balancers (connection- based hashing, not round-robin), network forwards (port forwarding to LAN IPs), DNS resolution (per-network, hostname.incus domain).
LB/forward backends require IP addresses: incus network load-balancer backend add and incus network forward port add require the target instance's IP address, NOT its name. Using instance names fails with "Invalid target address".
See notes/networking-guide.md for full tutorial with test results.

Migration into Incus

incus-migrate: official tool for importing disk images, running instances, or physical machines into Incus.
Disk format conversion: use qemu-img convert between vmdk, qcow2, raw, vdi, vhd formats. Incus accepts raw and qcow2.
Import workflow: convert disk → incus storage volume import → incus init --empty --vm → attach disk → start.
Container migration: docker export → incus import for filesystem- level container migration. Docker volumes must be copied separately.
See notes/migration-guide.md for full procedures per source hypervisor.

UTM support (future)

Design document at notes/utm-support.md.
UTM provides utmctl CLI for start/stop/status but not for VM creation (requires AppleScript or .utm bundle generation).
No blockstat equivalent -- install detection must use timeout + port polling.
Seed generation already works cross-platform (Phase 3 macOS compatibility).

Git workflow

Main branch: main
Development happens on feature branches
Remote: private Gitea at ssh://git@192.168.1.200:2222/maarten/incu-contrib.git

49 KiB Raw Blame History Unescape Escape