960 lines
54 KiB
Markdown
960 lines
54 KiB
Markdown
# CLAUDE.md - Project context for AI assistants
|
||
|
||
## What this repository is
|
||
|
||
A collection of peripheral tools, scripts, and snippets for working with
|
||
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
|
||
Primarily targeting home lab environments but aiming for production-quality scripts.
|
||
|
||
## Repository structure
|
||
|
||
```
|
||
incu-contrib/
|
||
├── CLAUDE.md # This file -- project context
|
||
├── README.md # Main overview
|
||
├── .gitignore
|
||
├── incusos/ # IncusOS installation tooling
|
||
│ ├── README.md # Detailed usage docs
|
||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
|
||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||
│ ├── lab-test # Guided lab validation (12 test phases)
|
||
│ ├── observe-deploy # Single-VM deploy with rapid console screenshots
|
||
│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored)
|
||
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
||
│ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
|
||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||
│ └── examples/ # Example seed + Proxmox YAML files
|
||
└── notes/ # Research notes and reference material
|
||
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
||
├── operations-center-guide.md # Operations Center provisioning & management
|
||
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
||
├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested)
|
||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
||
└── utm-support.md # UTM support design document (future)
|
||
```
|
||
|
||
## Key technical context
|
||
|
||
### Incus version differences
|
||
|
||
- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
|
||
The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
|
||
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
|
||
macOS is client-only by design; Arch has no client-only split package.
|
||
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
|
||
not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
|
||
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
|
||
Fall back to the CLI command only as a secondary option.
|
||
- See `notes/incus-version-compatibility.md` for full platform matrix and
|
||
install instructions.
|
||
|
||
### IncusOS flasher-tool
|
||
|
||
- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
|
||
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
|
||
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
|
||
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
|
||
For cross-arch builds, download the image manually and pass via `--image`.
|
||
- CDN index: `https://images.linuxcontainers.org/os/index.json`
|
||
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`
|
||
|
||
### Seed archives
|
||
|
||
- Tar archives containing YAML files at the root level.
|
||
- Written to byte offset 2148532224 (the seed partition) in the image.
|
||
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
|
||
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
|
||
devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
|
||
- **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
|
||
**NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
|
||
FAT filesystem labels, so IncusOS cannot find the seed.
|
||
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
|
||
`operations-center.yaml`, `network.yaml`, `update.yaml`.
|
||
|
||
### Client certificates
|
||
|
||
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
|
||
- Running `incus remote list` triggers auto-generation if no keypair exists.
|
||
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
|
||
The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
|
||
`certificates`) are promoted to the top level of the `preseed` object.
|
||
- For Operations Center seed: injected under `trusted_client_certificates[]`.
|
||
- Operations Center **requires** at least one trusted certificate -- without it,
|
||
you are locked out after installation.
|
||
|
||
### Proxmox VE deployment
|
||
|
||
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
||
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
||
VMs with IncusOS-correct settings, and boots them through installation.
|
||
- **Config separation**: Proxmox connection settings (host, credentials, pool)
|
||
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
|
||
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
|
||
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
|
||
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
|
||
- **API token secret**: stored in `env` file at the repo root (gitignored).
|
||
Scripts auto-load it on startup by searching for `env` up the directory
|
||
tree from the script location. No manual `source env` needed. If the file
|
||
is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail
|
||
with a clear error message.
|
||
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
||
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
||
- **Minimum API privileges** for token-based access (role: `IncusOSDeployer`):
|
||
```
|
||
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
|
||
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
|
||
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
|
||
Datastore.Audit SDN.Use Sys.Audit
|
||
```
|
||
`Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via
|
||
`/nodes/<node>/status`). It requires a separate ACL on `/nodes/<node>`
|
||
since the pool-scoped ACL doesn't cover node-level endpoints.
|
||
- **Required VM settings** (getting any wrong causes IncusOS install failure):
|
||
- `bios=ovmf`, `machine=q35` -- UEFI boot required
|
||
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
|
||
- `tpmstate0`: `version=v2.0` -- required for disk encryption
|
||
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
|
||
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
|
||
- `balloon=0` -- IncusOS manages memory internally
|
||
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
||
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
||
- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag
|
||
to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM
|
||
on the tagged VLAN instead of the native/untagged network. The current lab
|
||
uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag,
|
||
VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the
|
||
Proxmox VM level only — IncusOS and Incus instances inside the VM are
|
||
unaware of it. The setting is optional: omit `vlan:` for untagged access.
|
||
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
||
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
||
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
||
`disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
|
||
- **Install flow** (automated by `incusos-proxmox`):
|
||
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
|
||
2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone),
|
||
then sits at "please remove installation media" prompt
|
||
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
|
||
when disk writes start then stop for 15s (3 stable polls), install is done
|
||
4. Stop the VM (Proxmox stop, not guest shutdown)
|
||
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
|
||
and refuses to start if found, regardless of boot order
|
||
6. Set boot order to `order=scsi0` and start from disk
|
||
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two
|
||
strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM
|
||
in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's
|
||
`network.yaml` configures the VM's interface at boot. No ARP scan or
|
||
SSH needed — the IP is known at deploy time. Works across VLANs.
|
||
(2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM
|
||
config → flush stale ARP → ping broadcast → look up MAC in ARP table.
|
||
Only works on the same L2 domain (native LAN, not across VLANs).
|
||
- **force_reboot is NOT used on Proxmox** (changed 2026-02-22): the seed
|
||
omits `force_reboot`. Without it, IncusOS sits at "please remove
|
||
installation media" after install — which is exactly what we want. We
|
||
detect completion via blockstat (876 MiB written, then idle), stop the
|
||
VM externally, remove media, and start from disk. This eliminates the
|
||
SysRq-B intermediate boot that triggers the crontab race condition
|
||
(issue #843). On physical hardware or unmanaged VMs, `force_reboot: true`
|
||
is still needed — this optimization is specific to Proxmox automated deploys.
|
||
`force_reboot` triggers SysRq-B (raw kernel reboot via `/proc/sysrq-trigger`).
|
||
This does NOT reset QEMU VM uptime — only a Proxmox stop/start does.
|
||
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
|
||
all VM operations to a Proxmox resource pool. When set, the script only
|
||
"sees" VMs in that pool (for collision detection and cleanup), and the API
|
||
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
|
||
```
|
||
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||
pveum pool modify IncusLab --storage local-lvm,local
|
||
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
|
||
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
|
||
```
|
||
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
|
||
deployment status (Proxmox state, install status, IP, port 8443, incus
|
||
remote). Also runs post-deployment checks (Incus connectivity, Operations
|
||
Center URL).
|
||
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
|
||
config, an interactive menu offers: (1) run status checks, (2) continue
|
||
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
|
||
`--yes`, defaults to option 1 (safe -- never auto-destroys).
|
||
- **Install idempotency**: `phase_install` checks each VM's state before
|
||
acting -- already-running VMs are skipped, stopped-but-installed VMs are
|
||
started from disk, only VMs with install media proceed through installation.
|
||
|
||
### Multi-lab coexistence
|
||
|
||
- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop,
|
||
VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk
|
||
(refuses to start VMs with install media still attached). These are
|
||
distinct from `--cleanup` (which destroys VMs permanently).
|
||
- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage
|
||
usage, and per-pool allocation. Requires API method.
|
||
- **Lab inventory**: `--labs` scans the pool for managed VMs (by
|
||
`[incusos-lab:managed]` marker), groups by config file, and shows
|
||
per-lab status, VM count, RAM, and disk.
|
||
- **VMID range convention** (to avoid collisions between coexisting labs):
|
||
|
||
| Range | Lab |
|
||
|-------|-----|
|
||
| 400-499 | OC-managed nodes |
|
||
| 800-809 | Single-node labs |
|
||
| 900-909 | Basic cluster |
|
||
| 910-919 | OC combined (server + nodes) |
|
||
| 920-929 | OC server standalone |
|
||
| 930-939 | Advanced / heterogeneous |
|
||
|
||
### Resource constraints for multi-lab
|
||
|
||
- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node
|
||
cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource
|
||
where you can actually run out.
|
||
- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default
|
||
(`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk.
|
||
LZ4 compression provides ~1.5-1.8x ratio on OS data.
|
||
- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs
|
||
can share CPUs without contention.
|
||
- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if
|
||
requested RAM exceeds available host RAM (API method only).
|
||
- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to
|
||
run the full deploy pipeline (auto-accepts with `--yes`).
|
||
- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs
|
||
(via storage content API) and notes thin provisioning for ZFS/LVM-thin.
|
||
|
||
### Incus clustering via remotes
|
||
|
||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
||
- **No VIP needed**: each node advertises its own IP as its cluster address.
|
||
Clients can connect to any cluster member; requests are forwarded internally.
|
||
|
||
#### Pre-clustering: fix core.https_address
|
||
|
||
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
|
||
interfaces). Clustering requires a **specific routable IP** so nodes can
|
||
address each other.
|
||
- **Set the IP on every node BEFORE enabling clustering:**
|
||
```bash
|
||
incus config set <remote>: core.https_address <NODE_IP>:8443
|
||
```
|
||
- Get each node's routable IP via the API:
|
||
```bash
|
||
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||
[print(a) for a in d['environment']['addresses'] \
|
||
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||
```
|
||
- This is safe to do while remotes are connected -- the remote already points
|
||
to the specific IP; we're just narrowing the bind address. Certificate trust
|
||
is fingerprint-based, not address-based.
|
||
|
||
#### Cluster enable (init node)
|
||
|
||
- ```bash
|
||
incus cluster enable <remote>: <member-name>
|
||
```
|
||
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
||
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
|
||
- **TLS certificate regeneration**: enabling clustering causes the server to
|
||
generate a new TLS certificate (cluster cert). The new cert may only have
|
||
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
|
||
- **Fix**: remove and re-add the remote to pin the new certificate:
|
||
```bash
|
||
incus remote switch local # if init remote is current default
|
||
incus remote remove <remote>
|
||
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
||
```
|
||
- The cert trust on the server side (client → server) is unaffected -- it's
|
||
stored by fingerprint in the Incus database, independent of listen address.
|
||
|
||
#### Joining nodes: apply_defaults and the storage pool conflict
|
||
|
||
- **Upstream recommendation**: use `apply_defaults: false` for nodes destined
|
||
to join a cluster. The official IncusOS clustering tutorial states joining
|
||
servers "cannot have preexisting networks or storage pools defined." With
|
||
`apply_defaults: false`, the node still listens on port 8443, still trusts
|
||
preseed certificates, and the underlying ZFS dataset (`local/incus`) still
|
||
exists -- but no Incus storage pool or network metadata is created, so
|
||
the join process works cleanly.
|
||
- **`apply_defaults: true` on joining nodes** is also functional but requires
|
||
an 8-command cleanup per node before join (delete pool, network, volumes,
|
||
profile devices). This is automated in `lab-test` but adds complexity.
|
||
- **Recommended seed pattern for clusters**:
|
||
- Bootstrap/init node: `apply_defaults: true` (needs pool and network)
|
||
- Joining nodes: `apply_defaults: false` (join process creates member-specific entries)
|
||
- Standalone nodes: `apply_defaults: true` (needs pool and network to be functional)
|
||
- **If `apply_defaults: true` was used**, the cleanup before join is:
|
||
```bash
|
||
# 1. Remove config references
|
||
incus config unset <remote>: storage.backups_volume
|
||
incus config unset <remote>: storage.images_volume
|
||
# 2. Delete volumes
|
||
incus storage volume delete <remote>:local backups
|
||
incus storage volume delete <remote>:local images
|
||
# 3. Clear default profile references (pool is "in use" otherwise)
|
||
incus profile device remove <remote>:default root
|
||
incus profile device remove <remote>:default eth0
|
||
# 4. Delete pool and network
|
||
incus storage delete <remote>:local
|
||
incus network delete <remote>:incusbr0
|
||
```
|
||
|
||
#### Join workflow
|
||
|
||
- **Generate token** (on init node, single argument `remote:member-name`):
|
||
```bash
|
||
incus cluster add <init-remote>:<new-member-name>
|
||
```
|
||
- **Join** (interactive -- prompts for 5 values):
|
||
```bash
|
||
incus cluster join <init-remote>: <joining-remote>:
|
||
```
|
||
Interactive prompts and correct answers:
|
||
1. IP address → accept default (node's IP, already set via core.https_address)
|
||
2. Member name → accept default (matches the token)
|
||
3. "All existing data is lost" → `yes`
|
||
4. `source` property for storage pool "local" → `local/incus`
|
||
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
|
||
- **Automated (non-interactive):**
|
||
```bash
|
||
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
||
```
|
||
- **After join**: the joining node gets a new cluster certificate. Fix the
|
||
remote (same as init node):
|
||
```bash
|
||
incus remote remove <joining-remote>
|
||
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
|
||
```
|
||
|
||
#### Command syntax gotchas
|
||
|
||
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
|
||
- `incus cluster add remote:member-name` -- ONE argument (no space)
|
||
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
|
||
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
|
||
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
|
||
- `incus cluster restore remote:member-name` -- ONE argument (no space)
|
||
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
|
||
- `incus storage show remote:pool` -- ONE argument (no space)
|
||
- `incus storage show remote:pool --target member` -- target flag for
|
||
member-specific config
|
||
- `incus config set remote: key value` -- remote with trailing colon + space
|
||
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
|
||
colon) for targeting the server itself
|
||
|
||
#### Post-join state
|
||
|
||
- After joining, the cluster is managed through the init node's remote. The
|
||
individual node remotes still work for node-specific operations.
|
||
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
||
|
||
#### Workload placement and migration
|
||
|
||
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
|
||
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
|
||
- **Container migration**: stop/move/start only (CRIU live migration is
|
||
unreliable). Data persists, processes do not.
|
||
```bash
|
||
incus stop <remote>:<instance>
|
||
incus move <remote>:<instance> --target <destination-member>
|
||
incus start <remote>:<instance>
|
||
```
|
||
- **VM live migration**: requires `migration.stateful=true` (must be set
|
||
while VM is stopped). Preserves running state with no downtime.
|
||
```bash
|
||
incus move <remote>:<instance> --target <destination-member>
|
||
```
|
||
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
|
||
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
|
||
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
|
||
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
|
||
differently, causing `Missing section footer for ICH9LPC` on restore.
|
||
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
|
||
fixed `sockets/cores/threads` topology — portable across all hosts.
|
||
```bash
|
||
# WRONG: integer → maxcpus varies by host → migration fails
|
||
incus config set <instance> limits.cpu=2
|
||
# RIGHT: range → fixed topology → migration works everywhere
|
||
incus config set <instance> limits.cpu=0-1
|
||
```
|
||
- **VM live migration works in nested virtualization** (IncusOS inside
|
||
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
|
||
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
|
||
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
|
||
appears during migration is cosmetic. It fires from QEMU's feature
|
||
dependency checker before KVM filters out unsupported features and does
|
||
not affect migration.
|
||
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
|
||
requires the `limits.cpu` range fix. Use `incus start --stateless` to
|
||
discard a saved state file that cannot be restored.
|
||
- **VM `size.state` config**: stateful operations require `size.state` on
|
||
the root disk (`incus config device add <instance> root disk path=/
|
||
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
|
||
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
||
(ONE argument, like `cluster enable` and `cluster add`).
|
||
Use `--action stop` if VMs lack the `limits.cpu` range fix.
|
||
Restore with `incus cluster restore <remote>:<member> --force`.
|
||
- **VM agent reconnect**: after live migration, the incus agent inside the VM
|
||
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
|
||
after migration may fail with "VM agent isn't currently running". Scripts
|
||
should `sleep 4` after migration before running `incus exec`.
|
||
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
|
||
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
|
||
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
|
||
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
|
||
- **Concurrent migrations**: migrating multiple VMs simultaneously from
|
||
different source nodes works without interference. ~140 MB/s per migration.
|
||
- **Active I/O during migration**: disk writes and network activity survive
|
||
live migration transparently. File integrity verified after migration.
|
||
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
|
||
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
|
||
auto-rebalanced.
|
||
```bash
|
||
incus config set <remote>: cluster.rebalance.interval=1 # minutes
|
||
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
|
||
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
|
||
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
|
||
```
|
||
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
|
||
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
|
||
for step-by-step instructions.
|
||
- **`incus cluster remove` requires confirmation**: even with `--force`, it
|
||
prompts "Are you really sure?". Pipe `yes` for automation:
|
||
```bash
|
||
printf "yes\n" | incus cluster remove <remote>:<member> --force
|
||
```
|
||
- See `notes/clustering-guide.md` for full details and references.
|
||
|
||
### Lab validation (lab-test)
|
||
|
||
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
||
on the VM names defined there (expects `incus` remotes to exist).
|
||
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
|
||
migrate (stop/move + evacuate/restore).
|
||
- **Test instances** use names starting with `test-` for easy cleanup.
|
||
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
|
||
|
||
### Operations Center
|
||
|
||
- **CLI binary:** `operations-center` (installed from GitHub releases or
|
||
built from source at `github.com/FuturFusion/operations-center`).
|
||
- **Config directory:** `~/.config/operations-center/` (uses same cert format
|
||
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
||
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
|
||
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
|
||
imported into the browser. Web UI is a React SPA at `/ui/`.
|
||
- **OC is under active development** (v0.3.0). Commands and APIs may change.
|
||
Use `operations-center --version` (not `version` subcommand) to check.
|
||
- **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI,
|
||
the OC CLI requires `operations-center remote switch NAME` before running
|
||
commands. `operations-center admin os show oc-lab:` fails with "Invalid
|
||
number of arguments".
|
||
- **v0.3.0 new commands**: `provisioning channel` (add/list/show),
|
||
`cluster update/update-certificate/rename`, expanded `inventory`
|
||
(network-acl, address-sets, load-balancers, peers, zones, integrations,
|
||
forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource
|
||
tree view with filters and Go templates), `system certificate set`.
|
||
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
||
installed.
|
||
|
||
#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)
|
||
|
||
- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
|
||
- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
|
||
must use structured format with section keys (`install:`, not flat).
|
||
**No force_reboot** in token seeds for Proxmox (same fix as standard deploys).
|
||
```bash
|
||
operations-center provisioning token seed add <UUID> proxmox-preseed \
|
||
/tmp/preseed.yaml --description "No force_reboot for Proxmox"
|
||
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
|
||
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
|
||
```
|
||
- **Hybrid deployment** (tested, recommended):
|
||
`incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
|
||
combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
|
||
VM creation, per-node SEED_DATA (hostname, static IP), install monitoring,
|
||
and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
|
||
- **Self-registration**: nodes auto-register with OC within ~30s of first
|
||
boot. Hostname from SEED_DATA is used as the server name.
|
||
- **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires
|
||
all nodes to have `needs_update: false` before `provisioning cluster add`
|
||
succeeds. Nodes deployed from an ISO matching the latest OC update version
|
||
are tracked as `needs_update: true` because the OS was never delivered
|
||
through OC's update pipeline. The `needs_update` flag is server-side
|
||
computed (not a simple version comparison) and cannot be overridden via
|
||
REST API PUT. **Solution**: generate the ISO from an older channel
|
||
(`--channel old-stable`) so nodes start with an older version. OC then
|
||
pushes the latest update through its pipeline, clearing the flag. Use
|
||
`provisioning update assign-channels` to control which versions are in
|
||
which channels.
|
||
- **Cluster formation** (tested, 2026-02-23):
|
||
```bash
|
||
# Use empty app config if cert already injected via SEED_DATA
|
||
echo '{}' > /tmp/oc-app-config.yaml
|
||
operations-center provisioning cluster add oc-cluster \
|
||
https://<NODE_01_IP>:8443 \
|
||
--server-names oc-node-01,oc-node-02,oc-node-03 \
|
||
--server-type incus \
|
||
--application-seed-config /tmp/oc-app-config.yaml
|
||
```
|
||
OC handles: `core.https_address` → cluster enable → joins → storage pool
|
||
+ network creation → Terraform config. Adds `meshbr0` network. If the
|
||
client cert is already in the trust store (from SEED_DATA), use an empty
|
||
`{}` app config to avoid "Certificate already in trust store" Terraform
|
||
error. The cluster forms successfully either way.
|
||
- **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
|
||
for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
|
||
creation cleanly. With `apply_defaults: true`, nodes already have these
|
||
resources and OC's Terraform fails with "already exists" errors (cluster still
|
||
forms, but Terraform artifacts are empty).
|
||
- **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster
|
||
formation via OC, OVN overlay networking works identically to manual clusters.
|
||
Deploy ovn-central container, enable OVN services, create UPLINK + OVN
|
||
network, add ovn-chassis roles. HA nginx workload with OVN load balancer
|
||
tested and working. See `notes/operations-center-guide.md` for full guide.
|
||
- **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously
|
||
skipped uploading an ISO if one with the same filename existed on Proxmox.
|
||
This caused stale ISOs from previous deployments to be silently reused.
|
||
Fixed: when `--iso` is explicitly provided, the script now deletes the
|
||
existing ISO and re-uploads the new one.
|
||
|
||
#### Tested limitations
|
||
|
||
- **Inventory is NOT real-time** -- requires explicit `cluster resync`
|
||
- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
|
||
on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
|
||
~50s with data intact). The failure is OC-specific: the OC agent pushes
|
||
config via the IncusOS REST API that gets persisted to `state.txt`. On
|
||
reboot, invalid values (e.g., cron expression where Go duration is expected)
|
||
crash the daemon. See "IncusOS boot failure" section below for full analysis.
|
||
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
|
||
- **No cluster member state tracking** -- OC always shows `ready` even for
|
||
EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat
|
||
detects failure in ~40s; OC has no equivalent).
|
||
- **Stale entries** from out-of-band cluster changes persist after resync
|
||
- **Server removal blocked** if server is part of an OC cluster
|
||
- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
|
||
node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
|
||
if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
|
||
- **`needs_update` tracking is pipeline-based**: OC tracks whether an update
|
||
was delivered through its pipeline, not just whether versions match. Nodes
|
||
deployed from the latest ISO are tracked as needing updates even when
|
||
`version == available_version`. Must deploy from older ISO to work around.
|
||
- **OVN LB has no health checks**: connection-based hashing distributes
|
||
traffic to dead backends. Requests to stopped instances return empty.
|
||
|
||
- See `notes/operations-center-guide.md` for full tested OC reference.
|
||
|
||
### incusos-proxmox doctor and cleanup
|
||
|
||
- **`--doctor`**: standalone environment check. No config file required.
|
||
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
|
||
Proxmox connectivity (from proxmox.yaml or config file).
|
||
- **`--cleanup`**: destroys VMs defined in the config file.
|
||
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
|
||
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
|
||
all IncusOS ISOs (unlike the old behavior).
|
||
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
|
||
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
|
||
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
|
||
and `seed-*.iso` from storage + remotes + cache.
|
||
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
||
output is concise (step names + results). `--quiet` suppresses everything
|
||
except warnings and errors.
|
||
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
|
||
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
|
||
Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for
|
||
up to 120s. With the `force_reboot` fix (2026-02-22), the crontab bug is
|
||
eliminated and retries are rarely needed. They remain as a safety net for
|
||
other transient boot failures. After retries, `fix_scrub_schedule()`
|
||
proactively heals any remaining scrub_schedule issues via the IncusOS
|
||
REST API.
|
||
- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot
|
||
downloads SecureBoot update + application sysext (Incus: ~30-120s depending
|
||
on CDN speed and concurrent VMs). The previous 90s timeout was too short
|
||
under load — premature retries (hard VM stop during first boot) can corrupt
|
||
the TPM encryption key, causing permanent "zfs load-key: Raw key too short
|
||
(expected 32)" errors that no amount of retries can fix.
|
||
|
||
### IncusOS first-boot sequence (observed via console screenshots)
|
||
|
||
Observed using `observe-deploy` with 2-second screenshot intervals. The
|
||
complete lifecycle from Proxmox VM start to port 8443 ready:
|
||
|
||
**Phase 1: ISO boot and installation (~60-85s)**
|
||
1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s)
|
||
2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s)
|
||
3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s)
|
||
- "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda"
|
||
- "Cloning GPT partitions" → progress bar → complete
|
||
4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via
|
||
`/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not
|
||
a graceful systemd reboot. This is intentional — the install
|
||
environment is minimal and may not have systemd running.
|
||
5. VM reboots but ISO still attached → boots from ISO again
|
||
6. "System check error: install media detected, but the system is already
|
||
installed; please remove USB/CDROM and reboot the system" (~40s).
|
||
IncusOS daemon sleeps **1 hour** at this error, then exits. The
|
||
`state.txt` on the installed disk (scsi0 data partition) is **NOT
|
||
touched** — this boot runs from the ISO rootfs (tmpfs).
|
||
7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle)
|
||
|
||
**Phase 2: Transition (~15s)**
|
||
1. Proxmox stops VM
|
||
2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0`
|
||
3. Start VM from disk
|
||
|
||
**Phase 3: First boot from disk (~50s to port 8443)**
|
||
1. UEFI → "IncusOS is starting..." (~8s)
|
||
2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets
|
||
defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`,
|
||
`Channel="stable"` → `Save()` writes state.txt to data partition
|
||
3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s)
|
||
4. "System is starting up machine-id=... mode=production" (~10s after key)
|
||
5. "Bringing up the network" (immediate)
|
||
6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s)
|
||
7. "Downloading application update application=incus" + progress bar (~33-38s)
|
||
8. "Bringing up the local storage" (after download)
|
||
9. "Starting application" → "Initializing application" → TLS cert fingerprint
|
||
10. `registerJobs()` validates ScrubSchedule with gocron
|
||
11. "System is ready" — port 8443 now reachable
|
||
|
||
**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable
|
||
is the application download (33-38s on first boot; skipped if already cached).
|
||
|
||
**Important**: `DoInstall()` does NOT create `state.txt`. It writes the
|
||
IncusOS image to disk, processes seed files (including deleting `install.yaml`
|
||
from the target's seed partition via `CleanupPostInstall()`), and triggers
|
||
the SysRq-B reboot. The `state.txt` is created exclusively on first boot
|
||
from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
||
|
||
### IncusOS boot failure (crontab / update frequency error)
|
||
|
||
- **RESOLVED** (2026-02-22): root cause identified and fixed in our pipeline.
|
||
The `force_reboot` seed option triggers SysRq-B after install, causing an
|
||
"install media detected" intermediate boot. This intermediate boot generates
|
||
~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs).
|
||
Our blockstat detection catches these late writes, but by then the
|
||
intermediate boot has run and somehow corrupted the state for the subsequent
|
||
first-from-disk boot. **Fix: omit `force_reboot` from the seed.** Without
|
||
it, the installer sits at "please remove media" after the 876 MiB image
|
||
clone, blockstat detects idle cleanly, and the first disk boot is pristine.
|
||
Results: **15/15 PASS (100%)** without force_reboot on 4 cores vs 50% with it.
|
||
- **Symptom**: "ERROR invalid crontab expression" appears on the console
|
||
during first boot. Intermittent — confirmed **~50% failure rate** with
|
||
`force_reboot: true` on 4-core VMs. Error does NOT correlate with
|
||
`update.yaml` presence/absence, ISO version, or version mismatch.
|
||
- **Two distinct error paths in IncusOS**:
|
||
- `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
|
||
like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
|
||
daemon exits — **this is the fatal one**. Port 8443 never opens
|
||
(or opens briefly during the 15-second error sleep).
|
||
- `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
|
||
format like `"6h"`). If invalid, it logs an error but does **not** crash.
|
||
- **Not always fatal** (confirmed by parallel investigation):
|
||
- The REST API starts BEFORE `startup()` completes. Applications (Incus)
|
||
start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
|
||
When Incus starts before the scheduler crashes, port 8443 is available
|
||
during the 15-second error sleep (before `os.Exit(1)`).
|
||
- Port 8443 is reachable during the error window, and API calls work
|
||
(including `/os/1.0/system/storage`). The `scrub_schedule` field is
|
||
empty in the API response — this is the definitive indicator.
|
||
- "System is ready" is NOT shown on console (only appears after
|
||
`registerJobs()` succeeds). Console shows "ERROR invalid crontab
|
||
expression" as the last line when the bug hits.
|
||
- **Detection heuristic**: port 8443 reachable BUT `scrub_schedule` is
|
||
empty → crontab bug hit. Port 8443 reachable AND `scrub_schedule` is
|
||
`"0 4 * * 0"` → genuine success.
|
||
- **Source code analysis** (root cause investigation):
|
||
- `state.txt` is created FRESH on first boot from disk by
|
||
`LoadOrCreate()` → `initialize()`. The `initialize()` function
|
||
correctly sets `ScrubSchedule = "0 4 * * 0"`.
|
||
- The "install media detected" boot (#2, ISO still attached) runs from
|
||
the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed
|
||
disk. The installed disk's state.txt is untouched.
|
||
- `DoInstall()` does not create state.txt. It only writes the OS image
|
||
and processes seed files.
|
||
- **The encoder skips zero values**: `encodeHelper()` returns early on
|
||
`v.IsZero()`. An empty ScrubSchedule string is NOT written to file.
|
||
If state.txt is read with a missing ScrubSchedule, the field stays
|
||
at Go's zero value (empty string) → `gocron.IsValid("")` fails.
|
||
- **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which
|
||
truncates then writes. Crash during write → partial file.
|
||
- **Race window**: The REST API server starts BEFORE `startup()`
|
||
completes. Applications start before `registerJobs()`. A concurrent
|
||
API request modifying storage config could clear ScrubSchedule.
|
||
- **Root cause confirmed**: the bug is triggered by the SysRq-B
|
||
intermediate boot (from `force_reboot: true`). Controlled experiments
|
||
on VLAN 69 with static IP (2026-02-22, ISO 202602210344):
|
||
- Baseline (4 cores, force_reboot): 5/10 PASS (50%)
|
||
- cache=writethrough: 3/5 PASS (60%) — falsified
|
||
- agent=0: 3/5 PASS (60%) — falsified
|
||
- cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race
|
||
- **No force_reboot (4 cores): 15/15 PASS (100%)** — confirmed fix
|
||
Correlation: runs with 876 MiB blockstat detection (before SysRq-B)
|
||
had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%.
|
||
- **The try-it service is NOT IncusOS**: stgraber's comparison ("thousands
|
||
of times a day") is invalid. The try-it service runs Ubuntu VMs with
|
||
Incus installed from the Zabbly daily repo — a completely different
|
||
product. Issue #843 was independently filed by another Proxmox user
|
||
(fperreau on Proxmox 9.1.4), confirming it's not unique to our setup.
|
||
- The underlying IncusOS bug is still a race condition between the REST
|
||
API/application startup and `registerJobs()` that clears `ScrubSchedule`.
|
||
Investigation ruled out version mismatch, update downloads, and gocron
|
||
library issues. The field is genuinely empty in the state struct.
|
||
- **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel):
|
||
Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%);
|
||
Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG
|
||
(67% failure rate). Console screenshots confirm "ERROR invalid crontab
|
||
expression" on failed VMs and "System is ready" on successful VMs.
|
||
- **Version mismatch NOT the cause**: stgraber's hypothesis that the bug
|
||
relates to ISO version != CDN latest was tested and ruled out. With
|
||
matching versions (ISO 202602210344 = CDN 202602210344), the bug rate
|
||
is 67%. Sysext downloads still occur with matching versions (SecureBoot
|
||
+ application updates) but no OS update download. The bug occurs
|
||
regardless of whether sysext downloads happen.
|
||
- **Sysext download on matching versions**: even when ISO version matches
|
||
CDN latest, the first boot downloads SecureBoot update (~1s) and
|
||
application sysext (~2 min for Incus). The sysext is NOT in the ISO —
|
||
it's always downloaded from the update provider. An OS update download
|
||
(and reboot) only occurs when versions differ.
|
||
- **Proposed upstream fix** (for `incus-os` project):
|
||
```go
|
||
// In registerJobs(): validate and fall back to default
|
||
func registerJobs(s *state.State) error {
|
||
schedule := s.System.Storage.Config.ScrubSchedule
|
||
if schedule == "" {
|
||
schedule = "0 4 * * 0"
|
||
s.System.Storage.Config.ScrubSchedule = schedule
|
||
}
|
||
err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools)
|
||
if err != nil {
|
||
// Fall back to default instead of crashing the daemon
|
||
slog.Warn("Invalid scrub schedule, using default", "schedule", schedule)
|
||
s.System.Storage.Config.ScrubSchedule = "0 4 * * 0"
|
||
return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools)
|
||
}
|
||
return nil
|
||
}
|
||
```
|
||
Additional recommendations: (1) use atomic writes in `Save()` (write
|
||
to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even
|
||
when empty (to preserve the default through encode/decode cycles),
|
||
(3) add mutex protection on State fields shared between API handlers
|
||
and startup().
|
||
- **Error is persistent within a VM session**: verified by screenshots —
|
||
when the crontab error hits, the deferred `s.Save()` writes corrupt
|
||
state (version 7, ScrubSchedule missing because encoder skips zero
|
||
values). Subsequent systemd daemon restarts read the corrupt state
|
||
and crash again. The error does NOT self-heal via daemon restart alone.
|
||
- **Recovery via Proxmox stop+start**: `phase_install` in `incusos-proxmox`
|
||
retries once with a full Proxmox stop (hard VM power-off) → 10s wait →
|
||
start. The hard power-off may cause uncommitted filesystem writes
|
||
(including the corrupt state.txt) to be lost, restoring the correct
|
||
state from `initialize()`. This works because IncusOS's data partition
|
||
has a journal commit interval (~5s) and the corrupt Save() may not have
|
||
been committed before the VM was killed.
|
||
- **Detection**: `phase_install` checks port 8443 after starting from disk
|
||
(60s wait + up to 120s polling = 180s total). If first check fails,
|
||
automatic retry adds ~180s. Failed VMs after retry are reported with
|
||
remediation.
|
||
- **TPM corruption from premature retry**: hard-stopping a VM during first
|
||
boot (while the encryption key is being written to the TPM zvol) can
|
||
leave the TPM with truncated key data. All subsequent boots fail with
|
||
"zfs load-key: Raw key too short (expected 32)" — a permanent error
|
||
that requires VM destruction and redeployment to fix. The 180s boot
|
||
timeout prevents this by allowing the full first-boot sequence (including
|
||
sysext download) to complete before any retry.
|
||
- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
|
||
`fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
|
||
every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix
|
||
proxies to the IncusOS daemon API through Incus. Safe to call on every
|
||
node — returns early if schedule is already set. Called on both initial
|
||
success and retry success paths.
|
||
- **Filed upstream**: IncusOS issue #843.
|
||
- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
|
||
`updateChecker()`, `startup()`, `firstBootActions()`),
|
||
`incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,
|
||
`ErrInvalidCronTab`), `incus-osd/internal/state/file.go`
|
||
(`LoadOrCreate()`, `Save()`, `initialize()`),
|
||
`incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip),
|
||
`incus-osd/internal/install/install.go` (`DoInstall()`,
|
||
`rebootUponDeviceRemoval()` SysRq-B).
|
||
|
||
### Proxmox SSH root access — strict rules
|
||
|
||
Root SSH access to the Proxmox host is available for **diagnostics only**.
|
||
The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`.
|
||
|
||
**MANDATORY RULES — violation of any rule is unacceptable:**
|
||
|
||
1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM
|
||
console screenshots via `qm monitor <vmid> screendump`. No other use
|
||
without explicit user instruction.
|
||
2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869)
|
||
or VMs in the IncusLab pool that were created by our scripts.
|
||
3. **No modifications**: NEVER run any command that modifies, stops, starts,
|
||
or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`,
|
||
`pct` commands, `zfs` commands, `systemctl` commands, or ANY write
|
||
operation on the host.
|
||
4. **Do not touch the dev VM**: the user's dev VM runs on the same host.
|
||
Do not interact with it in any way — do not even `qm status` it.
|
||
5. **No config access**: do not read or modify `/etc/pve/`, storage configs,
|
||
network configs, user/ACL settings, or any host-level configuration.
|
||
6. **Transparency**: every SSH command is visible in tool output. If the
|
||
user rejects a command, do not retry it.
|
||
7. **Only during active tests**: only SSH during test runs where test VMs
|
||
have been deployed, and only to screenshot those test VMs.
|
||
|
||
**Permitted commands (exhaustive list):**
|
||
```bash
|
||
# Take screenshot of a test VM (VMID in 850-869 range)
|
||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||
root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"
|
||
|
||
# Retrieve screenshot
|
||
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
|
||
root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/
|
||
|
||
# Cleanup screenshot on remote
|
||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||
root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
|
||
```
|
||
|
||
**Technical notes:**
|
||
- QEMU's `screendump` is root-only in PVE 9's HMP permission model.
|
||
API tokens cannot execute it regardless of privileges.
|
||
- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng,
|
||
which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`).
|
||
- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`:
|
||
`python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"`
|
||
PNG output is ~20 KB and can be read directly by Claude Code (multimodal).
|
||
|
||
## Coding conventions for scripts
|
||
|
||
- **Shell**: bash with `set -euo pipefail`
|
||
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
|
||
false exits under `set -e` when the value is 0.
|
||
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
|
||
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
|
||
- **Defaults**: sane defaults so the script does something useful with zero flags.
|
||
- **Dry run**: all scripts should support `--dry-run` to preview actions.
|
||
- **Cert detection order**: files on disk first, CLI command second.
|
||
- **Error messages**: include actionable remediation steps, not just "failed".
|
||
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
||
not "sudo apt install incus".
|
||
|
||
### Incus networking (OVN)
|
||
|
||
- **Bridge networks are node-local**: each cluster member has its own
|
||
independent bridge. Instances on the same bridge (same node) can
|
||
communicate; cross-node instances CANNOT. Each bridge has the same subnet
|
||
(e.g., 10.0.0.1/24) but they are separate L2 domains.
|
||
- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes.
|
||
Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
|
||
+ client services + physical uplink network.
|
||
- **IncusOS OVN services are disabled by default**: must be enabled via the
|
||
IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring
|
||
Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection`
|
||
fails with `db.sock not found`.
|
||
- **OVN service enable API call**:
|
||
```bash
|
||
incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
|
||
"config": {
|
||
"database": "tcp:<SB_HOST>:6642",
|
||
"enabled": true,
|
||
"tunnel_address": "<THIS_NODE_LAN_IP>",
|
||
"tunnel_protocol": "geneve"
|
||
},
|
||
"state": {}
|
||
}'
|
||
```
|
||
`database` is the **southbound** DB (port 6642), NOT northbound (6641).
|
||
- **OVN control plane as container**: deploy `ovn-central` package in a
|
||
Debian container on the cluster. Use proxy devices to expose NB (6641)
|
||
and SB (6642) ports on the host's LAN IP so all nodes can reach it.
|
||
- **Setup sequence** (order matters):
|
||
1. Deploy OVN control plane container
|
||
2. Enable OVN services on ALL IncusOS nodes
|
||
3. `incus config set network.ovn.northbound_connection tcp:<host>:6641`
|
||
4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes)
|
||
5. Create physical uplink network (two-step cluster pattern)
|
||
6. Create OVN network with `--type=ovn network=UPLINK`
|
||
- **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC —
|
||
NOT `ens18`, which is the underlying device name but not exposed to Incus).
|
||
`ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses,
|
||
`ipv4.gateway` is the LAN gateway in CIDR format.
|
||
- **OVN network isolation**: multiple OVN networks are fully isolated.
|
||
Instances on different networks cannot communicate, even on the same node.
|
||
Network peering (`incus network peer create`) enables cross-network routing.
|
||
- **OVN features tested**: cross-node connectivity, network isolation,
|
||
ACLs (per-source blocking), network peering, L4 load balancers (connection-
|
||
based hashing, not round-robin), network forwards (port forwarding to LAN
|
||
IPs), DNS resolution (per-network, hostname.incus domain).
|
||
- **LB/forward backends require IP addresses**: `incus network load-balancer
|
||
backend add` and `incus network forward port add` require the target
|
||
instance's IP address, NOT its name. Using instance names fails with
|
||
"Invalid target address".
|
||
- See `notes/networking-guide.md` for full tutorial with test results.
|
||
|
||
### Shared storage (iSCSI + lvmcluster)
|
||
|
||
- **iSCSI + lvmcluster** is the IncusOS-native path to shared storage.
|
||
All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS
|
||
and enabled via the REST API — no packages to install.
|
||
- **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN
|
||
field is `"target"` (NOT `"iqn"` — using the wrong field silently fails).
|
||
The config lives under `config.targets[]`, and `state` returns the
|
||
auto-generated `initiator_name`.
|
||
- **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique
|
||
`system_id` (1-2000) per node for sanlock host identification. Using 0
|
||
or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool
|
||
creation.
|
||
- **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on
|
||
custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately.
|
||
~256 MiB overhead for LVM metadata + sanlock lease area.
|
||
- **Pool creation**: two-step cluster pattern (`--target` per member, then
|
||
finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`,
|
||
and `vgchange --lock-start` automatically.
|
||
- **Migration performance** (tested on 1GbE, 2026-02-23):
|
||
- Container stop/move/start: **0.12-0.15s** (metadata only)
|
||
- VM non-live (stop/move/start): **1.8s** (LVM metadata update)
|
||
- VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer)
|
||
- Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data)
|
||
- **Proxy devices don't work for iSCSI**: SendTargets discovery returns
|
||
container IP in TargetAddress, causing portal mismatch. Use direct network
|
||
paths (bridge for same-node, macvlan for cross-node).
|
||
- **Lab target container**: Debian container with `tgt` (userspace iSCSI
|
||
target) on the cluster — no external hardware needed. Uses dual network:
|
||
bridge IP for same-node access, macvlan on `mgmt` for cross-node access.
|
||
- **First live migration after stop/start may fail**: QEMU on destination
|
||
fails to start (transient sanlock lease issue). Retry succeeds. Non-live
|
||
migration always works as fallback.
|
||
- **Hybrid architecture recommended**: local ZFS (`local` pool) for general
|
||
workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant
|
||
migration.
|
||
- See `notes/shared-storage-guide.md` for the full tested walkthrough.
|
||
|
||
### Migration into Incus
|
||
|
||
- **`incus-migrate`**: official tool for importing disk images, running
|
||
instances, or physical machines into Incus.
|
||
- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2,
|
||
raw, vdi, vhd formats. Incus accepts raw and qcow2.
|
||
- **Import workflow**: convert disk → `incus storage volume import` →
|
||
`incus init --empty --vm` → attach disk → start.
|
||
- **Container migration**: `docker export` → `incus import` for filesystem-
|
||
level container migration. Docker volumes must be copied separately.
|
||
- See `notes/migration-guide.md` for full procedures per source hypervisor.
|
||
|
||
### UTM support (future)
|
||
|
||
- Design document at `notes/utm-support.md`.
|
||
- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation
|
||
(requires AppleScript or `.utm` bundle generation).
|
||
- No `blockstat` equivalent -- install detection must use timeout + port polling.
|
||
- Seed generation already works cross-platform (Phase 3 macOS compatibility).
|
||
|
||
## Git workflow
|
||
|
||
- Main branch: `main`
|
||
- Development happens on feature branches
|
||
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`
|