incus-contrib/CLAUDE.md

# CLAUDE.md - Project context for AI assistants

## What this repository is

A collection of peripheral tools, scripts, and snippets for working with
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
Primarily targeting home lab environments but aiming for production-quality scripts.

## Repository structure

```
incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator (cross-platform: Linux + macOS)
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
│   ├── lab-test           # Guided lab validation (12 test phases)
│   ├── observe-deploy     # Single-VM deploy with rapid console screenshots
│   ├── observe-runs/      # Screenshot output from observe-deploy (gitignored)
│   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
│   ├── ../env             # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
    ├── clustering-guide.md          # Detailed Incus clustering walkthrough
    ├── operations-center-guide.md   # Operations Center provisioning & management
    ├── migration-guide.md           # Migration paths into Incus from other hypervisors
    └── utm-support.md               # UTM support design document (future)
```

## Key technical context

### Incus version differences

- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
  The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
  macOS is client-only by design; Arch has no client-only split package.
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
  not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
  Fall back to the CLI command only as a secondary option.
- See `notes/incus-version-compatibility.md` for full platform matrix and
  install instructions.

### IncusOS flasher-tool

- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
  For cross-arch builds, download the image manually and pass via `--image`.
- CDN index: `https://images.linuxcontainers.org/os/index.json`
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`

### Seed archives

- Tar archives containing YAML files at the root level.
- Written to byte offset 2148532224 (the seed partition) in the image.
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
  - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
    devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
  - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
    **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
    FAT filesystem labels, so IncusOS cannot find the seed.
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
  `operations-center.yaml`, `network.yaml`, `update.yaml`.

### Client certificates

- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
- Running `incus remote list` triggers auto-generation if no keypair exists.
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
  The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
  `certificates`) are promoted to the top level of the `preseed` object.
- For Operations Center seed: injected under `trusted_client_certificates[]`.
- Operations Center **requires** at least one trusted certificate -- without it,
  you are locked out after installation.

### Proxmox VE deployment

- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
  via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
  VMs with IncusOS-correct settings, and boots them through installation.
- **Config separation**: Proxmox connection settings (host, credentials, pool)
  live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
  etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
  `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
  for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
- **API token secret**: stored in `env` file at the repo root (gitignored).
  Scripts auto-load it on startup by searching for `env` up the directory
  tree from the script location. No manual `source env` needed. If the file
  is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail
  with a clear error message.
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
  (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
- **Minimum API privileges** for token-based access (role: `IncusOSDeployer`):
  ```
  VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
  VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
  VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
  Datastore.Audit SDN.Use Sys.Audit
  ```
  `Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via
  `/nodes/<node>/status`). It requires a separate ACL on `/nodes/<node>`
  since the pool-scoped ACL doesn't cover node-level endpoints.
- **Required VM settings** (getting any wrong causes IncusOS install failure):
  - `bios=ovmf`, `machine=q35` -- UEFI boot required
  - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
  - `tpmstate0`: `version=v2.0` -- required for disk encryption
  - `cpu=host` -- needed for x86_64_v3 instruction set requirement
  - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
  - `balloon=0` -- IncusOS manages memory internally
  - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
  - Minimum 50 GiB disk, minimum 4096 MiB RAM
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
  IncusOS does **literal** string matching (not glob) on disk device IDs.
  `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
  `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
- **Install flow** (automated by `incusos-proxmox`):
  1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
  2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
  3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
     when disk writes start then stop for 15s (3 stable polls), install is done
  4. Stop the VM (Proxmox stop, not guest shutdown)
  5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
     and refuses to start if found, regardless of boot order
  6. Set boot order to `order=scsi0` and start from disk
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
  ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
  ping broadcast → look up MAC in ARP table. Verify with direct ping before
  trusting the result.
- **force_reboot is required**: without `force_reboot: true` in the seed,
  IncusOS sits at "please remove installation media" and waits indefinitely.
  It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
  (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
  all VM operations to a Proxmox resource pool. When set, the script only
  "sees" VMs in that pool (for collision detection and cleanup), and the API
  token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
  ```
  pveum pool add IncusLab --comment "IncusOS Lab VMs"
  pveum pool modify IncusLab --storage local-lvm,local
  pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
  pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
  ```
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
  deployment status (Proxmox state, install status, IP, port 8443, incus
  remote). Also runs post-deployment checks (Incus connectivity, Operations
  Center URL).
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
  config, an interactive menu offers: (1) run status checks, (2) continue
  install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
  `--yes`, defaults to option 1 (safe -- never auto-destroys).
- **Install idempotency**: `phase_install` checks each VM's state before
  acting -- already-running VMs are skipped, stopped-but-installed VMs are
  started from disk, only VMs with install media proceed through installation.

### Multi-lab coexistence

- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop,
  VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk
  (refuses to start VMs with install media still attached). These are
  distinct from `--cleanup` (which destroys VMs permanently).
- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage
  usage, and per-pool allocation. Requires API method.
- **Lab inventory**: `--labs` scans the pool for managed VMs (by
  `[incusos-lab:managed]` marker), groups by config file, and shows
  per-lab status, VM count, RAM, and disk.
- **VMID range convention** (to avoid collisions between coexisting labs):

  | Range | Lab |
  |-------|-----|
  | 400-499 | OC-managed nodes |
  | 800-809 | Single-node labs |
  | 900-909 | Basic cluster |
  | 910-919 | OC combined (server + nodes) |
  | 920-929 | OC server standalone |
  | 930-939 | Advanced / heterogeneous |

### Resource constraints for multi-lab

- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node
  cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource
  where you can actually run out.
- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default
  (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk.
  LZ4 compression provides ~1.5-1.8x ratio on OS data.
- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs
  can share CPUs without contention.
- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if
  requested RAM exceeds available host RAM (API method only).
- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to
  run the full deploy pipeline (auto-accepts with `--yes`).
- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs
  (via storage content API) and notes thin provisioning for ZFS/LVM-thin.

### Incus clustering via remotes

- **Cluster formation** is done entirely through the `incus` CLI using remotes.
  No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
- **No VIP needed**: each node advertises its own IP as its cluster address.
  Clients can connect to any cluster member; requests are forwarded internally.

#### Pre-clustering: fix core.https_address

- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
  interfaces). Clustering requires a **specific routable IP** so nodes can
  address each other.
- **Set the IP on every node BEFORE enabling clustering:**
  ```bash
  incus config set <remote>: core.https_address <NODE_IP>:8443
  ```
- Get each node's routable IP via the API:
  ```bash
  incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
  ```
- This is safe to do while remotes are connected -- the remote already points
  to the specific IP; we're just narrowing the bind address. Certificate trust
  is fingerprint-based, not address-based.

#### Cluster enable (init node)

- ```bash
  incus cluster enable <remote>: <member-name>
  ```
  Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
  The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
- **TLS certificate regeneration**: enabling clustering causes the server to
  generate a new TLS certificate (cluster cert). The new cert may only have
  SANs for `127.0.0.1` and `::1`, breaking the existing remote.
- **Fix**: remove and re-add the remote to pin the new certificate:
  ```bash
  incus remote switch local         # if init remote is current default
  incus remote remove <remote>
  incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
  ```
- The cert trust on the server side (client → server) is unaffected -- it's
  stored by fingerprint in the Incus database, independent of listen address.

#### Joining nodes: apply_defaults and the storage pool conflict

- **Upstream recommendation**: use `apply_defaults: false` for nodes destined
  to join a cluster. The official IncusOS clustering tutorial states joining
  servers "cannot have preexisting networks or storage pools defined." With
  `apply_defaults: false`, the node still listens on port 8443, still trusts
  preseed certificates, and the underlying ZFS dataset (`local/incus`) still
  exists -- but no Incus storage pool or network metadata is created, so
  the join process works cleanly.
- **`apply_defaults: true` on joining nodes** is also functional but requires
  an 8-command cleanup per node before join (delete pool, network, volumes,
  profile devices). This is automated in `lab-test` but adds complexity.
- **Recommended seed pattern for clusters**:
  - Bootstrap/init node: `apply_defaults: true` (needs pool and network)
  - Joining nodes: `apply_defaults: false` (join process creates member-specific entries)
  - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional)
- **If `apply_defaults: true` was used**, the cleanup before join is:
  ```bash
  # 1. Remove config references
  incus config unset <remote>: storage.backups_volume
  incus config unset <remote>: storage.images_volume
  # 2. Delete volumes
  incus storage volume delete <remote>:local backups
  incus storage volume delete <remote>:local images
  # 3. Clear default profile references (pool is "in use" otherwise)
  incus profile device remove <remote>:default root
  incus profile device remove <remote>:default eth0
  # 4. Delete pool and network
  incus storage delete <remote>:local
  incus network delete <remote>:incusbr0
  ```

#### Join workflow

- **Generate token** (on init node, single argument `remote:member-name`):
  ```bash
  incus cluster add <init-remote>:<new-member-name>
  ```
- **Join** (interactive -- prompts for 5 values):
  ```bash
  incus cluster join <init-remote>: <joining-remote>:
  ```
  Interactive prompts and correct answers:
  1. IP address → accept default (node's IP, already set via core.https_address)
  2. Member name → accept default (matches the token)
  3. "All existing data is lost" → `yes`
  4. `source` property for storage pool "local" → `local/incus`
  5. `zfs.pool_name` property for storage pool "local" → `local/incus`
- **Automated (non-interactive):**
  ```bash
  printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
  ```
- **After join**: the joining node gets a new cluster certificate. Fix the
  remote (same as init node):
  ```bash
  incus remote remove <joining-remote>
  incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
  ```

#### Command syntax gotchas

- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
- `incus cluster add remote:member-name` -- ONE argument (no space)
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
  "yes/no" even with `--force`, pipe `printf "yes\n"` for automation
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
- `incus cluster restore remote:member-name` -- ONE argument (no space)
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
- `incus storage show remote:pool` -- ONE argument (no space)
- `incus storage show remote:pool --target member` -- target flag for
  member-specific config
- `incus config set remote: key value` -- remote with trailing colon + space
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
  colon) for targeting the server itself

#### Post-join state

- After joining, the cluster is managed through the init node's remote. The
  individual node remotes still work for node-specific operations.
- **`lab-test`** automates cluster formation, workload testing, and migration.

#### Workload placement and migration

- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
- **Container migration**: stop/move/start only (CRIU live migration is
  unreliable). Data persists, processes do not.
  ```bash
  incus stop <remote>:<instance>
  incus move <remote>:<instance> --target <destination-member>
  incus start <remote>:<instance>
  ```
- **VM live migration**: requires `migration.stateful=true` (must be set
  while VM is stopped). Preserves running state with no downtime.
  ```bash
  incus move <remote>:<instance> --target <destination-member>
  ```
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
  not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
  CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
  Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
  differently, causing `Missing section footer for ICH9LPC` on restore.
  Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
  fixed `sockets/cores/threads` topology — portable across all hosts.
  ```bash
  # WRONG: integer → maxcpus varies by host → migration fails
  incus config set <instance> limits.cpu=2
  # RIGHT: range → fixed topology → migration works everywhere
  incus config set <instance> limits.cpu=0-1
  ```
- **VM live migration works in nested virtualization** (IncusOS inside
  Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
  10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
  appears during migration is cosmetic. It fires from QEMU's feature
  dependency checker before KVM filters out unsupported features and does
  not affect migration.
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
  requires the `limits.cpu` range fix. Use `incus start --stateless` to
  discard a saved state file that cannot be restored.
- **VM `size.state` config**: stateful operations require `size.state` on
  the root disk (`incus config device add <instance> root disk path=/
  pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
  (ONE argument, like `cluster enable` and `cluster add`).
  Use `--action stop` if VMs lack the `limits.cpu` range fix.
  Restore with `incus cluster restore <remote>:<member> --force`.
- **VM agent reconnect**: after live migration, the incus agent inside the VM
  needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
  after migration may fail with "VM agent isn't currently running". Scripts
  should `sleep 4` after migration before running `incus exec`.
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
  heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
  work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
  usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
- **Concurrent migrations**: migrating multiple VMs simultaneously from
  different source nodes works without interference. ~140 MB/s per migration.
- **Active I/O during migration**: disk writes and network activity survive
  live migration transparently. File integrity verified after migration.
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
  joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
  auto-rebalanced.
  ```bash
  incus config set <remote>: cluster.rebalance.interval=1     # minutes
  incus config set <remote>: cluster.rebalance.threshold=10    # imbalance %
  incus config set <remote>: cluster.rebalance.batch=2         # max VMs/run
  incus config set <remote>: cluster.rebalance.cooldown=5m     # wait between runs
  ```
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
  join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
  for step-by-step instructions.
- **`incus cluster remove` requires confirmation**: even with `--force`, it
  prompts "Are you really sure?". Pipe `yes` for automation:
  ```bash
  printf "yes\n" | incus cluster remove <remote>:<member> --force
  ```
- See `notes/clustering-guide.md` for full details and references.

### Lab validation (lab-test)

- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
  on the VM names defined there (expects `incus` remotes to exist).
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
  migrate (stop/move + evacuate/restore).
- **Test instances** use names starting with `test-` for easy cleanup.
- The script reports PASS/FAIL/SKIP for each test and prints a summary.

### Operations Center

- **CLI binary:** `operations-center` (installed from GitHub releases or
  built from source at `github.com/FuturFusion/operations-center`).
- **Config directory:** `~/.config/operations-center/` (uses same cert format
  as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
  imported into the browser.
- **OC is under active development** (v0.2.2). Commands and APIs may change.
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
  installed.

#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)

- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
  must use structured format with section keys (`install:`, not flat).
  ```bash
  operations-center provisioning token seed add <UUID> proxmox-preseed \
      /tmp/preseed.yaml --description "Force reboot for Proxmox"
  operations-center provisioning token seed get-image <UUID> proxmox-preseed \
      /tmp/IncusOS-oc.iso --type iso --architecture x86_64
  ```
- **Hybrid deployment** (tested, recommended):
  `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
  combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
  VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring,
  and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
- **Self-registration**: nodes auto-register with OC within ~30s of first
  boot. Hostname from SEED_DATA is used as the server name.
- **Cluster formation** (tested):
  ```bash
  operations-center provisioning cluster add oc-cluster \
      https://<NODE_01_IP>:8443 \
      --server-names oc-node-01,oc-node-02,oc-node-03 \
      --server-type incus \
      --application-seed-config /tmp/oc-app-config.yaml
  ```
  OC handles: `core.https_address` → cluster enable → joins → seed
  application → Terraform config. Adds `meshbr0` network.
- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
  nodes already have storage pool/network/cert. OC's Terraform fails with
  "already exists" errors but the cluster forms successfully. Use
  `apply_defaults: false` for OC-managed nodes to avoid this.

#### Tested limitations

- **Inventory is NOT real-time** -- requires explicit `cluster resync`
- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
  on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
  ~50s with data intact). The failure is OC-specific: the OC agent pushes
  config via the IncusOS REST API that gets persisted to `state.txt`. On
  reboot, invalid values (e.g., cron expression where Go duration is expected)
  crash the daemon. See "IncusOS boot failure" section below for full analysis.
  Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
- **No cluster member state tracking** -- OC always shows `ready` even for
  EVACUATED/OFFLINE nodes
- **Stale entries** from out-of-band cluster changes persist after resync
- **Server removal blocked** if server is part of an OC cluster

- See `notes/operations-center-guide.md` for full tested OC reference.

### incusos-proxmox doctor and cleanup

- **`--doctor`**: standalone environment check. No config file required.
  Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
  Proxmox connectivity (from proxmox.yaml or config file).
- **`--cleanup`**: destroys VMs defined in the config file.
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
  deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
  all IncusOS ISOs (unlike the old behavior).
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
  config required). Destroys all VMs with `[incusos-lab:managed]` marker.
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
  and `seed-*.iso` from storage + remotes + cache.
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
  output is concise (step names + results). `--quiet` suppresses everything
  except warnings and errors.

### IncusOS first-boot sequence (observed via console screenshots)

Observed using `observe-deploy` with 2-second screenshot intervals. The
complete lifecycle from Proxmox VM start to port 8443 ready:

**Phase 1: ISO boot and installation (~60-85s)**
1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s)
2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s)
3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s)
   - "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda"
   - "Cloning GPT partitions" → progress bar → complete
4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via
   `/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not
   a graceful systemd reboot. This is intentional — the install
   environment is minimal and may not have systemd running.
5. VM reboots but ISO still attached → boots from ISO again
6. "System check error: install media detected, but the system is already
   installed; please remove USB/CDROM and reboot the system" (~40s).
   IncusOS daemon sleeps **1 hour** at this error, then exits. The
   `state.txt` on the installed disk (scsi0 data partition) is **NOT
   touched** — this boot runs from the ISO rootfs (tmpfs).
7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle)

**Phase 2: Transition (~15s)**
1. Proxmox stops VM
2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0`
3. Start VM from disk

**Phase 3: First boot from disk (~50s to port 8443)**
1. UEFI → "IncusOS is starting..." (~8s)
2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets
   defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`,
   `Channel="stable"` → `Save()` writes state.txt to data partition
3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s)
4. "System is starting up machine-id=... mode=production" (~10s after key)
5. "Bringing up the network" (immediate)
6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s)
7. "Downloading application update application=incus" + progress bar (~33-38s)
8. "Bringing up the local storage" (after download)
9. "Starting application" → "Initializing application" → TLS cert fingerprint
10. `registerJobs()` validates ScrubSchedule with gocron
11. "System is ready" — port 8443 now reachable

**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable
is the application download (33-38s on first boot; skipped if already cached).

**Important**: `DoInstall()` does NOT create `state.txt`. It writes the
IncusOS image to disk, processes seed files (including deleting `install.yaml`
from the target's seed partition via `CleanupPostInstall()`), and triggers
the SysRq-B reboot. The `state.txt` is created exclusively on first boot
from the installed disk by `state.LoadOrCreate()` → `initialize()`.

### IncusOS boot failure (crontab / update frequency error)

- **Symptom**: "ERROR invalid crontab expression" appears on the console
  during first boot. Intermittent — observed in ~50% of deploys (2/4
  observational deploys). Error does NOT correlate with `update.yaml`
  presence/absence.
- **Two distinct error paths in IncusOS**:
  - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
    like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
    daemon exits — **this is the fatal one**. Port 8443 never opens
    (or opens briefly during the 15-second error sleep).
  - `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
    format like `"6h"`). If invalid, it logs an error but does **not** crash.
- **Not always fatal** (key finding from observational testing):
  - The REST API starts BEFORE `startup()` completes. Applications (Incus)
    start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
    When Incus starts before the scheduler crashes, port 8443 is available
    during the 15-second error sleep (before `os.Exit(1)`).
  - `observe-deploy` detects port 8443 during this window → reports PASS.
  - "System is ready" is NOT shown on console (only appears after
    `registerJobs()` succeeds).
- **Source code analysis** (root cause investigation):
  - `state.txt` is created FRESH on first boot from disk by
    `LoadOrCreate()` → `initialize()`. The `initialize()` function
    correctly sets `ScrubSchedule = "0 4 * * 0"`.
  - The "install media detected" boot (#2, ISO still attached) runs from
    the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed
    disk. The installed disk's state.txt is untouched.
  - `DoInstall()` does not create state.txt. It only writes the OS image
    and processes seed files.
  - **The encoder skips zero values**: `encodeHelper()` returns early on
    `v.IsZero()`. An empty ScrubSchedule string is NOT written to file.
    If state.txt is read with a missing ScrubSchedule, the field stays
    at Go's zero value (empty string) → `gocron.IsValid("")` fails.
  - **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which
    truncates then writes. Crash during write → partial file.
  - **Race window**: The REST API server starts BEFORE `startup()`
    completes. Applications start before `registerJobs()`. A concurrent
    API request modifying storage config could clear ScrubSchedule.
  - **Most likely root cause**: either (a) a race condition between the
    REST API/application startup and `registerJobs()` that clears
    `ScrubSchedule`, or (b) a gocron library edge case where
    `IsValid("0 4 * * 0", time.UTC, time.Now())` intermittently fails.
    Both explain the ~50% hit rate and lack of correlation with seed
    content.
- **Proposed upstream fix** (for `incus-os` project):
  ```go
  // In registerJobs(): validate and fall back to default
  func registerJobs(s *state.State) error {
      schedule := s.System.Storage.Config.ScrubSchedule
      if schedule == "" {
          schedule = "0 4 * * 0"
          s.System.Storage.Config.ScrubSchedule = schedule
      }
      err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools)
      if err != nil {
          // Fall back to default instead of crashing the daemon
          slog.Warn("Invalid scrub schedule, using default", "schedule", schedule)
          s.System.Storage.Config.ScrubSchedule = "0 4 * * 0"
          return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools)
      }
      return nil
  }
  ```
  Additional recommendations: (1) use atomic writes in `Save()` (write
  to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even
  when empty (to preserve the default through encode/decode cycles),
  (3) add mutex protection on State fields shared between API handlers
  and startup().
- **Recovery via restart**: `phase_install` in `incusos-proxmox` now
  automatically retries once (stop → 10s wait → start → re-check).
  After the first crash, the deferred `s.Save()` writes state. On the
  second boot, systemd restarts the daemon; if the root cause was
  transient (race condition or gocron edge case), the second boot
  succeeds.
- **Detection**: `phase_install` checks port 8443 after starting from disk
  (30s wait + up to 60s polling). If first check fails, automatic retry
  adds ~100s. Failed VMs after retry are reported with remediation.
- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
  `updateChecker()`, `startup()`, `firstBootActions()`),
  `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,
  `ErrInvalidCronTab`), `incus-osd/internal/state/file.go`
  (`LoadOrCreate()`, `Save()`, `initialize()`),
  `incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip),
  `incus-osd/internal/install/install.go` (`DoInstall()`,
  `rebootUponDeviceRemoval()` SysRq-B).

### Proxmox SSH root access — strict rules

Root SSH access to the Proxmox host is available for **diagnostics only**.
The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`.

**MANDATORY RULES — violation of any rule is unacceptable:**

1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM
   console screenshots via `qm monitor <vmid> screendump`. No other use
   without explicit user instruction.
2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869)
   or VMs in the IncusLab pool that were created by our scripts.
3. **No modifications**: NEVER run any command that modifies, stops, starts,
   or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`,
   `pct` commands, `zfs` commands, `systemctl` commands, or ANY write
   operation on the host.
4. **Do not touch the dev VM**: the user's dev VM runs on the same host.
   Do not interact with it in any way — do not even `qm status` it.
5. **No config access**: do not read or modify `/etc/pve/`, storage configs,
   network configs, user/ACL settings, or any host-level configuration.
6. **Transparency**: every SSH command is visible in tool output. If the
   user rejects a command, do not retry it.
7. **Only during active tests**: only SSH during test runs where test VMs
   have been deployed, and only to screenshot those test VMs.

**Permitted commands (exhaustive list):**
```bash
# Take screenshot of a test VM (VMID in 850-869 range)
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"

# Retrieve screenshot
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
    root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/

# Cleanup screenshot on remote
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
    root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
```

**Technical notes:**
- QEMU's `screendump` is root-only in PVE 9's HMP permission model.
  API tokens cannot execute it regardless of privileges.
- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng,
  which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`).
- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`:
  `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"`
  PNG output is ~20 KB and can be read directly by Claude Code (multimodal).

## Coding conventions for scripts

- **Shell**: bash with `set -euo pipefail`
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
  false exits under `set -e` when the value is 0.
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
- **Defaults**: sane defaults so the script does something useful with zero flags.
- **Dry run**: all scripts should support `--dry-run` to preview actions.
- **Cert detection order**: files on disk first, CLI command second.
- **Error messages**: include actionable remediation steps, not just "failed".
- **No hardcoded package managers**: say "install the Incus client" with a link,
  not "sudo apt install incus".

### Migration into Incus

- **`incus-migrate`**: official tool for importing disk images, running
  instances, or physical machines into Incus.
- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2,
  raw, vdi, vhd formats. Incus accepts raw and qcow2.
- **Import workflow**: convert disk → `incus storage volume import` →
  `incus init --empty --vm` → attach disk → start.
- **Container migration**: `docker export` → `incus import` for filesystem-
  level container migration. Docker volumes must be copied separately.
- See `notes/migration-guide.md` for full procedures per source hypervisor.

### UTM support (future)

- Design document at `notes/utm-support.md`.
- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation
  (requires AppleScript or `.utm` bundle generation).
- No `blockstat` equivalent -- install detection must use timeout + port polling.
- Seed generation already works cross-platform (Phase 3 macOS compatibility).

## Git workflow

- Main branch: `main`
- Development happens on feature branches
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`