incus-contrib/CLAUDE.md

# CLAUDE.md - Project context for AI assistants

## What this repository is

A collection of peripheral tools, scripts, and snippets for working with
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
Primarily targeting home lab environments but aiming for production-quality scripts.

## Repository structure

```
incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment
│   ├── lab-test           # Guided lab validation (workloads, clustering, migration)
│   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
```

## Key technical context

### Incus version differences

- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
  The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
  macOS is client-only by design; Arch has no client-only split package.
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
  not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
  Fall back to the CLI command only as a secondary option.
- See `notes/incus-version-compatibility.md` for full platform matrix and
  install instructions.

### IncusOS flasher-tool

- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
  For cross-arch builds, download the image manually and pass via `--image`.
- CDN index: `https://images.linuxcontainers.org/os/index.json`
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`

### Seed archives

- Tar archives containing YAML files at the root level.
- Written to byte offset 2148532224 (the seed partition) in the image.
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
  - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
    devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
  - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
    **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
    FAT filesystem labels, so IncusOS cannot find the seed.
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
  `operations-center.yaml`, `network.yaml`, `update.yaml`.

### Client certificates

- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
- Running `incus remote list` triggers auto-generation if no keypair exists.
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
  The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
  `certificates`) are promoted to the top level of the `preseed` object.
- For Operations Center seed: injected under `trusted_client_certificates[]`.
- Operations Center **requires** at least one trusted certificate -- without it,
  you are locked out after installation.

### Proxmox VE deployment

- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
  via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
  VMs with IncusOS-correct settings, and boots them through installation.
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
  (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
- **Minimum API privileges** for token-based access:
  ```
  VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
  VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
  VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
  Datastore.Audit SDN.Use
  ```
- **Required VM settings** (getting any wrong causes IncusOS install failure):
  - `bios=ovmf`, `machine=q35` -- UEFI boot required
  - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
  - `tpmstate0`: `version=v2.0` -- required for disk encryption
  - `cpu=host` -- needed for x86_64_v3 instruction set requirement
  - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
  - `balloon=0` -- IncusOS manages memory internally
  - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
  - Minimum 50 GiB disk, minimum 4096 MiB RAM
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
  IncusOS does **literal** string matching (not glob) on disk device IDs.
  `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
  `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
- **Install flow** (automated by `incusos-proxmox`):
  1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
  2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
  3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
     when disk writes start then stop for 15s (3 stable polls), install is done
  4. Stop the VM (Proxmox stop, not guest shutdown)
  5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
     and refuses to start if found, regardless of boot order
  6. Set boot order to `order=scsi0` and start from disk
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
  ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
  ping broadcast → look up MAC in ARP table. Verify with direct ping before
  trusting the result.
- **force_reboot is required**: without `force_reboot: true` in the seed,
  IncusOS sits at "please remove installation media" and waits indefinitely.
  It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
  (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
  all VM operations to a Proxmox resource pool. When set, the script only
  "sees" VMs in that pool (for collision detection and cleanup), and the API
  token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
  ```
  pveum pool add IncusLab --comment "IncusOS Lab VMs"
  pveum pool modify IncusLab --storage local-lvm,local
  pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
  ```
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
  deployment status (Proxmox state, install status, IP, port 8443, incus
  remote). Also runs post-deployment checks (Incus connectivity, Operations
  Center URL).
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
  config, an interactive menu offers: (1) run status checks, (2) continue
  install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
  `--yes`, defaults to option 1 (safe -- never auto-destroys).
- **Install idempotency**: `phase_install` checks each VM's state before
  acting -- already-running VMs are skipped, stopped-but-installed VMs are
  started from disk, only VMs with install media proceed through installation.

### Incus clustering via remotes

- **Cluster formation** is done entirely through the `incus` CLI using remotes.
  No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
- **Workflow:**
  ```bash
  # Enable clustering on first node
  incus cluster enable <remote1>: <member-name>

  # Generate join token (on cluster, for new member name)
  incus cluster add <remote1>: <new-member-name>

  # Join second node to cluster
  incus cluster join <remote1>: <remote2>:

  # Verify
  incus cluster list <remote1>:
  ```
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
  this ensures matching storage pool (`local`) and network bridge names, which
  are required for migration.
- After joining, the cluster is managed through the init node's remote. The
  individual node remotes still work for node-specific operations.
- **`lab-test`** automates cluster formation, workload testing, and migration.

### Lab validation (lab-test)

- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
  on the VM names defined there (expects `incus` remotes to exist).
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
  migrate (stop/move + evacuate/restore).
- **Test instances** use names starting with `test-` for easy cleanup.
- The script reports PASS/FAIL/SKIP for each test and prints a summary.

### Operations Center

- **CLI binary:** `operations-center` (installed from GitHub releases or
  built from source at `github.com/FuturFusion/operations-center`).
- **Config directory:** `~/.config/operations-center/` (uses same cert format
  as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
- **Provisioning workflow:** create token → get provisioned ISO → deploy nodes
  with that ISO → nodes auto-register with OC → form cluster via OC.
- **OC is under active development** (v0.2.2+). Commands and APIs may change.
  The `--doctor` command on `incusos-proxmox` reports whether the CLI is
  installed.

### incusos-proxmox doctor and cleanup

- **`--doctor`**: standalone environment check. No config file required.
  Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
  removes incus remotes + clears local ISO cache.
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
  `[incusos-lab:managed]` marker in the configured resource pool.

## Coding conventions for scripts

- **Shell**: bash with `set -euo pipefail`
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
  false exits under `set -e` when the value is 0.
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
- **Defaults**: sane defaults so the script does something useful with zero flags.
- **Dry run**: all scripts should support `--dry-run` to preview actions.
- **Cert detection order**: files on disk first, CLI command second.
- **Error messages**: include actionable remediation steps, not just "failed".
- **No hardcoded package managers**: say "install the Incus client" with a link,
  not "sudo apt install incus".

## Git workflow

- Main branch: `main`
- Development happens on feature branches
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`