# CLAUDE.md - Project context for AI assistants

## What this repository is

A collection of peripheral tools, scripts, and snippets for working with
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
Primarily targeting home lab environments but aiming for production-quality scripts.

## Repository structure

```
incu-contrib/
├── CLAUDE.md              # This file -- project context
├── README.md              # Main overview
├── .gitignore
├── incusos/               # IncusOS installation tooling
│   ├── README.md          # Detailed usage docs
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment
│   ├── TESTING.md         # Testing guide for incusos-proxmox
│   └── examples/          # Example seed + Proxmox YAML files
└── notes/                 # Research notes and reference material
```

## Key technical context

### Incus version differences

- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
  The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
  macOS is client-only by design; Arch has no client-only split package.
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
  not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
  Fall back to the CLI command only as a secondary option.
- See `notes/incus-version-compatibility.md` for full platform matrix and
  install instructions.

### IncusOS flasher-tool

- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
  For cross-arch builds, download the image manually and pass via `--image`.
- CDN index: `https://images.linuxcontainers.org/os/index.json`
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`

### Seed archives

- Tar archives containing YAML files at the root level.
- Written to byte offset 2148532224 (the seed partition) in the image.
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
  - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
    devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
  - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
    **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
    FAT filesystem labels, so IncusOS cannot find the seed.
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
  `operations-center.yaml`, `network.yaml`, `update.yaml`.

### Client certificates

- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
- Running `incus remote list` triggers auto-generation if no keypair exists.
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
  The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
  `certificates`) are promoted to the top level of the `preseed` object.
- For Operations Center seed: injected under `trusted_client_certificates[]`.
- Operations Center **requires** at least one trusted certificate -- without it,
  you are locked out after installation.

### Proxmox VE deployment

- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
  via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
  VMs with IncusOS-correct settings, and boots them through installation.
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
  (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
- **Minimum API privileges** for token-based access:
  ```
  VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
  VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
  VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
  Datastore.Audit SDN.Use
  ```
- **Required VM settings** (getting any wrong causes IncusOS install failure):
  - `bios=ovmf`, `machine=q35` -- UEFI boot required
  - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
  - `tpmstate0`: `version=v2.0` -- required for disk encryption
  - `cpu=host` -- needed for x86_64_v3 instruction set requirement
  - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
  - `balloon=0` -- IncusOS manages memory internally
  - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
  - Minimum 50 GiB disk, minimum 4096 MiB RAM
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
  IncusOS does **literal** string matching (not glob) on disk device IDs.
  `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
  `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
- **Install flow** (automated by `incusos-proxmox`):
  1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
  2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
  3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
     when disk writes start then stop for 15s (3 stable polls), install is done
  4. Stop the VM (Proxmox stop, not guest shutdown)
  5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
     and refuses to start if found, regardless of boot order
  6. Set boot order to `order=scsi0` and start from disk
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
  ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
  ping broadcast → look up MAC in ARP table. Verify with direct ping before
  trusting the result.
- **force_reboot is required**: without `force_reboot: true` in the seed,
  IncusOS sits at "please remove installation media" and waits indefinitely.
  It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
  (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).

## Coding conventions for scripts

- **Shell**: bash with `set -euo pipefail`
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
  false exits under `set -e` when the value is 0.
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
- **Defaults**: sane defaults so the script does something useful with zero flags.
- **Dry run**: all scripts should support `--dry-run` to preview actions.
- **Cert detection order**: files on disk first, CLI command second.
- **Error messages**: include actionable remediation steps, not just "failed".
- **No hardcoded package managers**: say "install the Incus client" with a link,
  not "sudo apt install incus".

## Git workflow

- Main branch: `main`
- Development happens on feature branches
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`