version and certificate refinement

CLAUDE.md

@@ -0,0 +1,84 @@
+# CLAUDE.md - Project context for AI assistants
+
+## What this repository is
+
+A collection of peripheral tools, scripts, and snippets for working with
+Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
+Primarily targeting home lab environments but aiming for production-quality scripts.
+
+## Repository structure
+
+```
+incu-contrib/
+├── CLAUDE.md              # This file -- project context
+├── README.md              # Main overview
+├── .gitignore
+├── incusos/               # IncusOS installation tooling
+│   ├── README.md          # Detailed usage docs
+│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
+│   ├── incusos-seed       # Seed archive generator
+│   └── examples/          # Example seed YAML files
+└── notes/                 # Research notes and reference material
+```
+
+## Key technical context
+
+### Incus version differences
+
+- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
+  The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
+- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
+  macOS is client-only by design; Arch has no client-only split package.
+- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
+  not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
+- Always prefer reading `~/.config/incus/client.crt` directly from disk.
+  Fall back to the CLI command only as a secondary option.
+- See `notes/incus-version-compatibility.md` for full platform matrix and
+  install instructions.
+
+### IncusOS flasher-tool
+
+- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
+- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
+- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
+- There is NO `--arch` flag -- architecture is determined by the downloaded image.
+  For cross-arch builds, download the image manually and pass via `--image`.
+- CDN index: `https://images.linuxcontainers.org/os/index.json`
+- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`
+
+### Seed archives
+
+- Tar archives containing YAML files at the root level.
+- Written to byte offset 2148532224 (the seed partition) in the image.
+- Alternative: FAT image labeled `SEED_DATA` as external boot media.
+- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
+  `operations-center.yaml`, `network.yaml`, `update.yaml`.
+
+### Client certificates
+
+- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
+- Running `incus remote list` triggers auto-generation if no keypair exists.
+- For Incus seed: injected under `preseed.server.certificates[]`.
+- For Operations Center seed: injected under `trusted_client_certificates[]`.
+- Operations Center **requires** at least one trusted certificate -- without it,
+  you are locked out after installation.
+
+## Coding conventions for scripts
+
+- **Shell**: bash with `set -euo pipefail`
+- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
+  false exits under `set -e` when the value is 0.
+- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
+- **Flags**: support both short (`-d`) and long (`--defaults`) options.
+- **Defaults**: sane defaults so the script does something useful with zero flags.
+- **Dry run**: all scripts should support `--dry-run` to preview actions.
+- **Cert detection order**: files on disk first, CLI command second.
+- **Error messages**: include actionable remediation steps, not just "failed".
+- **No hardcoded package managers**: say "install the Incus client" with a link,
+  not "sudo apt install incus".
+
+## Git workflow
+
+- Main branch: `main`
+- Development happens on feature branches
+- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`

README.md

@@ -23,12 +23,18 @@ See [`incusos/README.md`](incusos/README.md) for full documentation.
 
 ### [`notes/`](notes/)
 
-Research notes and reference material.
+Research notes and reference material:
+
+- [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) --
+  Incus package versions across platforms (macOS, Debian/Ubuntu, Arch),
+  install instructions, and certificate management notes.
+- [`iso-download-methods.md`](notes/iso-download-methods.md) --
+  Original research on ISO download/customization approaches.
 
 ## Quick Start
 
 ```bash
-# Install the flasher-tool (requires Go 1.21+)
+# Install the flasher-tool (requires Go 1.22+)
 go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
 
 # Build a default IncusOS ISO with your client cert injected
@@ -49,11 +55,14 @@ go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
 
 ## Requirements
 
-- **Go 1.21+** -- for installing the flasher-tool
+- **Go 1.22+** -- for installing the flasher-tool
 - **curl** -- for downloading images from the CDN
 - **tar**, **gzip** -- for seed archive creation and image decompression
+- **Incus client** (recommended) -- for automatic client certificate detection.
+  Works with any version; certificates are read directly from disk.
+  See [version compatibility notes](notes/incus-version-compatibility.md)
+  for install instructions per platform (macOS, Debian/Ubuntu, Arch).
 - **mtools** (optional) -- for creating FAT seed images
-- **incus** (optional) -- for automatic client certificate detection
 
 ## License
 

incusos/README.md

@@ -14,19 +14,37 @@ installation, so the freshly installed system is immediately manageable.
 
 ## Prerequisites
 
-- **Go 1.21+** -- for installing the flasher-tool
+**Required:**
-- **flasher-tool** -- the official IncusOS image builder
 
-```bash
+- **flasher-tool** -- the official IncusOS image builder (used by `incusos-iso`)
-go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
-export PATH="${GOPATH:-$HOME/go}/bin:$PATH"
-```
 
-Optional:
+  ```bash
-- **incus** -- for automatic client certificate detection
+  # Requires Go 1.22+
-- **mtools** / **dosfstools** -- for creating FAT seed images
+  go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
+  export PATH="${GOPATH:-$HOME/go}/bin:$PATH"
+  ```
+
+  The `incusos-iso` script will offer to install it automatically if Go is available.
+
+**Recommended:**
+
+- **Incus client** -- for automatic client certificate detection.
+  Any version works (certificates are read from disk), but 6.3+ has the
+  most complete CLI. See [`notes/incus-version-compatibility.md`](../notes/incus-version-compatibility.md)
+  for install instructions per platform.
+
+  | Platform | Install command | Notes |
+  |----------|----------------|-------|
+  | macOS | `brew install incus` | Client only (no daemon on macOS) |
+  | Debian/Ubuntu | `apt install incus-client` | Use [Zabbly repo](https://github.com/zabbly/incus) for latest |
+  | Arch Linux | `pacman -S incus` | Full daemon+client (no client-only split) |
+  | Any (source) | `go install github.com/lxc/incus/v6/cmd/incus@latest` | Requires Go 1.22+ |
+
+**Optional:**
+
+- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`)
 - **jq** or **python3** -- for parsing the CDN index (cross-architecture builds)
-- **curl** -- for downloading images
+- **curl** -- for downloading images from the CDN
 
 ## incusos-iso
 
@@ -217,13 +235,34 @@ done
 
 Both scripts automatically look for a client certificate in this order:
 
-1. `incus remote get-client-certificate` (if the `incus` CLI is installed)
+1. `~/.config/incus/client.crt` (on-disk, works with all Incus versions)
-2. `~/.config/incus/client.crt`
+2. `~/snap/incus/common/config/client.crt` (snap installations)
-3. `~/snap/incus/common/config/client.crt`
+3. `~/.config/lxc/client.crt` (legacy LXC path)
-4. `~/.config/lxc/client.crt`
+4. `incus remote get-client-certificate` (CLI fallback, requires Incus >= 6.3)
+
+The file-based detection is preferred because `incus remote get-client-certificate`
+was added in Incus 6.3 and is not available in the 6.0 LTS shipped by Debian stable.
+
+If no certificate exists yet, run `incus remote list` to trigger auto-generation
+of a client keypair at `~/.config/incus/client.{crt,key}`.
 
 Override with `--cert FILE` or disable with `--no-cert`.
 
+### Incus Version Compatibility
+
+| Feature | 6.0 LTS (Debian stable) | 6.3+ (Zabbly, Homebrew, Arch) |
+|---------|-------------------------|-------------------------------|
+| Certificate on disk | `~/.config/incus/client.crt` | `~/.config/incus/client.crt` |
+| `get-client-certificate` CLI | not available | available |
+| `incus admin os` commands | not available | available |
+| Script cert auto-detection | works (reads file directly) | works (reads file directly) |
+
+These scripts work with **any** Incus version because they read the certificate
+file from disk rather than relying on newer CLI commands.
+
+For full installation instructions per platform, see
+[`notes/incus-version-compatibility.md`](../notes/incus-version-compatibility.md).
+
 ### After Installation
 
 With an injected certificate, the remote is already trusted:

incusos/incusos-iso

@@ -470,21 +470,11 @@ detect_client_cert() {
         return
     fi
 
-    # Auto-detection
+    # Auto-detection: prefer reading cert files directly (works with all
+    # Incus versions including 6.0 LTS which lacks 'get-client-certificate').
     step "Auto-detecting client certificate"
 
-    # Try incus CLI
+    # 1) Check common on-disk certificate locations first
-    if command -v incus &>/dev/null; then
-        local cert
-        if cert=$(incus remote get-client-certificate 2>/dev/null); then
-            CERT_CONTENT="$cert"
-            success "Certificate obtained from Incus CLI"
-            return
-        fi
-        detail "Incus CLI available but could not retrieve certificate"
-    fi
-
-    # Try common file locations
     local cert_paths=(
         "${HOME}/.config/incus/client.crt"
         "${HOME}/snap/incus/common/config/client.crt"
@@ -494,29 +484,48 @@ detect_client_cert() {
     for path in "${cert_paths[@]}"; do
         if [[ -f "$path" ]]; then
             CERT_CONTENT=$(<"$path")
-            success "Certificate loaded from ${path}"
+            if [[ "$CERT_CONTENT" == *"BEGIN CERTIFICATE"* ]]; then
-            return
+                success "Certificate loaded from ${path}"
+                return
+            fi
+            detail "File exists but does not contain a valid PEM certificate: ${path}"
         fi
     done
 
-    # Determine how serious the missing cert is
+    # 2) Try the incus CLI (only works with Incus >= 6.3)
+    if command -v incus &>/dev/null; then
+        local cert
+        if cert=$(incus remote get-client-certificate 2>/dev/null); then
+            CERT_CONTENT="$cert"
+            success "Certificate obtained via Incus CLI"
+            return
+        fi
+        # CLI is installed but cert retrieval failed -- check why
+        local incus_version
+        incus_version=$(incus version 2>/dev/null | grep -oP 'Client version: \K[0-9.]+' || echo "unknown")
+        detail "Incus CLI v${incus_version} found but no certificate available"
+        detail "Generate one with: incus remote list  (triggers auto-generation)"
+    fi
+
+    # 3) Nothing found -- report based on context
     if [[ "$APP" == "operations-center" ]]; then
         error "No client certificate found"
         error "Operations Center requires at least one trusted certificate for access"
-        error ""
+        echo "" >&2
         error "Options:"
-        error "  --cert FILE    Provide a certificate file"
+        error "  --cert FILE     Provide a certificate file directly"
-        error "  Install the Incus client: sudo apt install incus-client"
+        error "  Generate one:   incus remote list  (creates ~/.config/incus/client.crt)"
-        error "  --no-cert      Skip certificate (you will be locked out!)"
+        error "  --no-cert       Skip certificate injection (you will be locked out!)"
         exit 1
     fi
 
     warn "No client certificate found"
-    if ! command -v incus &>/dev/null; then
+    if command -v incus &>/dev/null; then
+        warn "Run 'incus remote list' to generate a client certificate, then re-run this script"
+    else
         warn "The Incus client is not installed on this machine"
-        warn "Install it to enable auto-detection: sudo apt install incus-client"
+        warn "Install it to enable auto-detection (see: https://github.com/lxc/incus)"
     fi
-    warn "The installed system will require manual certificate trust setup"
     warn "Use --cert FILE to provide a certificate, or --no-cert to suppress this warning"
 }
 

incusos/incusos-seed

@@ -324,21 +324,11 @@ detect_client_cert() {
         return
     fi
 
-    # Auto-detection
+    # Auto-detection: prefer reading cert files directly (works with all
+    # Incus versions including 6.0 LTS which lacks 'get-client-certificate').
     step "Auto-detecting client certificate"
 
-    # Try incus CLI first
+    # 1) Check common on-disk certificate locations first
-    if command -v incus &>/dev/null; then
-        local cert
-        if cert=$(incus remote get-client-certificate 2>/dev/null); then
-            CERT_CONTENT="$cert"
-            success "Certificate obtained from Incus CLI"
-            return
-        fi
-        detail "Incus CLI available but could not retrieve certificate"
-    fi
-
-    # Try common file locations
     local cert_paths=(
         "${HOME}/.config/incus/client.crt"
         "${HOME}/snap/incus/common/config/client.crt"
@@ -348,29 +338,47 @@ detect_client_cert() {
     for path in "${cert_paths[@]}"; do
         if [[ -f "$path" ]]; then
             CERT_CONTENT=$(<"$path")
-            success "Certificate loaded from ${path}"
+            if [[ "$CERT_CONTENT" == *"BEGIN CERTIFICATE"* ]]; then
-            return
+                success "Certificate loaded from ${path}"
+                return
+            fi
+            detail "File exists but does not contain a valid PEM certificate: ${path}"
         fi
     done
 
-    # Determine how serious the missing cert is
+    # 2) Try the incus CLI (only works with Incus >= 6.3)
+    if command -v incus &>/dev/null; then
+        local cert
+        if cert=$(incus remote get-client-certificate 2>/dev/null); then
+            CERT_CONTENT="$cert"
+            success "Certificate obtained via Incus CLI"
+            return
+        fi
+        local incus_version
+        incus_version=$(incus version 2>/dev/null | grep -oP 'Client version: \K[0-9.]+' || echo "unknown")
+        detail "Incus CLI v${incus_version} found but no certificate available"
+        detail "Generate one with: incus remote list  (triggers auto-generation)"
+    fi
+
+    # 3) Nothing found -- report based on context
     if [[ "$APP" == "operations-center" ]]; then
         error "No client certificate found"
         error "Operations Center requires at least one trusted certificate for access"
-        error ""
+        echo "" >&2
         error "Options:"
-        error "  --cert FILE    Provide a certificate file"
+        error "  --cert FILE     Provide a certificate file directly"
-        error "  Install the Incus client: sudo apt install incus-client"
+        error "  Generate one:   incus remote list  (creates ~/.config/incus/client.crt)"
-        error "  --no-cert      Skip certificate (you will be locked out!)"
+        error "  --no-cert       Skip certificate injection (you will be locked out!)"
         exit 1
     fi
 
     warn "No client certificate found"
-    if ! command -v incus &>/dev/null; then
+    if command -v incus &>/dev/null; then
+        warn "Run 'incus remote list' to generate a client certificate, then re-run this script"
+    else
         warn "The Incus client is not installed on this machine"
-        warn "Install it to enable auto-detection: sudo apt install incus-client"
+        warn "Install it to enable auto-detection (see: https://github.com/lxc/incus)"
     fi
-    warn "The installed system will require manual certificate trust setup"
     warn "Use --cert FILE to provide a certificate, or --no-cert to suppress this warning"
 }
 

notes/incus-version-compatibility.md

@@ -0,0 +1,164 @@
+# Incus Version Compatibility Notes
+
+## Package landscape
+
+| Platform | Package | Current version | Type | Notes |
+|----------|---------|-----------------|------|-------|
+| Debian stable (apt) | `incus` / `incus-client` | 6.0 LTS | Frozen | Security fixes only until June 2029 |
+| Debian/Ubuntu (Zabbly) | `incus` / `incus-client` | 6.21 | Rolling | Official upstream repo; separate client-only package |
+| macOS (Homebrew) | `incus` | 6.21 | Rolling | Client only by design (no daemon on macOS) |
+| Arch Linux (extra) | `incus` | 6.21 | Rolling | Full daemon + client; no client-only split |
+| From source (Go) | -- | latest | Manual | Build just the client binary |
+
+Incus releases monthly. LTS (6.0.x) gets bug + security fixes for 2 years,
+then security-only for 3 more years after Incus 7.0 LTS ships.
+
+## CLI differences between 6.0 LTS and 6.3+
+
+### Commands added after 6.0
+
+| Command | Added in | Purpose |
+|---------|----------|---------|
+| `incus remote get-client-certificate` | 6.3 | Export client cert via CLI |
+| `incus admin os system ...` | 6.x | IncusOS system management |
+
+### Client certificate locations (all versions)
+
+The client certificate and key are always stored on disk regardless of version:
+
+```
+~/.config/incus/client.crt    # PEM certificate
+~/.config/incus/client.key    # Private key
+```
+
+On snap installations:
+```
+~/snap/incus/common/config/client.crt
+~/snap/incus/common/config/client.key
+```
+
+Legacy LXC path:
+```
+~/.config/lxc/client.crt
+~/.config/lxc/client.key
+```
+
+Running any remote-related command (e.g. `incus remote list`) triggers
+auto-generation of the keypair if it doesn't exist yet.
+
+## Installing the Incus client
+
+These instructions cover installing the **client only** (the `incus` CLI
+for managing remote servers). If you also need the daemon, see the
+platform-specific notes below.
+
+### macOS (Homebrew)
+
+Homebrew provides the client only (macOS cannot run the daemon).
+
+```bash
+brew install incus
+```
+
+Verify:
+
+```bash
+incus version
+```
+
+Certificate location: `~/.config/incus/client.crt`
+
+### Debian / Ubuntu (Zabbly upstream repo)
+
+Distribution packages ship 6.0 LTS which is missing newer CLI commands.
+Use the Zabbly repo for the latest version.
+
+**Remove distro package first** (if installed):
+
+```bash
+sudo apt remove --purge incus incus-client
+sudo apt autoremove
+```
+
+**Add the Zabbly repository:**
+
+```bash
+sudo mkdir -p /etc/apt/keyrings
+
+curl -fsSL https://pkgs.zabbly.com/key.asc | \
+    sudo gpg --dearmor -o /etc/apt/keyrings/zabbly.gpg
+
+echo "deb [signed-by=/etc/apt/keyrings/zabbly.gpg] \
+    https://pkgs.zabbly.com/incus/stable \
+    $(lsb_release -cs) main" | \
+    sudo tee /etc/apt/sources.list.d/incus-stable.list
+```
+
+**Install client only:**
+
+```bash
+sudo apt update
+sudo apt install incus-client
+```
+
+Or install the full daemon + client with `sudo apt install incus`.
+
+Supported distributions: Debian 11+, Ubuntu 22.04+.
+
+### Arch Linux
+
+Arch ships the latest Incus in the `extra` repository, but only as a
+combined daemon + client package. There is no client-only split.
+
+**Full package (daemon + client):**
+
+```bash
+sudo pacman -S incus
+```
+
+**Client only from source** (if you don't want the daemon):
+
+```bash
+go install github.com/lxc/incus/v6/cmd/incus@latest
+```
+
+This puts the binary in `$GOPATH/bin` (typically `~/go/bin`).
+
+### From source (any platform with Go)
+
+Build just the client binary directly:
+
+```bash
+go install github.com/lxc/incus/v6/cmd/incus@latest
+export PATH="${GOPATH:-$HOME/go}/bin:$PATH"
+```
+
+Requires Go 1.22+.
+
+### Verify installation
+
+```bash
+incus version
+# Client version: 6.21
+```
+
+Generate a client certificate (if one doesn't exist yet):
+
+```bash
+incus remote list
+ls ~/.config/incus/client.crt
+```
+
+## Multi-machine certificate identity
+
+When managing IncusOS nodes from multiple machines, use the **same**
+client certificate on all management machines. Copy the keypair:
+
+```bash
+# From source machine to target machine
+scp ~/.config/incus/client.crt ~/.config/incus/client.key user@target:~/.config/incus/
+```
+
+This ensures all machines are recognized as the same trusted client.
+Using different certificates means each must be individually trusted
+on every managed node.