incus-contrib/incusos/README.md

# IncusOS Tools

Scripts for building IncusOS installation media and seed configurations.

## Overview

| Script | Purpose |
|--------|---------|
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
| [`lab-test`](#lab-test) | Guided lab validation (workloads, clustering, migration) |

All scripts support automatic client certificate injection from the local Incus
installation, so the freshly installed system is immediately manageable.

## Prerequisites

**Required:**

- **flasher-tool** -- the official IncusOS image builder (used by `incusos-iso`)

  ```bash
  # Requires Go 1.22+
  go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
  export PATH="${GOPATH:-$HOME/go}/bin:$PATH"
  ```

  The `incusos-iso` script will offer to install it automatically if Go is available.

**Recommended:**

- **Incus client** -- for automatic client certificate detection.
  Any version works (certificates are read from disk), but 6.3+ has the
  most complete CLI. See [`notes/incus-version-compatibility.md`](../notes/incus-version-compatibility.md)
  for install instructions per platform.

  | Platform | Install command | Notes |
  |----------|----------------|-------|
  | macOS | `brew install incus` | Client only (no daemon on macOS) |
  | Debian/Ubuntu | `apt install incus-client` | Use [Zabbly repo](https://github.com/zabbly/incus) for latest |
  | Arch Linux | `pacman -S incus` | Full daemon+client (no client-only split) |
  | Any (source) | `go install github.com/lxc/incus/v6/cmd/incus@latest` | Requires Go 1.22+ |

**Optional:**

- **genisoimage** -- for creating ISO 9660 seed images (`incusos-seed --format iso`).
  Required by `incusos-proxmox` (Proxmox CD-ROM devices need ISO 9660, not FAT).
- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`).
  Only for USB/block device use; FAT labels are NOT detected on CD-ROM devices.
- **jq** or **python3** -- for parsing the CDN index (cross-architecture builds)
- **curl** -- for downloading images from the CDN

## incusos-iso

Wrapper around the official flasher-tool that adds:

- **Cross-architecture builds** -- build aarch64 images on x86_64 and vice versa
- **Automatic seed generation** -- no need to manually create seed files
- **Client certificate injection** -- auto-detects your local Incus certificate
- **Batch mode** -- generate all common ISO variants in one command
- **Sensible defaults** -- minimal flags needed for common use cases

### Quick Start

```bash
# Build a default IncusOS ISO for the current architecture
./incusos-iso

# Incus with default configuration (ZFS pool, bridge, port 8443)
./incusos-iso --defaults

# Operations Center for aarch64
./incusos-iso --arch aarch64 --app operations-center --defaults

# Build a USB image instead of ISO
./incusos-iso --format img --defaults

# Generate all variants (2 arches x 2 apps x 2 default modes = 8 images)
./incusos-iso --batch

# Preview what would be built
./incusos-iso --batch --dry-run
```

### Options Reference

```
-a, --arch ARCH          x86_64 or aarch64 (default: auto-detect)
-A, --app APP            incus or operations-center (default: incus)
-d, --defaults           Apply default config (ZFS pool, bridge, port 8443)
-c, --cert FILE          Client certificate PEM file (default: auto-detect)
    --no-cert            Skip certificate injection
-F, --format FORMAT      iso or img (default: iso)
-o, --output FILE        Output filename (default: auto-generated)
-C, --channel CHANNEL    stable or testing (default: stable)
-s, --seed FILE          Use pre-built seed tar (skip seed generation)
-i, --image FILE         Use local base image (skip CDN download)
-b, --batch              Generate all common variants
    --batch-arches LIST  Architectures for batch (default: x86_64,aarch64)
    --batch-apps LIST    Apps for batch (default: incus,operations-center)
    --output-dir DIR     Output directory for batch (default: .)
    --disk-target ID     Target disk pattern (e.g. "virtio-*")
    --force-install      Overwrite existing installation
    --force-reboot       Auto-reboot after installation
    --missing-tpm        Allow install without TPM 2.0
    --missing-secureboot Allow install without Secure Boot
    --hostname NAME      Set system hostname
    --dry-run            Show what would happen
-q, --quiet              Suppress info output
-h, --help               Full help text
-V, --version            Show version
```

### Output Naming Convention

Auto-generated filenames follow the pattern:

```
incusos-{app}-{defaults|bare}-{arch}.{iso|img}
```

Examples:
- `incusos-incus-defaults-x86_64.iso`
- `incusos-ops-center-bare-aarch64.iso`
- `incusos-incus-defaults-x86_64.img`

### Batch Mode Matrix

With `--batch`, the following variants are generated:

| # | Architecture | Application | Defaults | Filename |
|---|-------------|-------------|----------|----------|
| 1 | x86_64 | Incus | yes | `incusos-incus-defaults-x86_64.iso` |
| 2 | x86_64 | Incus | no | `incusos-incus-bare-x86_64.iso` |
| 3 | x86_64 | Ops Center | yes | `incusos-ops-center-defaults-x86_64.iso` |
| 4 | x86_64 | Ops Center | no | `incusos-ops-center-bare-x86_64.iso` |
| 5 | aarch64 | Incus | yes | `incusos-incus-defaults-aarch64.iso` |
| 6 | aarch64 | Incus | no | `incusos-incus-bare-aarch64.iso` |
| 7 | aarch64 | Ops Center | yes | `incusos-ops-center-defaults-aarch64.iso` |
| 8 | aarch64 | Ops Center | no | `incusos-ops-center-bare-aarch64.iso` |

Customize with `--batch-arches` and `--batch-apps`:

```bash
# Only x86_64 Incus variants
./incusos-iso --batch --batch-arches x86_64 --batch-apps incus
```

## incusos-seed

Generates IncusOS seed archives containing the YAML configuration files
that customize an IncusOS installation. Output can be:

- **tar archive** (default) -- for use with `flasher-tool --seed`
- **ISO 9660 image** -- for use as a CD-ROM SEED_DATA device (e.g. Proxmox `ide3`)
- **FAT image** -- for use as a USB/block SEED_DATA device (NOT for CD-ROM)

### Quick Start

```bash
# Standalone Incus with defaults and auto-detected certificate
./incusos-seed --defaults

# Operations Center seed
./incusos-seed --app operations-center --defaults --cert ~/.config/incus/client.crt

# Cluster node (no defaults) with forced install and reboot
./incusos-seed --force-install --force-reboot

# FAT image for external boot media
./incusos-seed --format fat --defaults --output seed.img

# Preview generated YAML without creating files
./incusos-seed --defaults --dry-run
```

### Options Reference

```
-A, --app APP            incus or operations-center (default: incus)
-d, --defaults           Apply default configuration
-c, --cert FILE          Client certificate PEM file (default: auto-detect)
    --no-cert            Skip certificate injection
-o, --output FILE        Output file (default: incusos-seed.tar)
-f, --format FORMAT      tar, fat, or iso (default: tar)
    --fat-size SIZE      FAT image size in MiB (default: 10)
    --disk-target ID     Target disk pattern
    --force-install      Overwrite existing installation
    --force-reboot       Auto-reboot after installation
    --missing-tpm        Allow install without TPM 2.0
    --missing-secureboot Allow install without Secure Boot
    --hostname NAME      Set hostname
    --domain NAME        Set domain
    --dns SERVERS        Comma-separated DNS servers
    --channel CHANNEL    stable or testing (default: stable)
    --auto-reboot        Auto-reboot for updates
    --dry-run            Preview YAML output
-q, --quiet              Suppress info output
-h, --help               Full help text
-V, --version            Show version
```

### Seed Files Reference

The following YAML files are generated inside the archive:

| File | Always | Description |
|------|--------|-------------|
| `install.yaml` | yes | Installation target, security, reboot settings |
| `applications.yaml` | yes | Application selection |
| `incus.yaml` | when app=incus | Incus preseed (defaults, certificates) |
| `operations-center.yaml` | when app=ops-center | Operations Center config |
| `network.yaml` | when network opts set | Hostname, DNS, static IP |
| `update.yaml` | when non-default | Update channel, auto-reboot |

### Using ISO 9660 Images (SEED_DATA for CD-ROM)

The ISO output format creates a small ISO 9660 image with volume label `SEED_DATA`.
This is the **required** format when attaching seed data as a CD-ROM device (e.g.
Proxmox `ide3`). FAT filesystem labels are NOT detected by the Linux kernel on
CD-ROM (`/dev/sr*`) devices — only ISO 9660 volume labels work.

```bash
# Generate ISO seed for Proxmox deployment
./incusos-seed --format iso --defaults --output seed-node-1.iso
```

`incusos-proxmox` uses this format automatically.

### Using FAT Images (SEED_DATA for USB/Block Devices)

The FAT output format creates a small disk image labeled `SEED_DATA` that can be
attached to a VM as a secondary block device or written to USB. IncusOS detects
this label at boot and reads the seed configuration from it.

**Note:** FAT images do NOT work when attached as CD-ROM devices. Use ISO 9660
format instead for CD-ROM use cases.

This is useful for the "pristine image" workflow on bare metal:

1. Download one uncustomized IncusOS ISO
2. Generate per-node seed images with `incusos-seed --format fat`
3. Boot each machine with both the ISO and the node-specific seed USB

```bash
# Generate seeds for three cluster nodes (USB/block device use)
for i in 1 2 3; do
    ./incusos-seed --format fat \
        --hostname "node-${i}" \
        --force-install \
        --output "seed-node-${i}.img"
done
```

## incusos-proxmox

Declarative deployment of IncusOS VMs on Proxmox VE. Define your lab in a YAML
file and deploy with a single command. Handles seed generation, ISO upload, VM
creation with IncusOS-correct settings, and automated installation.

IncusOS requires very specific Proxmox VM settings (UEFI without pre-enrolled
keys, TPM 2.0, VirtIO-SCSI, `host` CPU type, minimum 50 GiB disk, 4 GiB RAM).
This script ensures all settings are correct, preventing install failures.

### Quick Start

```bash
# Preview what would be deployed
./incusos-proxmox --dry-run examples/proxmox-lab.yaml

# Validate config and check Proxmox connectivity
./incusos-proxmox --phase validate examples/proxmox-lab.yaml

# Deploy everything
./incusos-proxmox examples/proxmox-lab.yaml

# Check deployment status
./incusos-proxmox --status examples/proxmox-lab.yaml

# Deploy a single VM from the config
./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml

# Tear down all VMs defined in config
./incusos-proxmox --cleanup examples/proxmox-lab.yaml
```

### Options Reference

```
-p, --phase PHASE       validate, prepare, upload, create, install, status, all (default)
-m, --method METHOD     Override connection method: ssh (default), api
-H, --host HOST         Override Proxmox host from config
-i, --iso FILE          Use local ISO (skip CDN download)
-c, --cert FILE         Client certificate PEM file (default: auto-detect)
    --no-cert           Skip certificate injection
    --vm NAME           Operate on a single VM only
-y, --yes               Skip confirmation prompts
    --doctor            Check tool versions and environment health
    --status            Check deployment status (shortcut for --phase status)
    --cleanup           Destroy VMs defined in config
    --cleanup --deep    Also remove ISOs, seeds, remotes, and local cache
    --cleanup-all       Destroy ALL managed VMs in the resource pool
    --force-cleanup     Destroy VMs even without management marker
    --dry-run           Preview without executing
-q, --quiet             Suppress info output
-h, --help              Full help text
-V, --version           Show version
```

### Doctor Check

Run `--doctor` to verify your environment without a config file:

```bash
./incusos-proxmox --doctor
```

This checks: incus client, flasher-tool, operations-center CLI (optional),
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
availability.

If a config file is also provided, additionally checks Proxmox connectivity,
pool existence, and API token validity:

```bash
./incusos-proxmox --doctor examples/lab-cluster.yaml
```

### Deep Cleanup

Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:

```bash
# VMs + ISOs + seeds + incus remotes + local cache
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml

# Pool-wide: destroy all managed VMs + artifacts
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
```

`--deep` additionally:
- Deletes seed ISOs from Proxmox storage
- Deletes IncusOS ISOs from Proxmox storage
- Removes matching `incus remote` entries
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)

### Deployment Phases

| Phase | What it does |
|-------|-------------|
| `validate` | Parse YAML, check fields, verify Proxmox connectivity, allocate VMIDs |
| `prepare` | Download IncusOS ISO from CDN, generate per-VM SEED_DATA images |
| `upload` | Upload ISO and seed images to Proxmox ISO storage |
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |

### Config File Format

```yaml
proxmox:
  host: 192.168.1.10       # Proxmox host (required)
  node: pve                 # Node name (default: pve)
  storage: local-lvm        # VM disk storage (default: local-lvm)
  iso_storage: local        # ISO storage (default: local)
  bridge: vmbr0             # Network bridge (default: vmbr0)
  pool: IncusLab            # Resource pool (optional, for isolation)
  method: ssh               # ssh (default) or api
  ssh_user: root            # For ssh method

defaults:
  cores: 4
  memory: 8192              # MiB, min 4096
  disk: 64                  # GiB, min 50
  channel: stable
  start_vmid: 200           # Auto-assign VMIDs from here

vms:
  - name: my-vm
    app: incus              # incus or operations-center
    apply_defaults: true
    cores: 4                # Override defaults per VM
    memory: 8192
    disk: 64
    vmid: 200               # Optional explicit VMID
    cluster: my-cluster     # Future: cluster formation
    cluster_role: init      # init or join
```

### Connection Methods

**SSH (default):** Requires key-based SSH authentication to the Proxmox host.
Simplest for homelab use.

```bash
# Test connectivity
ssh root@192.168.1.10 pvesh get /version
```

**API:** Requires an API token. Set up minimal privileges:

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0

export PROXMOX_TOKEN_SECRET="<secret-from-above>"
```

### Resource Pool Isolation

For tighter access control, create a Proxmox resource pool and scope the API
token to it instead of `/`:

```bash
# On Proxmox (one-time setup)
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```

Then in your config file:

```yaml
proxmox:
  pool: IncusLab
```

When a pool is configured, the script only "sees" VMs in that pool — both for
collision detection and cleanup. The VM management marker includes the pool name
for traceability.

### Re-runs, Status, and Reconcile

**Status check:** Use `--status` to verify deployment health without modifying anything:

```bash
./incusos-proxmox --status examples/proxmox-lab.yaml
```

Shows per-VM: Proxmox state, install status, IP address, port 8443 reachability,
and incus remote connectivity.

**Re-run safety:** When re-running a full deployment (`--phase all`), the script
detects existing VMs and presents an interactive menu:

1. Run status checks and post-deployment verification
2. Continue deployment (skip existing, install incomplete VMs)
3. Destroy and redeploy cleanly
4. Abort

With `--yes`, defaults to option 1 (safe — never auto-destroys).

**Install idempotency:** The install phase checks each VM's state before acting:
already-running VMs are skipped, stopped-but-installed VMs are started from disk,
and only VMs with install media still attached proceed through installation.

### Proxmox VM Settings

Every VM is created with these IncusOS-specific settings:

| Setting | Value | Reason |
|---------|-------|--------|
| BIOS | OVMF (UEFI) | IncusOS requires UEFI boot |
| Machine | q35 | Modern chipset for UEFI |
| EFI disk | pre-enrolled-keys=0 | IncusOS enrolls its own Secure Boot keys |
| TPM | v2.0 | Required for disk encryption |
| CPU | host | Needs x86_64_v3 instruction set |
| SCSI controller | virtio-scsi-pci | VirtIO-blk is broken with IncusOS |
| Balloon | 0 (disabled) | IncusOS manages memory internally |
| SEED_DATA | ISO 9660 image on ide3 CD-ROM | Per-VM seed configuration |

### Prerequisites

- **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing
- **genisoimage** -- for creating ISO 9660 seed images
- **ssh** + **scp** (for SSH method) or **curl** (for API method)
- **incusos-seed** -- sibling script (auto-detected in same directory)
- **Bash 4+** -- associative arrays
- **Incus client** (recommended) -- for client certificate auto-detection

### Example Configurations

- [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster)
- [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM

## lab-test

Guided lab validation script for IncusOS deployments. Walks through workload
testing, cluster formation, and migration tests step by step. Interactive by
default — prints what it will do, confirms, executes, and reports pass/fail.

### Quick Start

```bash
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
./incusos/lab-test examples/lab-cluster.yaml

# Deploy and verify only
./incusos/lab-test --phase deploy examples/lab-cluster.yaml

# Single-node workload tests
./incusos/lab-test --phase single examples/lab-cluster.yaml

# Form cluster from deployed nodes
./incusos/lab-test --phase cluster examples/lab-cluster.yaml

# Test migration and evacuation
./incusos/lab-test --phase migrate examples/lab-cluster.yaml

# Full run without prompts
./incusos/lab-test --yes examples/lab-cluster.yaml

# Clean up test instances (keeps VMs)
./incusos/lab-test --cleanup examples/lab-cluster.yaml
```

### Phases

| Phase | What it does |
|-------|-------------|
| `deploy` | Deploy VMs (via incusos-proxmox), verify incus remotes, check connectivity |
| `single` | Launch a test container and VM on the first node, exec, cleanup |
| `cluster` | Enable clustering on node 1, join remaining nodes, verify membership |
| `workload` | Launch instances on the cluster (scheduler-placed + targeted), cleanup |
| `migrate` | Stop/move container between nodes, evacuate/restore a node |

### Options

```
-p, --phase PHASE     deploy, single, cluster, workload, migrate, all (default)
    --skip-deploy     Skip deploy/status check (assume VMs already running)
    --cleanup         Remove test instances without destroying VMs
-y, --yes             Skip confirmation prompts
    --dry-run         Preview actions
-q, --quiet           Suppress info output
-h, --help            Full help text
-V, --version         Show version
```

## Client Certificate Auto-Detection

Both scripts automatically look for a client certificate in this order:

1. `~/.config/incus/client.crt` (on-disk, works with all Incus versions)
2. `~/snap/incus/common/config/client.crt` (snap installations)
3. `~/.config/lxc/client.crt` (legacy LXC path)
4. `incus remote get-client-certificate` (CLI fallback, requires Incus >= 6.3)

The file-based detection is preferred because `incus remote get-client-certificate`
was added in Incus 6.3 and is not available in the 6.0 LTS shipped by Debian stable.

If no certificate exists yet, run `incus remote list` to trigger auto-generation
of a client keypair at `~/.config/incus/client.{crt,key}`.

Override with `--cert FILE` or disable with `--no-cert`.

### Incus Version Compatibility

| Feature | 6.0 LTS (Debian stable) | 6.3+ (Zabbly, Homebrew, Arch) |
|---------|-------------------------|-------------------------------|
| Certificate on disk | `~/.config/incus/client.crt` | `~/.config/incus/client.crt` |
| `get-client-certificate` CLI | not available | available |
| `incus admin os` commands | not available | available |
| Script cert auto-detection | works (reads file directly) | works (reads file directly) |

These scripts work with **any** Incus version because they read the certificate
file from disk rather than relying on newer CLI commands.

For full installation instructions per platform, see
[`notes/incus-version-compatibility.md`](../notes/incus-version-compatibility.md).

### After Installation

With an injected certificate, the remote is already trusted — no authentication
prompt appears:

```bash
incus remote add my-server <ip-address> --accept-certificate
incus remote list
```

For web UI access, export a PKCS#12 bundle and import it into your browser:

```bash
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
```

## Examples

See the [`examples/`](examples/) directory for sample seed configurations:

- [`incus-standalone.yaml`](examples/incus-standalone.yaml) -- Single Incus server with defaults
- [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
- [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes

## Security Notes

- **TPM 2.0** and **Secure Boot** are required by default. Use `--missing-tpm` and
  `--missing-secureboot` only for lab environments or VMs that lack these features.
- All storage pools are **automatically encrypted** by IncusOS.
- Retrieve encryption recovery keys after installation with:
  `incus admin os system security show`
- Keep recovery keys safe -- they are needed if TPM state changes.