500 lines
26 KiB
Markdown
500 lines
26 KiB
Markdown
# CLAUDE.md - Project context for AI assistants
|
|
|
|
## What this repository is
|
|
|
|
A collection of peripheral tools, scripts, and snippets for working with
|
|
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
|
|
Primarily targeting home lab environments but aiming for production-quality scripts.
|
|
|
|
## Repository structure
|
|
|
|
```
|
|
incu-contrib/
|
|
├── CLAUDE.md # This file -- project context
|
|
├── README.md # Main overview
|
|
├── .gitignore
|
|
├── incusos/ # IncusOS installation tooling
|
|
│ ├── README.md # Detailed usage docs
|
|
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
|
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
|
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
|
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
|
│ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
|
|
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
|
│ └── examples/ # Example seed + Proxmox YAML files
|
|
└── notes/ # Research notes and reference material
|
|
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
|
├── operations-center-guide.md # Operations Center provisioning & management
|
|
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
|
└── utm-support.md # UTM support design document (future)
|
|
```
|
|
|
|
## Key technical context
|
|
|
|
### Incus version differences
|
|
|
|
- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
|
|
The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
|
|
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
|
|
macOS is client-only by design; Arch has no client-only split package.
|
|
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
|
|
not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
|
|
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
|
|
Fall back to the CLI command only as a secondary option.
|
|
- See `notes/incus-version-compatibility.md` for full platform matrix and
|
|
install instructions.
|
|
|
|
### IncusOS flasher-tool
|
|
|
|
- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
|
|
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
|
|
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
|
|
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
|
|
For cross-arch builds, download the image manually and pass via `--image`.
|
|
- CDN index: `https://images.linuxcontainers.org/os/index.json`
|
|
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`
|
|
|
|
### Seed archives
|
|
|
|
- Tar archives containing YAML files at the root level.
|
|
- Written to byte offset 2148532224 (the seed partition) in the image.
|
|
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
|
|
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
|
|
devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
|
|
- **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
|
|
**NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
|
|
FAT filesystem labels, so IncusOS cannot find the seed.
|
|
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
|
|
`operations-center.yaml`, `network.yaml`, `update.yaml`.
|
|
|
|
### Client certificates
|
|
|
|
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
|
|
- Running `incus remote list` triggers auto-generation if no keypair exists.
|
|
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
|
|
The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
|
|
`certificates`) are promoted to the top level of the `preseed` object.
|
|
- For Operations Center seed: injected under `trusted_client_certificates[]`.
|
|
- Operations Center **requires** at least one trusted certificate -- without it,
|
|
you are locked out after installation.
|
|
|
|
### Proxmox VE deployment
|
|
|
|
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
|
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
|
VMs with IncusOS-correct settings, and boots them through installation.
|
|
- **Config separation**: Proxmox connection settings (host, credentials, pool)
|
|
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
|
|
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
|
|
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
|
|
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
|
|
- **API token secret**: stored in `env` file at the repo root (gitignored).
|
|
Source it before running scripts: `source env` (exports `PROXMOX_TOKEN_SECRET`).
|
|
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
|
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
|
- **Minimum API privileges** for token-based access (role: `IncusOSDeployer`):
|
|
```
|
|
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
|
|
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
|
|
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
|
|
Datastore.Audit SDN.Use Sys.Audit
|
|
```
|
|
`Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via
|
|
`/nodes/<node>/status`). It requires a separate ACL on `/nodes/<node>`
|
|
since the pool-scoped ACL doesn't cover node-level endpoints.
|
|
- **Required VM settings** (getting any wrong causes IncusOS install failure):
|
|
- `bios=ovmf`, `machine=q35` -- UEFI boot required
|
|
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
|
|
- `tpmstate0`: `version=v2.0` -- required for disk encryption
|
|
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
|
|
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
|
|
- `balloon=0` -- IncusOS manages memory internally
|
|
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
|
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
|
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
|
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
|
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
|
`disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
|
|
- **Install flow** (automated by `incusos-proxmox`):
|
|
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
|
|
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
|
|
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
|
|
when disk writes start then stop for 15s (3 stable polls), install is done
|
|
4. Stop the VM (Proxmox stop, not guest shutdown)
|
|
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
|
|
and refuses to start if found, regardless of boot order
|
|
6. Set boot order to `order=scsi0` and start from disk
|
|
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
|
|
ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
|
|
ping broadcast → look up MAC in ARP table. Verify with direct ping before
|
|
trusting the result.
|
|
- **force_reboot is required**: without `force_reboot: true` in the seed,
|
|
IncusOS sits at "please remove installation media" and waits indefinitely.
|
|
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
|
|
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
|
|
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
|
|
all VM operations to a Proxmox resource pool. When set, the script only
|
|
"sees" VMs in that pool (for collision detection and cleanup), and the API
|
|
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
|
|
```
|
|
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
|
pveum pool modify IncusLab --storage local-lvm,local
|
|
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
|
|
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
|
|
```
|
|
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
|
|
deployment status (Proxmox state, install status, IP, port 8443, incus
|
|
remote). Also runs post-deployment checks (Incus connectivity, Operations
|
|
Center URL).
|
|
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
|
|
config, an interactive menu offers: (1) run status checks, (2) continue
|
|
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
|
|
`--yes`, defaults to option 1 (safe -- never auto-destroys).
|
|
- **Install idempotency**: `phase_install` checks each VM's state before
|
|
acting -- already-running VMs are skipped, stopped-but-installed VMs are
|
|
started from disk, only VMs with install media proceed through installation.
|
|
|
|
### Multi-lab coexistence
|
|
|
|
- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop,
|
|
VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk
|
|
(refuses to start VMs with install media still attached). These are
|
|
distinct from `--cleanup` (which destroys VMs permanently).
|
|
- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage
|
|
usage, and per-pool allocation. Requires API method.
|
|
- **Lab inventory**: `--labs` scans the pool for managed VMs (by
|
|
`[incusos-lab:managed]` marker), groups by config file, and shows
|
|
per-lab status, VM count, RAM, and disk.
|
|
- **VMID range convention** (to avoid collisions between coexisting labs):
|
|
|
|
| Range | Lab |
|
|
|-------|-----|
|
|
| 400-499 | OC-managed nodes |
|
|
| 800-809 | Single-node labs |
|
|
| 900-909 | Basic cluster |
|
|
| 910-919 | OC combined (server + nodes) |
|
|
| 920-929 | OC server standalone |
|
|
| 930-939 | Advanced / heterogeneous |
|
|
|
|
### Incus clustering via remotes
|
|
|
|
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
|
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
|
- **No VIP needed**: each node advertises its own IP as its cluster address.
|
|
Clients can connect to any cluster member; requests are forwarded internally.
|
|
|
|
#### Pre-clustering: fix core.https_address
|
|
|
|
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
|
|
interfaces). Clustering requires a **specific routable IP** so nodes can
|
|
address each other.
|
|
- **Set the IP on every node BEFORE enabling clustering:**
|
|
```bash
|
|
incus config set <remote>: core.https_address <NODE_IP>:8443
|
|
```
|
|
- Get each node's routable IP via the API:
|
|
```bash
|
|
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
|
[print(a) for a in d['environment']['addresses'] \
|
|
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
|
```
|
|
- This is safe to do while remotes are connected -- the remote already points
|
|
to the specific IP; we're just narrowing the bind address. Certificate trust
|
|
is fingerprint-based, not address-based.
|
|
|
|
#### Cluster enable (init node)
|
|
|
|
- ```bash
|
|
incus cluster enable <remote>: <member-name>
|
|
```
|
|
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
|
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
|
|
- **TLS certificate regeneration**: enabling clustering causes the server to
|
|
generate a new TLS certificate (cluster cert). The new cert may only have
|
|
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
|
|
- **Fix**: remove and re-add the remote to pin the new certificate:
|
|
```bash
|
|
incus remote switch local # if init remote is current default
|
|
incus remote remove <remote>
|
|
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
|
```
|
|
- The cert trust on the server side (client → server) is unaffected -- it's
|
|
stored by fingerprint in the Incus database, independent of listen address.
|
|
|
|
#### Joining nodes: the apply_defaults storage pool conflict
|
|
|
|
- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
|
|
network bridge on each node at install time. When joining a cluster that
|
|
already has a `local` pool defined, the join wizard asks for `source` and
|
|
`zfs.pool_name` properties but then tries to set them as cluster-wide config.
|
|
Incus rejects this because they are **member-specific** keys.
|
|
- **The fix**: delete the storage pool and network on the joining node before
|
|
joining. The join process then creates the member-specific pool entry
|
|
correctly.
|
|
```bash
|
|
# 1. Remove config references
|
|
incus config unset <remote>: storage.backups_volume
|
|
incus config unset <remote>: storage.images_volume
|
|
# 2. Delete volumes
|
|
incus storage volume delete <remote>:local backups
|
|
incus storage volume delete <remote>:local images
|
|
# 3. Clear default profile references (pool is "in use" otherwise)
|
|
incus profile device remove <remote>:default root
|
|
incus profile device remove <remote>:default eth0
|
|
# 4. Delete pool and network
|
|
incus storage delete <remote>:local
|
|
incus network delete <remote>:incusbr0
|
|
```
|
|
|
|
#### Join workflow
|
|
|
|
- **Generate token** (on init node, single argument `remote:member-name`):
|
|
```bash
|
|
incus cluster add <init-remote>:<new-member-name>
|
|
```
|
|
- **Join** (interactive -- prompts for 5 values):
|
|
```bash
|
|
incus cluster join <init-remote>: <joining-remote>:
|
|
```
|
|
Interactive prompts and correct answers:
|
|
1. IP address → accept default (node's IP, already set via core.https_address)
|
|
2. Member name → accept default (matches the token)
|
|
3. "All existing data is lost" → `yes`
|
|
4. `source` property for storage pool "local" → `local/incus`
|
|
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
|
|
- **Automated (non-interactive):**
|
|
```bash
|
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
|
```
|
|
- **After join**: the joining node gets a new cluster certificate. Fix the
|
|
remote (same as init node):
|
|
```bash
|
|
incus remote remove <joining-remote>
|
|
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
|
|
```
|
|
|
|
#### Command syntax gotchas
|
|
|
|
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
|
|
- `incus cluster add remote:member-name` -- ONE argument (no space)
|
|
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
|
|
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
|
|
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
|
|
- `incus cluster restore remote:member-name` -- ONE argument (no space)
|
|
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
|
|
- `incus storage show remote:pool` -- ONE argument (no space)
|
|
- `incus storage show remote:pool --target member` -- target flag for
|
|
member-specific config
|
|
- `incus config set remote: key value` -- remote with trailing colon + space
|
|
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
|
|
colon) for targeting the server itself
|
|
|
|
#### Post-join state
|
|
|
|
- After joining, the cluster is managed through the init node's remote. The
|
|
individual node remotes still work for node-specific operations.
|
|
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
|
|
|
#### Workload placement and migration
|
|
|
|
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
|
|
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
|
|
- **Container migration**: stop/move/start only (CRIU live migration is
|
|
unreliable). Data persists, processes do not.
|
|
```bash
|
|
incus stop <remote>:<instance>
|
|
incus move <remote>:<instance> --target <destination-member>
|
|
incus start <remote>:<instance>
|
|
```
|
|
- **VM live migration**: requires `migration.stateful=true` (must be set
|
|
while VM is stopped). Preserves running state with no downtime.
|
|
```bash
|
|
incus move <remote>:<instance> --target <destination-member>
|
|
```
|
|
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
|
|
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
|
|
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
|
|
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
|
|
differently, causing `Missing section footer for ICH9LPC` on restore.
|
|
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
|
|
fixed `sockets/cores/threads` topology — portable across all hosts.
|
|
```bash
|
|
# WRONG: integer → maxcpus varies by host → migration fails
|
|
incus config set <instance> limits.cpu=2
|
|
# RIGHT: range → fixed topology → migration works everywhere
|
|
incus config set <instance> limits.cpu=0-1
|
|
```
|
|
- **VM live migration works in nested virtualization** (IncusOS inside
|
|
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
|
|
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
|
|
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
|
|
appears during migration is cosmetic. It fires from QEMU's feature
|
|
dependency checker before KVM filters out unsupported features and does
|
|
not affect migration.
|
|
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
|
|
requires the `limits.cpu` range fix. Use `incus start --stateless` to
|
|
discard a saved state file that cannot be restored.
|
|
- **VM `size.state` config**: stateful operations require `size.state` on
|
|
the root disk (`incus config device add <instance> root disk path=/
|
|
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
|
|
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
|
(ONE argument, like `cluster enable` and `cluster add`).
|
|
Use `--action stop` if VMs lack the `limits.cpu` range fix.
|
|
Restore with `incus cluster restore <remote>:<member> --force`.
|
|
- **VM agent reconnect**: after live migration, the incus agent inside the VM
|
|
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
|
|
after migration may fail with "VM agent isn't currently running". Scripts
|
|
should `sleep 4` after migration before running `incus exec`.
|
|
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
|
|
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
|
|
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
|
|
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
|
|
- **Concurrent migrations**: migrating multiple VMs simultaneously from
|
|
different source nodes works without interference. ~140 MB/s per migration.
|
|
- **Active I/O during migration**: disk writes and network activity survive
|
|
live migration transparently. File integrity verified after migration.
|
|
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
|
|
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
|
|
auto-rebalanced.
|
|
```bash
|
|
incus config set <remote>: cluster.rebalance.interval=1 # minutes
|
|
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
|
|
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
|
|
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
|
|
```
|
|
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
|
|
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
|
|
for step-by-step instructions.
|
|
- **`incus cluster remove` requires confirmation**: even with `--force`, it
|
|
prompts "Are you really sure?". Pipe `yes` for automation:
|
|
```bash
|
|
printf "yes\n" | incus cluster remove <remote>:<member> --force
|
|
```
|
|
- See `notes/clustering-guide.md` for full details and references.
|
|
|
|
### Lab validation (lab-test)
|
|
|
|
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
|
on the VM names defined there (expects `incus` remotes to exist).
|
|
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
|
|
migrate (stop/move + evacuate/restore).
|
|
- **Test instances** use names starting with `test-` for easy cleanup.
|
|
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
|
|
|
|
### Operations Center
|
|
|
|
- **CLI binary:** `operations-center` (installed from GitHub releases or
|
|
built from source at `github.com/FuturFusion/operations-center`).
|
|
- **Config directory:** `~/.config/operations-center/` (uses same cert format
|
|
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
|
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
|
|
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
|
|
imported into the browser.
|
|
- **OC is under active development** (v0.2.2). Commands and APIs may change.
|
|
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
|
installed.
|
|
|
|
#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)
|
|
|
|
- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
|
|
- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
|
|
must use structured format with section keys (`install:`, not flat).
|
|
```bash
|
|
operations-center provisioning token seed add <UUID> proxmox-preseed \
|
|
/tmp/preseed.yaml --description "Force reboot for Proxmox"
|
|
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
|
|
/tmp/IncusOS-oc.iso --type iso --architecture x86_64
|
|
```
|
|
- **Hybrid deployment** (tested, recommended):
|
|
`incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
|
|
combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
|
|
VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring,
|
|
and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
|
|
- **Self-registration**: nodes auto-register with OC within ~30s of first
|
|
boot. Hostname from SEED_DATA is used as the server name.
|
|
- **Cluster formation** (tested):
|
|
```bash
|
|
operations-center provisioning cluster add oc-cluster \
|
|
https://<NODE_01_IP>:8443 \
|
|
--server-names oc-node-01,oc-node-02,oc-node-03 \
|
|
--server-type incus \
|
|
--application-seed-config /tmp/oc-app-config.yaml
|
|
```
|
|
OC handles: `core.https_address` → cluster enable → joins → seed
|
|
application → Terraform config. Adds `meshbr0` network.
|
|
- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
|
|
nodes already have storage pool/network/cert. OC's Terraform fails with
|
|
"already exists" errors but the cluster forms successfully.
|
|
|
|
#### Tested limitations
|
|
|
|
- **Inventory is NOT real-time** -- requires explicit `cluster resync`
|
|
- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot likely corrupts
|
|
the dqlite/Incus database through unclean shutdown. OS boots (pingable)
|
|
but Incus daemon never starts. Open research item. Only fix: destroy
|
|
and redeploy. Use Proxmox stop/start instead.
|
|
- **No cluster member state tracking** -- OC always shows `ready` even for
|
|
EVACUATED/OFFLINE nodes
|
|
- **Stale entries** from out-of-band cluster changes persist after resync
|
|
- **Server removal blocked** if server is part of an OC cluster
|
|
|
|
- See `notes/operations-center-guide.md` for full tested OC reference.
|
|
|
|
### incusos-proxmox doctor and cleanup
|
|
|
|
- **`--doctor`**: standalone environment check. No config file required.
|
|
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
|
|
Proxmox connectivity (from proxmox.yaml or config file).
|
|
- **`--cleanup`**: destroys VMs defined in the config file.
|
|
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
|
|
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
|
|
all IncusOS ISOs (unlike the old behavior).
|
|
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
|
|
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
|
|
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
|
|
and `seed-*.iso` from storage + remotes + cache.
|
|
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
|
output is concise (step names + results). `--quiet` suppresses everything
|
|
except warnings and errors.
|
|
|
|
## Coding conventions for scripts
|
|
|
|
- **Shell**: bash with `set -euo pipefail`
|
|
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
|
|
false exits under `set -e` when the value is 0.
|
|
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
|
|
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
|
|
- **Defaults**: sane defaults so the script does something useful with zero flags.
|
|
- **Dry run**: all scripts should support `--dry-run` to preview actions.
|
|
- **Cert detection order**: files on disk first, CLI command second.
|
|
- **Error messages**: include actionable remediation steps, not just "failed".
|
|
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
|
not "sudo apt install incus".
|
|
|
|
### Migration into Incus
|
|
|
|
- **`incus-migrate`**: official tool for importing disk images, running
|
|
instances, or physical machines into Incus.
|
|
- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2,
|
|
raw, vdi, vhd formats. Incus accepts raw and qcow2.
|
|
- **Import workflow**: convert disk → `incus storage volume import` →
|
|
`incus init --empty --vm` → attach disk → start.
|
|
- **Container migration**: `docker export` → `incus import` for filesystem-
|
|
level container migration. Docker volumes must be copied separately.
|
|
- See `notes/migration-guide.md` for full procedures per source hypervisor.
|
|
|
|
### UTM support (future)
|
|
|
|
- Design document at `notes/utm-support.md`.
|
|
- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation
|
|
(requires AppleScript or `.utm` bundle generation).
|
|
- No `blockstat` equivalent -- install detection must use timeout + port polling.
|
|
- Seed generation already works cross-platform (Phase 3 macOS compatibility).
|
|
|
|
## Git workflow
|
|
|
|
- Main branch: `main`
|
|
- Development happens on feature branches
|
|
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`
|