Modularize CLAUDE.md: rules, helpers, skills (866→112 lines)
Split monolithic CLAUDE.md into three layers: - CLAUDE.md (112 lines): behavioral focus, capabilities, safety rules - .claude/rules/ (8 files, 503 lines): topic-specific context with paths: frontmatter for conditional loading - .claude/skills/ (2 files): /screenshot and /proxmox-api commands Add helper scripts (incusos/helpers/): - proxmox-screenshot: SSH→screendump→PPM→PNG pipeline with VMID safety - proxmox-api: authenticated API calls handling ! in token quoting Fix git workflow (master branch, both origin+aether remotes). Add capabilities section for screenshots, API calls, live AWX output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
0b82b9ce3b
commit
c4efcbe30d
|
|
@ -0,0 +1,56 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "ansible/**"
|
||||||
|
- "incusos/deploy-awx"
|
||||||
|
- "incusos/awx-manifests/*"
|
||||||
|
- "notes/awx-guide.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# AWX Integration (Ansible for Aether)
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
- AWX: open-source Ansible platform. Debian 12 VM with K3s + AWX Operator.
|
||||||
|
- Lab VM: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69).
|
||||||
|
- **AWX URL**: `http://192.168.102.161:30080` (K3s NodePort, not Traefik ingress).
|
||||||
|
- `deploy-awx` script: `--deploy`, `--status`, `--heal`, `--configure`,
|
||||||
|
`--join-aether`, `--cleanup`, `--doctor`.
|
||||||
|
|
||||||
|
## Live job output
|
||||||
|
|
||||||
|
Capture AWX job output while running (don't just poll status):
|
||||||
|
```bash
|
||||||
|
curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
|
||||||
|
-H "Authorization: Bearer $AWX_TOKEN"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Aether extra vars
|
||||||
|
|
||||||
|
Aether passes `ffsdn_` prefixed vars: `ffsdn_instance_name`, `ffsdn_instance_ip`,
|
||||||
|
`ffsdn_cluster_id`, `ffsdn_cluster_name`, `ffsdn_deployed_by`, `ffsdn_image_os`,
|
||||||
|
`ffsdn_image_release`, `ffsdn_image_alias`.
|
||||||
|
|
||||||
|
It does NOT pass: `vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`.
|
||||||
|
|
||||||
|
## Playbook pattern: Incus REST API, not SSH
|
||||||
|
|
||||||
|
- AWX cannot SSH to containers on incusbr0 (not routable, nftables blocks).
|
||||||
|
- Use `uri` module with client cert for file push + exec.
|
||||||
|
- Cert at `/runner/project/incus-client.crt` (EE mounts at `/runner/project/`,
|
||||||
|
NOT `/var/lib/awx/projects/`).
|
||||||
|
|
||||||
|
## Manual project (local_path)
|
||||||
|
|
||||||
|
- EE containers can't reach private git repo.
|
||||||
|
- Use `scm_type: ""`, `local_path: "incus-contrib"`.
|
||||||
|
- Push playbooks to AWX task pod at `/var/lib/awx/projects/incus-contrib/playbooks/`.
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- **Self-referencing vars**: `vm_ip: "{{ vm_ip | default('') }}"` causes
|
||||||
|
infinite recursion. Use `ffsdn_*` vars directly.
|
||||||
|
- **`environment` is reserved**: resolves to `[]` as extra var.
|
||||||
|
- **AWX config API bug**: `PUT /api/clusters/{id}/awx-config` returns
|
||||||
|
"Invalid cluster ID". Workaround: direct PostgreSQL UPDATE.
|
||||||
|
- **Lifecycle hooks**: post-deploy failure -> auto-rollback (instance deleted).
|
||||||
|
Decommission failure does NOT block deletion.
|
||||||
|
|
@ -0,0 +1,97 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/lab-test"
|
||||||
|
- "incusos/incusos-proxmox"
|
||||||
|
- "notes/clustering-guide.md"
|
||||||
|
- "notes/production-lab-guide.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Incus Clustering
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
- Cluster formation via `incus` CLI using remotes. No SSH needed (IncusOS is immutable).
|
||||||
|
- No VIP needed: each node advertises its own IP. Requests forwarded internally.
|
||||||
|
|
||||||
|
## Pre-clustering: fix core.https_address
|
||||||
|
|
||||||
|
- Default `:8443` (wildcard). Clustering needs specific routable IP.
|
||||||
|
- Set on every node BEFORE clustering:
|
||||||
|
```bash
|
||||||
|
incus config set <remote>: core.https_address <NODE_IP>:8443
|
||||||
|
```
|
||||||
|
- Get routable IP:
|
||||||
|
```bash
|
||||||
|
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||||
|
[print(a) for a in d['environment']['addresses'] \
|
||||||
|
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cluster enable (init node)
|
||||||
|
|
||||||
|
- ```bash
|
||||||
|
incus cluster enable <remote>: <member-name>
|
||||||
|
```
|
||||||
|
TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
||||||
|
- Generates new TLS cert. Fix remote:
|
||||||
|
```bash
|
||||||
|
incus remote switch local
|
||||||
|
incus remote remove <remote>
|
||||||
|
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
||||||
|
```
|
||||||
|
|
||||||
|
## apply_defaults for clusters
|
||||||
|
|
||||||
|
- **Bootstrap node**: `apply_defaults: true` (needs pool + network)
|
||||||
|
- **Joining nodes**: `apply_defaults: false` (recommended, clean join)
|
||||||
|
- **Standalone**: `apply_defaults: true`
|
||||||
|
- If `apply_defaults: true` on joiners, 8-command cleanup needed before join
|
||||||
|
(delete pool, network, volumes, profile devices).
|
||||||
|
|
||||||
|
## Join workflow
|
||||||
|
|
||||||
|
- **Token**: `incus cluster add <init-remote>:<new-member-name>` (ONE arg)
|
||||||
|
- **Join**: `incus cluster join <init-remote>: <joining-remote>:` (TWO args)
|
||||||
|
- **Automated**:
|
||||||
|
```bash
|
||||||
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
||||||
|
```
|
||||||
|
- After join: fix remote (same as init).
|
||||||
|
|
||||||
|
## Command syntax gotchas
|
||||||
|
|
||||||
|
- `cluster enable remote: member-name` -- TWO args
|
||||||
|
- `cluster add remote:member-name` -- ONE arg
|
||||||
|
- `cluster remove remote:member-name --force` -- ONE arg, prompts "yes/no"
|
||||||
|
- `cluster evacuate remote:member-name` -- ONE arg
|
||||||
|
- `cluster restore remote:member-name` -- ONE arg
|
||||||
|
- `cluster join init-remote: joining-remote:` -- TWO args
|
||||||
|
- `config set remote: key value` -- remote with trailing colon + space
|
||||||
|
- General: `remote:resource` for resource, `remote:` for server itself
|
||||||
|
|
||||||
|
## Workload placement and migration
|
||||||
|
|
||||||
|
- **Targeted launch**: `incus launch images:debian/12 <remote>:name --target <member>`
|
||||||
|
- **Container migration**: stop/move/start only (no CRIU).
|
||||||
|
- **VM live migration**: requires `migration.stateful=true` (set while stopped).
|
||||||
|
- **limits.cpu MUST be a range** (e.g., `0-1`), not integer. Integer causes
|
||||||
|
`maxcpus` mismatch -> `Missing section footer for ICH9LPC`.
|
||||||
|
- **size.state required**: `size.state=2GiB` on root disk for stateful ops.
|
||||||
|
- **Agent reconnect**: sleep 4s after migration before `incus exec`.
|
||||||
|
- **Evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
||||||
|
Use `--action stop` without limits.cpu range fix.
|
||||||
|
|
||||||
|
## Cluster rebalancing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus config set <remote>: cluster.rebalance.interval=1
|
||||||
|
incus config set <remote>: cluster.rebalance.threshold=10
|
||||||
|
incus config set <remote>: cluster.rebalance.batch=2
|
||||||
|
incus config set <remote>: cluster.rebalance.cooldown=5m
|
||||||
|
```
|
||||||
|
|
||||||
|
## Lab validation (lab-test)
|
||||||
|
|
||||||
|
- Reads same YAML config as `incusos-proxmox`.
|
||||||
|
- Phases: deploy, single, cluster, workload, migrate.
|
||||||
|
- Test instances: `test-*` prefix. Reports PASS/FAIL/SKIP.
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/incusos-iso"
|
||||||
|
- "incusos/incusos-seed"
|
||||||
|
- "incusos/incusos-proxmox"
|
||||||
|
- "incusos/helpers/*"
|
||||||
|
- "incusos/examples/*"
|
||||||
|
---
|
||||||
|
|
||||||
|
# IncusOS Scripts Context
|
||||||
|
|
||||||
|
## Incus version differences
|
||||||
|
|
||||||
|
- **Debian stable ships Incus 6.0 LTS** (behind upstream).
|
||||||
|
Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
|
||||||
|
- **macOS (Homebrew)** and **Arch Linux** track latest upstream (currently 6.21).
|
||||||
|
macOS is client-only; Arch has no client-only split package.
|
||||||
|
- `incus remote get-client-certificate` added in **6.3+** -- does not exist
|
||||||
|
in 6.0 LTS. Scripts must never depend on it as the only cert path.
|
||||||
|
- Always prefer reading `~/.config/incus/client.crt` from disk first.
|
||||||
|
- See `notes/incus-version-compatibility.md` for full matrix.
|
||||||
|
|
||||||
|
## IncusOS flasher-tool
|
||||||
|
|
||||||
|
- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
|
||||||
|
- Flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
|
||||||
|
- There is NO `--seed-tar` flag -- it's `--seed` (or `-s`).
|
||||||
|
- There is NO `--arch` flag -- architecture from downloaded image.
|
||||||
|
- CDN index: `https://images.linuxcontainers.org/os/index.json`
|
||||||
|
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`
|
||||||
|
|
||||||
|
## Seed archives
|
||||||
|
|
||||||
|
- Tar archives with YAML files at root level.
|
||||||
|
- Written to byte offset 2148532224 (seed partition) in the image.
|
||||||
|
- External boot media labeled `SEED_DATA` as CD-ROM:
|
||||||
|
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM.
|
||||||
|
- **FAT image**: works for USB but NOT CD-ROM (kernel sr_mod doesn't expose
|
||||||
|
FAT labels).
|
||||||
|
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
|
||||||
|
`operations-center.yaml`, `network.yaml`, `update.yaml`.
|
||||||
|
|
||||||
|
## Client certificates
|
||||||
|
|
||||||
|
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
|
||||||
|
- `incus remote list` triggers auto-generation if no keypair exists.
|
||||||
|
- Incus seed: `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
|
||||||
|
`InitPreseed.Server` uses `yaml:",inline"` so fields are promoted to top level.
|
||||||
|
- OC seed: `trusted_client_certificates[]`.
|
||||||
|
- OC **requires** at least one trusted cert or you're locked out.
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/incusos-proxmox"
|
||||||
|
- "incusos/lab-test"
|
||||||
|
- "incusos/examples/*"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lab Infrastructure
|
||||||
|
|
||||||
|
## Multi-lab coexistence
|
||||||
|
|
||||||
|
- `--lab-down config.yaml`: stops VMs (stay on disk).
|
||||||
|
- `--lab-up config.yaml`: starts stopped VMs (refuses if media attached).
|
||||||
|
- `--cleanup`: destroys permanently. These are distinct operations.
|
||||||
|
- `--resources`: host RAM/CPU/storage + per-pool allocation (API only).
|
||||||
|
- `--labs`: scans pool for `[incusos-lab:managed]` VMs, groups by config.
|
||||||
|
- `--lab-up` auto-deploy: offers full pipeline if no VMs exist yet.
|
||||||
|
|
||||||
|
## VMID range convention
|
||||||
|
|
||||||
|
| Range | Lab |
|
||||||
|
|---------|--------------------------------|
|
||||||
|
| 400-499 | OC-managed nodes |
|
||||||
|
| 800-809 | Single-node labs |
|
||||||
|
| 900-909 | Basic cluster |
|
||||||
|
| 910-919 | OC combined (server + nodes) |
|
||||||
|
| 920-929 | OC server standalone |
|
||||||
|
| 930-939 | Advanced / heterogeneous |
|
||||||
|
|
||||||
|
## Resource constraints
|
||||||
|
|
||||||
|
- **RAM is the bottleneck**: 4 GiB/VM minimum. 3-node = 12 GiB, OC lab = 28 GiB.
|
||||||
|
- **Storage**: thin provisioned (ZFS sparse). 3x 50 GiB = ~7-8 GiB actual.
|
||||||
|
- **CPU**: 4 cores/VM, 20 on host. Not a constraint.
|
||||||
|
- Pre-deploy checks warn if RAM exceeds available (API method).
|
||||||
|
|
@ -0,0 +1,58 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "notes/networking-guide.md"
|
||||||
|
- "notes/shared-storage-guide.md"
|
||||||
|
- "notes/migration-guide.md"
|
||||||
|
- "notes/utm-support.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Networking, Storage, and Migration
|
||||||
|
|
||||||
|
## OVN networking
|
||||||
|
|
||||||
|
- **Bridge networks are node-local**: same subnet but separate L2 domains.
|
||||||
|
- **OVN**: cross-node L2 overlay via Geneve tunnels. Sub-ms latency.
|
||||||
|
- **IncusOS OVN services disabled by default**: enable via `/os/1.0/services/ovn` on EVERY node.
|
||||||
|
- OVN service enable:
|
||||||
|
```bash
|
||||||
|
incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
|
||||||
|
"config": {
|
||||||
|
"database": "tcp:<SB_HOST>:6642",
|
||||||
|
"enabled": true,
|
||||||
|
"tunnel_address": "<THIS_NODE_LAN_IP>",
|
||||||
|
"tunnel_protocol": "geneve"
|
||||||
|
}, "state": {}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
`database` is **southbound** (6642), NOT northbound (6641).
|
||||||
|
- **Setup sequence**: deploy OVN container -> enable services -> set northbound
|
||||||
|
connection -> add ovn-chassis roles -> create uplink -> create OVN network.
|
||||||
|
- **Uplink**: `parent=mgmt` (NOT `ens18`). `ipv4.ovn.ranges` + `ipv4.gateway`.
|
||||||
|
- **LB/forward backends need IP addresses**, not instance names.
|
||||||
|
- See `notes/networking-guide.md` for full tutorial.
|
||||||
|
|
||||||
|
## Shared storage (iSCSI + lvmcluster)
|
||||||
|
|
||||||
|
- IncusOS-native: iSCSI initiator, lvmlockd, sanlock built in.
|
||||||
|
- **iSCSI API**: `/os/1.0/services/iscsi`. Field is `"target"` (NOT `"iqn"`).
|
||||||
|
- **LVM API**: `/os/1.0/services/lvm`. `system_id` must be 1-2000 (not 0).
|
||||||
|
- **lvmcluster**: thick provisioning. Two-step cluster pattern for pool creation.
|
||||||
|
- **Migration perf** (1GbE): container 0.15s, VM non-live 1.8s, VM live ~6s.
|
||||||
|
- **Proxy devices don't work for iSCSI** (portal mismatch). Use direct paths.
|
||||||
|
- **First live migration after stop/start may fail** (transient sanlock). Retry works.
|
||||||
|
- **Hybrid recommended**: local ZFS + shared lvmcluster for HA VMs.
|
||||||
|
- See `notes/shared-storage-guide.md` for full walkthrough.
|
||||||
|
|
||||||
|
## Migration into Incus
|
||||||
|
|
||||||
|
- `incus-migrate`: official import tool. `qemu-img convert` for format conversion.
|
||||||
|
- Workflow: convert disk -> `incus storage volume import` -> `incus init --empty --vm`
|
||||||
|
-> attach -> start.
|
||||||
|
- Docker: `docker export` -> `incus import`.
|
||||||
|
- See `notes/migration-guide.md` for per-hypervisor procedures.
|
||||||
|
|
||||||
|
## UTM support (future)
|
||||||
|
|
||||||
|
- Design doc at `notes/utm-support.md`.
|
||||||
|
- `utmctl` for start/stop but not VM creation (AppleScript or `.utm` bundle).
|
||||||
|
- No `blockstat` equivalent -- timeout + port polling for install detection.
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "notes/operations-center-guide.md"
|
||||||
|
- "incusos/incusos-proxmox"
|
||||||
|
- "incusos/examples/*oc*"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Operations Center
|
||||||
|
|
||||||
|
## Basics
|
||||||
|
|
||||||
|
- **CLI**: `operations-center` (from GitHub releases or `github.com/FuturFusion/operations-center`).
|
||||||
|
- **Config**: `~/.config/operations-center/` (copy certs from `~/.config/incus/`).
|
||||||
|
- **Port**: 8443. Browser needs PKCS#12 client cert (`client.pfx`). Web UI at `/ui/`.
|
||||||
|
- **v0.3.0** (active dev). Use `--version` to check.
|
||||||
|
- **No `remote:` suffix**: requires `operations-center remote switch NAME`.
|
||||||
|
|
||||||
|
## Provisioning workflow
|
||||||
|
|
||||||
|
- **No brownfield**: nodes must boot from OC-provisioned ISO.
|
||||||
|
- **Token seeds**: `provisioning token seed add/get-image`. No `force_reboot` for Proxmox.
|
||||||
|
```bash
|
||||||
|
operations-center provisioning token seed add <UUID> proxmox-preseed \
|
||||||
|
/tmp/preseed.yaml --description "No force_reboot for Proxmox"
|
||||||
|
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
|
||||||
|
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
|
||||||
|
```
|
||||||
|
- **Hybrid deploy** (recommended):
|
||||||
|
`incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
|
||||||
|
- **Self-registration**: nodes register with OC in ~30s after first boot.
|
||||||
|
|
||||||
|
## needs_update blocker (critical)
|
||||||
|
|
||||||
|
- OC requires `needs_update: false` for `provisioning cluster add`.
|
||||||
|
- Nodes from latest ISO are `needs_update: true` (never went through OC pipeline).
|
||||||
|
- **Fix**: use `--channel old-stable` for ISO. OC pushes update, clears flag.
|
||||||
|
|
||||||
|
## Cluster formation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo '{}' > /tmp/oc-app-config.yaml
|
||||||
|
operations-center provisioning cluster add oc-cluster \
|
||||||
|
https://<NODE_01_IP>:8443 \
|
||||||
|
--server-names oc-node-01,oc-node-02,oc-node-03 \
|
||||||
|
--server-type incus \
|
||||||
|
--application-seed-config /tmp/oc-app-config.yaml
|
||||||
|
```
|
||||||
|
- Use `apply_defaults: false` for OC nodes (recommended).
|
||||||
|
- Empty `{}` app config if cert already in trust store from SEED_DATA.
|
||||||
|
|
||||||
|
## Tested limitations
|
||||||
|
|
||||||
|
- Inventory NOT real-time -- requires `cluster resync`
|
||||||
|
- OC reboot breaks OC-managed Proxmox nodes (daemon crash from invalid state.txt)
|
||||||
|
- Fix: destroy and redeploy. Proxmox stop/start is safe.
|
||||||
|
- No cluster member state tracking (always shows `ready`)
|
||||||
|
- Stale entries from out-of-band changes persist
|
||||||
|
- Server removal blocked if part of OC cluster
|
||||||
|
- `needs_update` is pipeline-based, not version-based
|
||||||
|
- OVN LB has no health checks
|
||||||
|
|
||||||
|
## ISO upload skip bug (fixed)
|
||||||
|
|
||||||
|
- `--iso` now deletes existing ISO before re-upload (prevents stale reuse).
|
||||||
|
|
@ -0,0 +1,104 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/incusos-proxmox"
|
||||||
|
- "incusos/observe-deploy"
|
||||||
|
- "incusos/helpers/*"
|
||||||
|
- "incusos/examples/*"
|
||||||
|
- "incusos/TESTING.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Proxmox VE Deployment
|
||||||
|
|
||||||
|
## incusos-proxmox overview
|
||||||
|
|
||||||
|
- Reads YAML config, generates per-VM SEED_DATA via `incusos-seed --format iso`,
|
||||||
|
uploads ISO + seeds to Proxmox, creates VMs, boots through installation.
|
||||||
|
- **Config merge**: `proxmox.yaml` base -> lab config `proxmox:` overlay -> CLI flags.
|
||||||
|
Auto-discovers `proxmox.yaml` in script dir then cwd; override with `--proxmox FILE`.
|
||||||
|
- **API token**: `env` file at repo root (gitignored). Auto-loaded on startup.
|
||||||
|
- **Methods**: SSH (default) or API (`curl -k https://host:8006/api2/json/...`).
|
||||||
|
|
||||||
|
## Required VM settings (wrong = install failure)
|
||||||
|
|
||||||
|
- `bios=ovmf`, `machine=q35` -- UEFI required
|
||||||
|
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls own Secure Boot keys
|
||||||
|
- `tpmstate0`: `version=v2.0` -- required for disk encryption
|
||||||
|
- `cpu=host` -- x86_64_v3 instruction set requirement
|
||||||
|
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk broken with IncusOS
|
||||||
|
- `balloon=0` -- IncusOS manages memory internally
|
||||||
|
- `ide3` -- SEED_DATA ISO 9660 as second CD-ROM
|
||||||
|
- Minimum 50 GiB disk, 4096 MiB RAM
|
||||||
|
|
||||||
|
## Install flow (automated)
|
||||||
|
|
||||||
|
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
|
||||||
|
2. IncusOS installs to scsi0 (~876 MiB), sits at "remove media" prompt
|
||||||
|
3. Detect completion: poll `blockstat.scsi0.wr_bytes` -- writes start then stop
|
||||||
|
4. Stop VM (Proxmox stop)
|
||||||
|
5. **Delete ide2 and ide3** -- IncusOS refuses to start if media present
|
||||||
|
6. Set `boot: order=scsi0`, start from disk
|
||||||
|
|
||||||
|
## IP detection
|
||||||
|
|
||||||
|
- **Static IP** (preferred): `ip: ADDR/PREFIX` per VM + `gateway:` + `dns:`.
|
||||||
|
- **ARP-based** (DHCP fallback): MAC from VM config -> ARP table. Same L2 only.
|
||||||
|
|
||||||
|
## VLAN tagging
|
||||||
|
|
||||||
|
- `vlan:` config key adds tag to NIC (`tag=69`). Current lab: VLAN 69 (192.168.100.0/22).
|
||||||
|
|
||||||
|
## Disk target
|
||||||
|
|
||||||
|
- Do NOT specify `disk-target` for Proxmox VMs. IncusOS does **literal** string
|
||||||
|
matching. `scsi-*` does NOT match the actual device ID. Omit and let auto-detect.
|
||||||
|
|
||||||
|
## Resource pool isolation
|
||||||
|
|
||||||
|
- `proxmox.pool` scopes VM operations to a Proxmox pool. API token ACL on `/pool/<name>`.
|
||||||
|
|
||||||
|
## Reconcile and idempotency
|
||||||
|
|
||||||
|
- Re-runs with existing VMs: interactive menu (status/continue/destroy/abort).
|
||||||
|
- `--yes` defaults to status checks (never auto-destroys).
|
||||||
|
- `phase_install` checks each VM state before acting.
|
||||||
|
|
||||||
|
## Doctor and cleanup
|
||||||
|
|
||||||
|
- `--doctor`: environment check, no config needed.
|
||||||
|
- `--cleanup`: destroys config VMs. `--cleanup --deep`: also ISOs + remotes.
|
||||||
|
- `--cleanup-all`: pool-wide. `--cleanup-all --deep`: all IncusOS ISOs.
|
||||||
|
- `--verbose`/`-v`, `--quiet`, `--retries N` (default 3).
|
||||||
|
|
||||||
|
## Boot timeout
|
||||||
|
|
||||||
|
- **180s** (60s sleep + 120s poll). First boot downloads sysext (~30-120s).
|
||||||
|
Do NOT reduce -- premature retries can corrupt TPM encryption key permanently.
|
||||||
|
|
||||||
|
## Disk resize
|
||||||
|
|
||||||
|
- Proxmox `qm resize` only grows. Partition 11 auto-expands via `systemd-repart`.
|
||||||
|
- ZFS pool does NOT auto-expand. Manual expansion via privileged container needed.
|
||||||
|
See `notes/aether-guide.md` for procedure.
|
||||||
|
|
||||||
|
## First-boot sequence
|
||||||
|
|
||||||
|
1. ISO boot + install (~60-85s)
|
||||||
|
2. Transition: detect complete, stop, remove media, start (~15s)
|
||||||
|
3. First disk boot: encryption key, sysext download, "System ready" (~50s)
|
||||||
|
4. **Total: ~130-215s** to port 8443
|
||||||
|
|
||||||
|
## Crontab bug (issue #843)
|
||||||
|
|
||||||
|
- **Eliminated** by omitting `force_reboot`. Upstream bug only with `force_reboot: true`.
|
||||||
|
- Empty `ScrubSchedule` crashes daemon. `fix_scrub_schedule()` auto-heals.
|
||||||
|
- **TPM corruption risk**: hard-stop during first boot permanently corrupts.
|
||||||
|
180s timeout prevents this.
|
||||||
|
|
||||||
|
## API privileges (role: IncusOSDeployer)
|
||||||
|
|
||||||
|
```
|
||||||
|
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
|
||||||
|
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
|
||||||
|
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
|
||||||
|
Datastore.Audit SDN.Use Sys.Audit
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
# Proxmox SSH Root Access -- STRICT SAFETY RULES
|
||||||
|
|
||||||
|
Root SSH to Proxmox host is for **diagnostics only**.
|
||||||
|
Password: `PROXMOX_ROOT_PASSWORD` in `env` file.
|
||||||
|
|
||||||
|
## MANDATORY RULES -- violation is unacceptable
|
||||||
|
|
||||||
|
1. **Screenshots only**: ONLY use SSH for `qm monitor <vmid> screendump`.
|
||||||
|
No other commands without explicit user instruction.
|
||||||
|
2. **Test VMs only**: screenshot VMs in VMID range 400-499 or 800-939,
|
||||||
|
or VMs in the IncusLab pool created by our scripts.
|
||||||
|
3. **No modifications**: NEVER `qm set/stop/start/destroy`, `pct`,
|
||||||
|
`zfs`, `systemctl`, or ANY write operation on the host.
|
||||||
|
4. **Do not touch the dev VM**: user's dev VM is on the same host.
|
||||||
|
Do not interact with it in any way.
|
||||||
|
5. **No config access**: do not read/modify `/etc/pve/`, storage/network
|
||||||
|
configs, user/ACL settings, or host-level configuration.
|
||||||
|
6. **Transparency**: every SSH command is visible. If user rejects, stop.
|
||||||
|
7. **Only during active tests**: only when test VMs are deployed.
|
||||||
|
|
||||||
|
## Permitted commands (exhaustive)
|
||||||
|
|
||||||
|
Use `incusos/helpers/proxmox-screenshot VMID` instead of raw SSH.
|
||||||
|
|
||||||
|
Manual fallback (only if helper unavailable):
|
||||||
|
```bash
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||||
|
root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
|
||||||
|
root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||||
|
root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Technical notes
|
||||||
|
|
||||||
|
- `screendump` is root-only in PVE 9 HMP. API tokens cannot do it.
|
||||||
|
- Use `.ppm` format -- PVE 9.1 lacks libpng for PNG screendump.
|
||||||
|
- PPM ~3 MB -> PNG ~20 KB via `python3 -c "from PIL import Image; ..."`.
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
---
|
||||||
|
name: proxmox-api
|
||||||
|
description: Make an authenticated Proxmox API call
|
||||||
|
user_invocable: true
|
||||||
|
arguments:
|
||||||
|
- name: args
|
||||||
|
description: "METHOD PATH [--json]"
|
||||||
|
required: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Proxmox API Skill
|
||||||
|
|
||||||
|
Make an authenticated Proxmox API call with proper token handling.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Run the helper script:
|
||||||
|
```bash
|
||||||
|
/home/maarten/dev/incus-contrib/incusos/helpers/proxmox-api {{args}}
|
||||||
|
```
|
||||||
|
2. Parse the JSON response
|
||||||
|
3. Present the results in a readable format
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The helper handles the `!` in `automation@pve!deploy` safely
|
||||||
|
- Reads host and token ID from `proxmox.yaml`, secret from env file
|
||||||
|
- Add `--json` for pretty-printed output
|
||||||
|
- Common paths:
|
||||||
|
- `/nodes/pve/qemu` -- list all VMs
|
||||||
|
- `/nodes/pve/qemu/{vmid}/status/current` -- VM status
|
||||||
|
- `/nodes/pve/status` -- host resources
|
||||||
|
- `/cluster/resources` -- cluster-wide resources
|
||||||
|
- `/pools/{pool}` -- pool members
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
---
|
||||||
|
name: screenshot
|
||||||
|
description: Take a console screenshot of a Proxmox VM and view it
|
||||||
|
user_invocable: true
|
||||||
|
arguments:
|
||||||
|
- name: vmid
|
||||||
|
description: The VM ID to screenshot
|
||||||
|
required: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Screenshot Skill
|
||||||
|
|
||||||
|
Take a console screenshot of a Proxmox VM and display it.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Run the helper script to capture the screenshot:
|
||||||
|
```bash
|
||||||
|
/home/maarten/dev/incus-contrib/incusos/helpers/proxmox-screenshot {{vmid}}
|
||||||
|
```
|
||||||
|
2. Read the output path from stdout
|
||||||
|
3. Use the Read tool to view the PNG image
|
||||||
|
4. Describe what the VM console shows (boot stage, errors, prompts, etc.)
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The helper handles SSH, PPM capture, and PNG conversion automatically
|
||||||
|
- Requires `PROXMOX_ROOT_PASSWORD` in env file and `proxmox.yaml` for host
|
||||||
|
- Only works for VMIDs in allowed ranges (400-499, 800-939)
|
||||||
|
- Use this PROACTIVELY during deploy scenarios to monitor boot progress
|
||||||
894
CLAUDE.md
894
CLAUDE.md
|
|
@ -9,858 +9,104 @@ Primarily targeting home lab environments but aiming for production-quality scri
|
||||||
## Repository structure
|
## Repository structure
|
||||||
|
|
||||||
```
|
```
|
||||||
incu-contrib/
|
incus-contrib/
|
||||||
├── CLAUDE.md # This file -- project context
|
├── CLAUDE.md # This file -- behavioral rules + project overview
|
||||||
|
├── .claude/rules/ # Topic-specific context (auto-loaded by paths:)
|
||||||
|
├── .claude/skills/ # Slash commands: /screenshot, /proxmox-api
|
||||||
├── README.md # Main overview
|
├── README.md # Main overview
|
||||||
├── .gitignore
|
├── .gitignore
|
||||||
├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks
|
├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks
|
||||||
│ ├── ansible.cfg # Project-level Ansible config
|
│ ├── ansible.cfg
|
||||||
│ └── playbooks/
|
│ └── playbooks/
|
||||||
│ ├── post-deploy.yml # Runs after Aether creates an instance (Incus API)
|
│ ├── post-deploy.yml # Runs after Aether creates an instance
|
||||||
│ └── decommission.yml # Runs before Aether deletes an instance
|
│ └── decommission.yml # Runs before Aether deletes an instance
|
||||||
├── incusos/ # IncusOS installation tooling
|
├── incusos/ # IncusOS installation tooling
|
||||||
│ ├── README.md # Detailed usage docs
|
│ ├── README.md
|
||||||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||||
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
|
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
||||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||||
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||||
|
│ ├── helpers/
|
||||||
|
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||||
|
│ │ └── proxmox-api # Authenticated API calls (handles ! in token)
|
||||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||||
│ ├── observe-deploy # Single-VM deploy with rapid console screenshots
|
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||||
│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored)
|
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||||
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
│ ├── TESTING.md
|
||||||
│ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
|
|
||||||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
|
||||||
│ └── examples/ # Example seed + Proxmox YAML files
|
│ └── examples/ # Example seed + Proxmox YAML files
|
||||||
└── notes/ # Research notes and reference material
|
├── env # PROXMOX_TOKEN_SECRET + PROXMOX_ROOT_PASSWORD (gitignored)
|
||||||
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
└── notes/ # Reference guides (clustering, networking, OC, AWX, etc.)
|
||||||
├── operations-center-guide.md # Operations Center provisioning & management
|
|
||||||
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
|
||||||
├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested)
|
|
||||||
├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end)
|
|
||||||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
|
||||||
├── aether-guide.md # Aether management platform (deploy, blueprints, API)
|
|
||||||
├── awx-guide.md # AWX + Aether Ansible automation
|
|
||||||
├── incus-version-compatibility.md # Incus versions across platforms
|
|
||||||
├── iso-download-methods.md # ISO download/customization research
|
|
||||||
└── utm-support.md # UTM support design document (future)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key technical context
|
## Capabilities you have
|
||||||
|
|
||||||
### Incus version differences
|
### Proxmox VM screenshots
|
||||||
|
Take console screenshots during deploys to see boot progress or errors:
|
||||||
|
```
|
||||||
|
incusos/helpers/proxmox-screenshot VMID [/tmp/output.png]
|
||||||
|
```
|
||||||
|
Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during
|
||||||
|
deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
|
||||||
|
|
||||||
- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream.
|
### Proxmox API calls
|
||||||
The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu.
|
Make authenticated API calls (handles the `!` in `automation@pve!deploy`):
|
||||||
- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21).
|
```
|
||||||
macOS is client-only by design; Arch has no client-only split package.
|
incusos/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
|
||||||
- `incus remote get-client-certificate` was added in **Incus 6.3+** and does
|
incusos/helpers/proxmox-api GET /nodes/pve/status --json
|
||||||
not exist in 6.0 LTS. Scripts must never depend on it as the only cert path.
|
incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
|
||||||
- Always prefer reading `~/.config/incus/client.crt` directly from disk.
|
```
|
||||||
Fall back to the CLI command only as a secondary option.
|
|
||||||
- See `notes/incus-version-compatibility.md` for full platform matrix and
|
|
||||||
install instructions.
|
|
||||||
|
|
||||||
### IncusOS flasher-tool
|
### Live AWX job output
|
||||||
|
Capture output while a job is running (don't just poll status):
|
||||||
- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest`
|
|
||||||
- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version`
|
|
||||||
- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`).
|
|
||||||
- There is NO `--arch` flag -- architecture is determined by the downloaded image.
|
|
||||||
For cross-arch builds, download the image manually and pass via `--image`.
|
|
||||||
- CDN index: `https://images.linuxcontainers.org/os/index.json`
|
|
||||||
- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz`
|
|
||||||
|
|
||||||
### Seed archives
|
|
||||||
|
|
||||||
- Tar archives containing YAML files at the root level.
|
|
||||||
- Written to byte offset 2148532224 (the seed partition) in the image.
|
|
||||||
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
|
|
||||||
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
|
|
||||||
devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
|
|
||||||
- **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
|
|
||||||
**NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
|
|
||||||
FAT filesystem labels, so IncusOS cannot find the seed.
|
|
||||||
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
|
|
||||||
`operations-center.yaml`, `network.yaml`, `update.yaml`.
|
|
||||||
|
|
||||||
### Client certificates
|
|
||||||
|
|
||||||
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
|
|
||||||
- Running `incus remote list` triggers auto-generation if no keypair exists.
|
|
||||||
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
|
|
||||||
The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
|
|
||||||
`certificates`) are promoted to the top level of the `preseed` object.
|
|
||||||
- For Operations Center seed: injected under `trusted_client_certificates[]`.
|
|
||||||
- Operations Center **requires** at least one trusted certificate -- without it,
|
|
||||||
you are locked out after installation.
|
|
||||||
|
|
||||||
### Proxmox VE deployment
|
|
||||||
|
|
||||||
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
|
||||||
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
|
||||||
VMs with IncusOS-correct settings, and boots them through installation.
|
|
||||||
- **Config separation**: Proxmox connection settings (host, credentials, pool)
|
|
||||||
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
|
|
||||||
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
|
|
||||||
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
|
|
||||||
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
|
|
||||||
- **API token secret**: stored in `env` file at the repo root (gitignored).
|
|
||||||
Scripts auto-load it on startup by searching for `env` up the directory
|
|
||||||
tree from the script location. No manual `source env` needed. If the file
|
|
||||||
is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail
|
|
||||||
with a clear error message.
|
|
||||||
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
|
||||||
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
|
||||||
- **Minimum API privileges** for token-based access (role: `IncusOSDeployer`):
|
|
||||||
```
|
|
||||||
VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory
|
|
||||||
VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType
|
|
||||||
VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate
|
|
||||||
Datastore.Audit SDN.Use Sys.Audit
|
|
||||||
```
|
|
||||||
`Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via
|
|
||||||
`/nodes/<node>/status`). It requires a separate ACL on `/nodes/<node>`
|
|
||||||
since the pool-scoped ACL doesn't cover node-level endpoints.
|
|
||||||
- **Required VM settings** (getting any wrong causes IncusOS install failure):
|
|
||||||
- `bios=ovmf`, `machine=q35` -- UEFI boot required
|
|
||||||
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
|
|
||||||
- `tpmstate0`: `version=v2.0` -- required for disk encryption
|
|
||||||
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
|
|
||||||
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
|
|
||||||
- `balloon=0` -- IncusOS manages memory internally
|
|
||||||
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
|
||||||
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
|
||||||
- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag
|
|
||||||
to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM
|
|
||||||
on the tagged VLAN instead of the native/untagged network. The current lab
|
|
||||||
uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag,
|
|
||||||
VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the
|
|
||||||
Proxmox VM level only — IncusOS and Incus instances inside the VM are
|
|
||||||
unaware of it. The setting is optional: omit `vlan:` for untagged access.
|
|
||||||
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
|
||||||
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
|
||||||
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
|
||||||
`disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
|
|
||||||
- **Install flow** (automated by `incusos-proxmox`):
|
|
||||||
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
|
|
||||||
2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone),
|
|
||||||
then sits at "please remove installation media" prompt
|
|
||||||
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
|
|
||||||
when disk writes start then stop for 15s (3 stable polls), install is done
|
|
||||||
4. Stop the VM (Proxmox stop, not guest shutdown)
|
|
||||||
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
|
|
||||||
and refuses to start if found, regardless of boot order
|
|
||||||
6. Set boot order to `order=scsi0` and start from disk
|
|
||||||
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two
|
|
||||||
strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM
|
|
||||||
in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's
|
|
||||||
`network.yaml` configures the VM's interface at boot. No ARP scan or
|
|
||||||
SSH needed — the IP is known at deploy time. Works across VLANs.
|
|
||||||
(2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM
|
|
||||||
config → flush stale ARP → ping broadcast → look up MAC in ARP table.
|
|
||||||
Only works on the same L2 domain (native LAN, not across VLANs).
|
|
||||||
- **force_reboot is NOT used on Proxmox**: the seed omits `force_reboot`.
|
|
||||||
IncusOS sits at "please remove installation media" after install. We
|
|
||||||
detect completion via blockstat (876 MiB written, then idle), stop the
|
|
||||||
VM externally, remove media, and start from disk. This avoids the
|
|
||||||
crontab race condition (issue #843). On physical hardware,
|
|
||||||
`force_reboot: true` is still needed (no external orchestrator).
|
|
||||||
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
|
|
||||||
all VM operations to a Proxmox resource pool. When set, the script only
|
|
||||||
"sees" VMs in that pool (for collision detection and cleanup), and the API
|
|
||||||
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
|
|
||||||
```
|
|
||||||
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
|
||||||
pveum pool modify IncusLab --storage local-lvm,local
|
|
||||||
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
|
|
||||||
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
|
|
||||||
```
|
|
||||||
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
|
|
||||||
deployment status (Proxmox state, install status, IP, port 8443, incus
|
|
||||||
remote). Also runs post-deployment checks (Incus connectivity, Operations
|
|
||||||
Center URL).
|
|
||||||
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
|
|
||||||
config, an interactive menu offers: (1) run status checks, (2) continue
|
|
||||||
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
|
|
||||||
`--yes`, defaults to option 1 (safe -- never auto-destroys).
|
|
||||||
- **Install idempotency**: `phase_install` checks each VM's state before
|
|
||||||
acting -- already-running VMs are skipped, stopped-but-installed VMs are
|
|
||||||
started from disk, only VMs with install media proceed through installation.
|
|
||||||
|
|
||||||
### Multi-lab coexistence
|
|
||||||
|
|
||||||
- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop,
|
|
||||||
VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk
|
|
||||||
(refuses to start VMs with install media still attached). These are
|
|
||||||
distinct from `--cleanup` (which destroys VMs permanently).
|
|
||||||
- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage
|
|
||||||
usage, and per-pool allocation. Requires API method.
|
|
||||||
- **Lab inventory**: `--labs` scans the pool for managed VMs (by
|
|
||||||
`[incusos-lab:managed]` marker), groups by config file, and shows
|
|
||||||
per-lab status, VM count, RAM, and disk.
|
|
||||||
- **VMID range convention** (to avoid collisions between coexisting labs):
|
|
||||||
|
|
||||||
| Range | Lab |
|
|
||||||
|-------|-----|
|
|
||||||
| 400-499 | OC-managed nodes |
|
|
||||||
| 800-809 | Single-node labs |
|
|
||||||
| 900-909 | Basic cluster |
|
|
||||||
| 910-919 | OC combined (server + nodes) |
|
|
||||||
| 920-929 | OC server standalone |
|
|
||||||
| 930-939 | Advanced / heterogeneous |
|
|
||||||
|
|
||||||
### Resource constraints for multi-lab
|
|
||||||
|
|
||||||
- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node
|
|
||||||
cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource
|
|
||||||
where you can actually run out.
|
|
||||||
- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default
|
|
||||||
(`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk.
|
|
||||||
LZ4 compression provides ~1.5-1.8x ratio on OS data.
|
|
||||||
- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs
|
|
||||||
can share CPUs without contention.
|
|
||||||
- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if
|
|
||||||
requested RAM exceeds available host RAM (API method only).
|
|
||||||
- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to
|
|
||||||
run the full deploy pipeline (auto-accepts with `--yes`).
|
|
||||||
- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs
|
|
||||||
(via storage content API) and notes thin provisioning for ZFS/LVM-thin.
|
|
||||||
|
|
||||||
### Incus clustering via remotes
|
|
||||||
|
|
||||||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
|
||||||
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
|
||||||
- **No VIP needed**: each node advertises its own IP as its cluster address.
|
|
||||||
Clients can connect to any cluster member; requests are forwarded internally.
|
|
||||||
|
|
||||||
#### Pre-clustering: fix core.https_address
|
|
||||||
|
|
||||||
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
|
|
||||||
interfaces). Clustering requires a **specific routable IP** so nodes can
|
|
||||||
address each other.
|
|
||||||
- **Set the IP on every node BEFORE enabling clustering:**
|
|
||||||
```bash
|
|
||||||
incus config set <remote>: core.https_address <NODE_IP>:8443
|
|
||||||
```
|
|
||||||
- Get each node's routable IP via the API:
|
|
||||||
```bash
|
|
||||||
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
|
||||||
[print(a) for a in d['environment']['addresses'] \
|
|
||||||
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
|
||||||
```
|
|
||||||
- This is safe to do while remotes are connected -- the remote already points
|
|
||||||
to the specific IP; we're just narrowing the bind address. Certificate trust
|
|
||||||
is fingerprint-based, not address-based.
|
|
||||||
|
|
||||||
#### Cluster enable (init node)
|
|
||||||
|
|
||||||
- ```bash
|
|
||||||
incus cluster enable <remote>: <member-name>
|
|
||||||
```
|
|
||||||
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
|
||||||
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
|
|
||||||
- **TLS certificate regeneration**: enabling clustering causes the server to
|
|
||||||
generate a new TLS certificate (cluster cert). The new cert may only have
|
|
||||||
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
|
|
||||||
- **Fix**: remove and re-add the remote to pin the new certificate:
|
|
||||||
```bash
|
|
||||||
incus remote switch local # if init remote is current default
|
|
||||||
incus remote remove <remote>
|
|
||||||
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
|
||||||
```
|
|
||||||
- The cert trust on the server side (client → server) is unaffected -- it's
|
|
||||||
stored by fingerprint in the Incus database, independent of listen address.
|
|
||||||
|
|
||||||
#### Joining nodes: apply_defaults and the storage pool conflict
|
|
||||||
|
|
||||||
- **Upstream recommendation**: use `apply_defaults: false` for nodes destined
|
|
||||||
to join a cluster. The official IncusOS clustering tutorial states joining
|
|
||||||
servers "cannot have preexisting networks or storage pools defined." With
|
|
||||||
`apply_defaults: false`, the node still listens on port 8443, still trusts
|
|
||||||
preseed certificates, and the underlying ZFS dataset (`local/incus`) still
|
|
||||||
exists -- but no Incus storage pool or network metadata is created, so
|
|
||||||
the join process works cleanly.
|
|
||||||
- **`apply_defaults: true` on joining nodes** is also functional but requires
|
|
||||||
an 8-command cleanup per node before join (delete pool, network, volumes,
|
|
||||||
profile devices). This is automated in `lab-test` but adds complexity.
|
|
||||||
- **Recommended seed pattern for clusters**:
|
|
||||||
- Bootstrap/init node: `apply_defaults: true` (needs pool and network)
|
|
||||||
- Joining nodes: `apply_defaults: false` (join process creates member-specific entries)
|
|
||||||
- Standalone nodes: `apply_defaults: true` (needs pool and network to be functional)
|
|
||||||
- **If `apply_defaults: true` was used**, the cleanup before join is:
|
|
||||||
```bash
|
|
||||||
# 1. Remove config references
|
|
||||||
incus config unset <remote>: storage.backups_volume
|
|
||||||
incus config unset <remote>: storage.images_volume
|
|
||||||
# 2. Delete volumes
|
|
||||||
incus storage volume delete <remote>:local backups
|
|
||||||
incus storage volume delete <remote>:local images
|
|
||||||
# 3. Clear default profile references (pool is "in use" otherwise)
|
|
||||||
incus profile device remove <remote>:default root
|
|
||||||
incus profile device remove <remote>:default eth0
|
|
||||||
# 4. Delete pool and network
|
|
||||||
incus storage delete <remote>:local
|
|
||||||
incus network delete <remote>:incusbr0
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Join workflow
|
|
||||||
|
|
||||||
- **Generate token** (on init node, single argument `remote:member-name`):
|
|
||||||
```bash
|
|
||||||
incus cluster add <init-remote>:<new-member-name>
|
|
||||||
```
|
|
||||||
- **Join** (interactive -- prompts for 5 values):
|
|
||||||
```bash
|
|
||||||
incus cluster join <init-remote>: <joining-remote>:
|
|
||||||
```
|
|
||||||
Interactive prompts and correct answers:
|
|
||||||
1. IP address → accept default (node's IP, already set via core.https_address)
|
|
||||||
2. Member name → accept default (matches the token)
|
|
||||||
3. "All existing data is lost" → `yes`
|
|
||||||
4. `source` property for storage pool "local" → `local/incus`
|
|
||||||
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
|
|
||||||
- **Automated (non-interactive):**
|
|
||||||
```bash
|
|
||||||
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
|
||||||
```
|
|
||||||
- **After join**: the joining node gets a new cluster certificate. Fix the
|
|
||||||
remote (same as init node):
|
|
||||||
```bash
|
|
||||||
incus remote remove <joining-remote>
|
|
||||||
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Command syntax gotchas
|
|
||||||
|
|
||||||
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
|
|
||||||
- `incus cluster add remote:member-name` -- ONE argument (no space)
|
|
||||||
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
|
|
||||||
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
|
|
||||||
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
|
|
||||||
- `incus cluster restore remote:member-name` -- ONE argument (no space)
|
|
||||||
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
|
|
||||||
- `incus storage show remote:pool` -- ONE argument (no space)
|
|
||||||
- `incus storage show remote:pool --target member` -- target flag for
|
|
||||||
member-specific config
|
|
||||||
- `incus config set remote: key value` -- remote with trailing colon + space
|
|
||||||
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
|
|
||||||
colon) for targeting the server itself
|
|
||||||
|
|
||||||
#### Post-join state
|
|
||||||
|
|
||||||
- After joining, the cluster is managed through the init node's remote. The
|
|
||||||
individual node remotes still work for node-specific operations.
|
|
||||||
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
|
||||||
|
|
||||||
#### Workload placement and migration
|
|
||||||
|
|
||||||
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
|
|
||||||
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
|
|
||||||
- **Container migration**: stop/move/start only (CRIU live migration is
|
|
||||||
unreliable). Data persists, processes do not.
|
|
||||||
```bash
|
|
||||||
incus stop <remote>:<instance>
|
|
||||||
incus move <remote>:<instance> --target <destination-member>
|
|
||||||
incus start <remote>:<instance>
|
|
||||||
```
|
|
||||||
- **VM live migration**: requires `migration.stateful=true` (must be set
|
|
||||||
while VM is stopped). Preserves running state with no downtime.
|
|
||||||
```bash
|
|
||||||
incus move <remote>:<instance> --target <destination-member>
|
|
||||||
```
|
|
||||||
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
|
|
||||||
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
|
|
||||||
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
|
|
||||||
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
|
|
||||||
differently, causing `Missing section footer for ICH9LPC` on restore.
|
|
||||||
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
|
|
||||||
fixed `sockets/cores/threads` topology — portable across all hosts.
|
|
||||||
```bash
|
|
||||||
# WRONG: integer → maxcpus varies by host → migration fails
|
|
||||||
incus config set <instance> limits.cpu=2
|
|
||||||
# RIGHT: range → fixed topology → migration works everywhere
|
|
||||||
incus config set <instance> limits.cpu=0-1
|
|
||||||
```
|
|
||||||
- **VM live migration works in nested virtualization** (IncusOS inside
|
|
||||||
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
|
|
||||||
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
|
|
||||||
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
|
|
||||||
appears during migration is cosmetic. It fires from QEMU's feature
|
|
||||||
dependency checker before KVM filters out unsupported features and does
|
|
||||||
not affect migration.
|
|
||||||
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
|
|
||||||
requires the `limits.cpu` range fix. Use `incus start --stateless` to
|
|
||||||
discard a saved state file that cannot be restored.
|
|
||||||
- **VM `size.state` config**: stateful operations require `size.state` on
|
|
||||||
the root disk (`incus config device add <instance> root disk path=/
|
|
||||||
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
|
|
||||||
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
|
||||||
(ONE argument, like `cluster enable` and `cluster add`).
|
|
||||||
Use `--action stop` if VMs lack the `limits.cpu` range fix.
|
|
||||||
Restore with `incus cluster restore <remote>:<member> --force`.
|
|
||||||
- **VM agent reconnect**: after live migration, the incus agent inside the VM
|
|
||||||
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
|
|
||||||
after migration may fail with "VM agent isn't currently running". Scripts
|
|
||||||
should `sleep 4` after migration before running `incus exec`.
|
|
||||||
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
|
|
||||||
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
|
|
||||||
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
|
|
||||||
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
|
|
||||||
- **Concurrent migrations**: migrating multiple VMs simultaneously from
|
|
||||||
different source nodes works without interference. ~140 MB/s per migration.
|
|
||||||
- **Active I/O during migration**: disk writes and network activity survive
|
|
||||||
live migration transparently. File integrity verified after migration.
|
|
||||||
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
|
|
||||||
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
|
|
||||||
auto-rebalanced.
|
|
||||||
```bash
|
|
||||||
incus config set <remote>: cluster.rebalance.interval=1 # minutes
|
|
||||||
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
|
|
||||||
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
|
|
||||||
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
|
|
||||||
```
|
|
||||||
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
|
|
||||||
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
|
|
||||||
for step-by-step instructions.
|
|
||||||
- **`incus cluster remove` requires confirmation**: even with `--force`, it
|
|
||||||
prompts "Are you really sure?". Pipe `yes` for automation:
|
|
||||||
```bash
|
|
||||||
printf "yes\n" | incus cluster remove <remote>:<member> --force
|
|
||||||
```
|
|
||||||
- See `notes/clustering-guide.md` for full details and references.
|
|
||||||
|
|
||||||
### Lab validation (lab-test)
|
|
||||||
|
|
||||||
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
|
||||||
on the VM names defined there (expects `incus` remotes to exist).
|
|
||||||
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
|
|
||||||
migrate (stop/move + evacuate/restore).
|
|
||||||
- **Test instances** use names starting with `test-` for easy cleanup.
|
|
||||||
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
|
|
||||||
|
|
||||||
### Operations Center
|
|
||||||
|
|
||||||
- **CLI binary:** `operations-center` (installed from GitHub releases or
|
|
||||||
built from source at `github.com/FuturFusion/operations-center`).
|
|
||||||
- **Config directory:** `~/.config/operations-center/` (uses same cert format
|
|
||||||
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
|
||||||
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
|
|
||||||
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
|
|
||||||
imported into the browser. Web UI is a React SPA at `/ui/`.
|
|
||||||
- **OC is under active development** (v0.3.0). Commands and APIs may change.
|
|
||||||
Use `operations-center --version` (not `version` subcommand) to check.
|
|
||||||
- **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI,
|
|
||||||
the OC CLI requires `operations-center remote switch NAME` before running
|
|
||||||
commands. `operations-center admin os show oc-lab:` fails with "Invalid
|
|
||||||
number of arguments".
|
|
||||||
- **v0.3.0 new commands**: `provisioning channel` (add/list/show),
|
|
||||||
`cluster update/update-certificate/rename`, expanded `inventory`
|
|
||||||
(network-acl, address-sets, load-balancers, peers, zones, integrations,
|
|
||||||
forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource
|
|
||||||
tree view with filters and Go templates), `system certificate set`.
|
|
||||||
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
|
||||||
installed.
|
|
||||||
|
|
||||||
#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster)
|
|
||||||
|
|
||||||
- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
|
|
||||||
- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
|
|
||||||
must use structured format with section keys (`install:`, not flat).
|
|
||||||
**No force_reboot** in token seeds for Proxmox (same fix as standard deploys).
|
|
||||||
```bash
|
|
||||||
operations-center provisioning token seed add <UUID> proxmox-preseed \
|
|
||||||
/tmp/preseed.yaml --description "No force_reboot for Proxmox"
|
|
||||||
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
|
|
||||||
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
|
|
||||||
```
|
|
||||||
- **Hybrid deployment** (tested, recommended):
|
|
||||||
`incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
|
|
||||||
combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
|
|
||||||
VM creation, per-node SEED_DATA (hostname, static IP), install monitoring,
|
|
||||||
and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
|
|
||||||
- **Self-registration**: nodes auto-register with OC within ~30s of first
|
|
||||||
boot. Hostname from SEED_DATA is used as the server name.
|
|
||||||
- **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires
|
|
||||||
all nodes to have `needs_update: false` before `provisioning cluster add`
|
|
||||||
succeeds. Nodes deployed from an ISO matching the latest OC update version
|
|
||||||
are tracked as `needs_update: true` because the OS was never delivered
|
|
||||||
through OC's update pipeline. The `needs_update` flag is server-side
|
|
||||||
computed (not a simple version comparison) and cannot be overridden via
|
|
||||||
REST API PUT. **Solution**: generate the ISO from an older channel
|
|
||||||
(`--channel old-stable`) so nodes start with an older version. OC then
|
|
||||||
pushes the latest update through its pipeline, clearing the flag. Use
|
|
||||||
`provisioning update assign-channels` to control which versions are in
|
|
||||||
which channels.
|
|
||||||
- **Cluster formation** (tested, 2026-02-23):
|
|
||||||
```bash
|
|
||||||
# Use empty app config if cert already injected via SEED_DATA
|
|
||||||
echo '{}' > /tmp/oc-app-config.yaml
|
|
||||||
operations-center provisioning cluster add oc-cluster \
|
|
||||||
https://<NODE_01_IP>:8443 \
|
|
||||||
--server-names oc-node-01,oc-node-02,oc-node-03 \
|
|
||||||
--server-type incus \
|
|
||||||
--application-seed-config /tmp/oc-app-config.yaml
|
|
||||||
```
|
|
||||||
OC handles: `core.https_address` → cluster enable → joins → storage pool
|
|
||||||
+ network creation → Terraform config. Adds `meshbr0` network. If the
|
|
||||||
client cert is already in the trust store (from SEED_DATA), use an empty
|
|
||||||
`{}` app config to avoid "Certificate already in trust store" Terraform
|
|
||||||
error. The cluster forms successfully either way.
|
|
||||||
- **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
|
|
||||||
for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
|
|
||||||
creation cleanly. With `apply_defaults: true`, nodes already have these
|
|
||||||
resources and OC's Terraform fails with "already exists" errors (cluster still
|
|
||||||
forms, but Terraform artifacts are empty).
|
|
||||||
- **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster
|
|
||||||
formation via OC, OVN overlay networking works identically to manual clusters.
|
|
||||||
Deploy ovn-central container, enable OVN services, create UPLINK + OVN
|
|
||||||
network, add ovn-chassis roles. HA nginx workload with OVN load balancer
|
|
||||||
tested and working. See `notes/operations-center-guide.md` for full guide.
|
|
||||||
- **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously
|
|
||||||
skipped uploading an ISO if one with the same filename existed on Proxmox.
|
|
||||||
This caused stale ISOs from previous deployments to be silently reused.
|
|
||||||
Fixed: when `--iso` is explicitly provided, the script now deletes the
|
|
||||||
existing ISO and re-uploads the new one.
|
|
||||||
|
|
||||||
#### Tested limitations
|
|
||||||
|
|
||||||
- **Inventory is NOT real-time** -- requires explicit `cluster resync`
|
|
||||||
- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
|
|
||||||
on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
|
|
||||||
~50s with data intact). The failure is OC-specific: the OC agent pushes
|
|
||||||
config via the IncusOS REST API that gets persisted to `state.txt`. On
|
|
||||||
reboot, invalid values (e.g., cron expression where Go duration is expected)
|
|
||||||
crash the daemon. See "IncusOS boot failure" section below for full analysis.
|
|
||||||
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
|
|
||||||
- **No cluster member state tracking** -- OC always shows `ready` even for
|
|
||||||
EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat
|
|
||||||
detects failure in ~40s; OC has no equivalent).
|
|
||||||
- **Stale entries** from out-of-band cluster changes persist after resync
|
|
||||||
- **Server removal blocked** if server is part of an OC cluster
|
|
||||||
- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
|
|
||||||
node auto-rejoins cluster in ~60s. Containers auto-start.
|
|
||||||
- **`needs_update` tracking is pipeline-based**: OC tracks whether an update
|
|
||||||
was delivered through its pipeline, not just whether versions match. Nodes
|
|
||||||
deployed from the latest ISO are tracked as needing updates even when
|
|
||||||
`version == available_version`. Must deploy from older ISO to work around.
|
|
||||||
- **OVN LB has no health checks**: connection-based hashing distributes
|
|
||||||
traffic to dead backends. Requests to stopped instances return empty.
|
|
||||||
|
|
||||||
- See `notes/operations-center-guide.md` for full tested OC reference.
|
|
||||||
|
|
||||||
### incusos-proxmox doctor and cleanup
|
|
||||||
|
|
||||||
- **`--doctor`**: standalone environment check. No config file required.
|
|
||||||
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
|
|
||||||
Proxmox connectivity (from proxmox.yaml or config file).
|
|
||||||
- **`--cleanup`**: destroys VMs defined in the config file.
|
|
||||||
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
|
|
||||||
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
|
|
||||||
all IncusOS ISOs (unlike the old behavior).
|
|
||||||
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
|
|
||||||
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
|
|
||||||
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
|
|
||||||
and `seed-*.iso` from storage + remotes + cache.
|
|
||||||
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
|
||||||
output is concise (step names + results). `--quiet` suppresses everything
|
|
||||||
except warnings and errors.
|
|
||||||
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
|
|
||||||
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
|
|
||||||
Retries are rarely needed since omitting `force_reboot`. After boot,
|
|
||||||
`fix_scrub_schedule()` proactively heals any scrub_schedule issues.
|
|
||||||
- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot
|
|
||||||
downloads application sysext (~30-120s depending on CDN speed). Do not
|
|
||||||
reduce — premature retries can corrupt the TPM encryption key permanently.
|
|
||||||
|
|
||||||
### IncusOS disk resize (Proxmox VMs)
|
|
||||||
|
|
||||||
- **Disk grow is non-destructive**: Proxmox `qm resize` only grows disks.
|
|
||||||
- **Partition 11 auto-expands**: `systemd-repart` runs in initrd on every
|
|
||||||
boot. The `local-data` partition has no `SizeMaxBytes`, so it fills all
|
|
||||||
remaining space automatically.
|
|
||||||
- **ZFS pool does NOT auto-expand**: IncusOS creates the pool without
|
|
||||||
`autoexpand=on`. After partition grow, `zpool list` shows the correct
|
|
||||||
`EXPANDSZ` but the pool stays at its original size.
|
|
||||||
- **Manual expansion via privileged container**: create a container with
|
|
||||||
`security.privileged=true`, pass `/dev/zfs` (unix-char) and `/dev/sda11`
|
|
||||||
(unix-block), install `zfsutils-linux`, create a symlink at
|
|
||||||
`/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11` →
|
|
||||||
`/dev/sda11`, then run `zpool online -e local
|
|
||||||
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11`. Clean up the container
|
|
||||||
after expansion. Full procedure in `notes/aether-guide.md`.
|
|
||||||
- **Warning**: running `zpool online -e` with a non-existent device path
|
|
||||||
inside the container causes pool SUSPENSION. Recovery: Proxmox stop/start
|
|
||||||
(ZFS re-imports cleanly, no data loss).
|
|
||||||
|
|
||||||
### IncusOS first-boot sequence
|
|
||||||
|
|
||||||
Complete lifecycle from Proxmox VM start to port 8443 ready:
|
|
||||||
|
|
||||||
**Phase 1: ISO boot and installation (~60-85s)**
|
|
||||||
1. UEFI firmware → IncusOS boot menu → "Starting install of IncusOS to local disk"
|
|
||||||
2. "Cloning GPT partitions" → progress bar → complete (~876 MiB written)
|
|
||||||
3. Without `force_reboot`, installer sits at "please remove installation media"
|
|
||||||
|
|
||||||
**Phase 2: Transition (~15s)**
|
|
||||||
1. `incusos-proxmox` detects install complete via blockstat (876 MiB written, then idle)
|
|
||||||
2. Proxmox stops VM, removes ide2 (ISO) and ide3 (seed), sets `boot: order=scsi0`
|
|
||||||
3. Start VM from disk
|
|
||||||
|
|
||||||
**Phase 3: First boot from disk (~50s to port 8443)**
|
|
||||||
1. UEFI → "IncusOS is starting..."
|
|
||||||
2. `state.LoadOrCreate()` creates `state.txt` with defaults
|
|
||||||
3. "Auto-generating encryption recovery key" (~10s)
|
|
||||||
4. "Downloading SecureBoot update" + "Downloading application update" (~33-38s)
|
|
||||||
5. "Starting application" → "System is ready" — port 8443 now reachable
|
|
||||||
|
|
||||||
**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable
|
|
||||||
is the application sysext download (~33-38s on first boot; skipped if cached).
|
|
||||||
|
|
||||||
### IncusOS crontab bug (upstream issue #843)
|
|
||||||
|
|
||||||
**Status:** Eliminated in our pipeline by omitting `force_reboot` from seeds.
|
|
||||||
The upstream bug still exists but only triggers with `force_reboot: true`.
|
|
||||||
|
|
||||||
- **What it is**: a race condition in IncusOS where `ScrubSchedule` ends up
|
|
||||||
empty in `state.txt`, causing `registerJobs()` → `gocron.IsValid("")` to
|
|
||||||
fail and the daemon to exit. Port 8443 never opens (or opens briefly
|
|
||||||
before the daemon crashes).
|
|
||||||
- **Our fix**: omit `force_reboot` from the seed. Without it, there is no
|
|
||||||
SysRq-B intermediate boot, and the race condition does not trigger.
|
|
||||||
Result: **100% success rate** (vs ~50% with `force_reboot` on 4-core VMs).
|
|
||||||
- **Detection**: port 8443 reachable but `scrub_schedule` is empty in
|
|
||||||
`GET /os/1.0/system/storage` → bug hit. `incusos-proxmox` checks this
|
|
||||||
automatically.
|
|
||||||
- **Auto-heal**: `incusos-proxmox` includes `fix_scrub_schedule()` which
|
|
||||||
sets `scrub_schedule` to `"0 4 * * 0"` via `PUT /os/1.0/system/storage`
|
|
||||||
on every deployed node as a safety net.
|
|
||||||
- **TPM corruption risk**: hard-stopping a VM during first boot (while the
|
|
||||||
encryption key is being written) can permanently corrupt the TPM. Error:
|
|
||||||
"zfs load-key: Raw key too short (expected 32)". Only fix is VM destruction
|
|
||||||
and redeployment. The 180s boot timeout in `incusos-proxmox` prevents this.
|
|
||||||
- **On physical hardware**: `force_reboot: true` is still needed (no external
|
|
||||||
orchestrator to remove install media). The bug may occur; recovery is a
|
|
||||||
manual power cycle.
|
|
||||||
|
|
||||||
### Proxmox SSH root access — strict rules
|
|
||||||
|
|
||||||
Root SSH access to the Proxmox host is available for **diagnostics only**.
|
|
||||||
The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`.
|
|
||||||
|
|
||||||
**MANDATORY RULES — violation of any rule is unacceptable:**
|
|
||||||
|
|
||||||
1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM
|
|
||||||
console screenshots via `qm monitor <vmid> screendump`. No other use
|
|
||||||
without explicit user instruction.
|
|
||||||
2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869)
|
|
||||||
or VMs in the IncusLab pool that were created by our scripts.
|
|
||||||
3. **No modifications**: NEVER run any command that modifies, stops, starts,
|
|
||||||
or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`,
|
|
||||||
`pct` commands, `zfs` commands, `systemctl` commands, or ANY write
|
|
||||||
operation on the host.
|
|
||||||
4. **Do not touch the dev VM**: the user's dev VM runs on the same host.
|
|
||||||
Do not interact with it in any way — do not even `qm status` it.
|
|
||||||
5. **No config access**: do not read or modify `/etc/pve/`, storage configs,
|
|
||||||
network configs, user/ACL settings, or any host-level configuration.
|
|
||||||
6. **Transparency**: every SSH command is visible in tool output. If the
|
|
||||||
user rejects a command, do not retry it.
|
|
||||||
7. **Only during active tests**: only SSH during test runs where test VMs
|
|
||||||
have been deployed, and only to screenshot those test VMs.
|
|
||||||
|
|
||||||
**Permitted commands (exhaustive list):**
|
|
||||||
```bash
|
```bash
|
||||||
# Take screenshot of a test VM (VMID in 850-869 range)
|
curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
|
||||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
-H "Authorization: Bearer $AWX_TOKEN"
|
||||||
root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"
|
|
||||||
|
|
||||||
# Retrieve screenshot
|
|
||||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
|
|
||||||
root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/
|
|
||||||
|
|
||||||
# Cleanup screenshot on remote
|
|
||||||
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
|
||||||
root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Technical notes:**
|
## Critical safety rules
|
||||||
- QEMU's `screendump` is root-only in PVE 9's HMP permission model.
|
|
||||||
API tokens cannot execute it regardless of privileges.
|
|
||||||
- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng,
|
|
||||||
which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`).
|
|
||||||
- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`:
|
|
||||||
`python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"`
|
|
||||||
PNG output is ~20 KB and can be read directly by Claude Code (multimodal).
|
|
||||||
|
|
||||||
## Coding conventions for scripts
|
- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`.
|
||||||
|
- **Never hard-stop VMs during first boot** -- corrupts TPM permanently.
|
||||||
|
- **Boot timeout is 180s** -- do not reduce (sysext download takes 30-120s).
|
||||||
|
- **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition.
|
||||||
|
- **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters).
|
||||||
|
|
||||||
|
## Coding conventions
|
||||||
|
|
||||||
- **Shell**: bash with `set -euo pipefail`
|
- **Shell**: bash with `set -euo pipefail`
|
||||||
- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid
|
- **Arithmetic**: `var=$((var + 1))` not `((var++))` (set -e safety)
|
||||||
false exits under `set -e` when the value is 0.
|
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use `setup_colors()` pattern
|
||||||
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern.
|
- **Flags**: both short (`-d`) and long (`--defaults`)
|
||||||
- **Flags**: support both short (`-d`) and long (`--defaults`) options.
|
- **Defaults**: useful behavior with zero flags
|
||||||
- **Defaults**: sane defaults so the script does something useful with zero flags.
|
- **Dry run**: all scripts support `--dry-run`
|
||||||
- **Dry run**: all scripts should support `--dry-run` to preview actions.
|
- **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second
|
||||||
- **Cert detection order**: files on disk first, CLI command second.
|
- **Errors**: include actionable remediation steps
|
||||||
- **Error messages**: include actionable remediation steps, not just "failed".
|
- **No hardcoded package managers**: say "install the Incus client" with a link
|
||||||
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
|
||||||
not "sudo apt install incus".
|
|
||||||
|
|
||||||
### Incus networking (OVN)
|
## Technical context loading
|
||||||
|
|
||||||
- **Bridge networks are node-local**: each cluster member has its own
|
Detailed technical context loads automatically from `.claude/rules/` based on
|
||||||
independent bridge. Instances on the same bridge (same node) can
|
which files are being edited:
|
||||||
communicate; cross-node instances CANNOT. Each bridge has the same subnet
|
|
||||||
(e.g., 10.0.0.1/24) but they are separate L2 domains.
|
|
||||||
- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes.
|
|
||||||
Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
|
|
||||||
+ client services + physical uplink network.
|
|
||||||
- **IncusOS OVN services are disabled by default**: must be enabled via the
|
|
||||||
IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring
|
|
||||||
Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection`
|
|
||||||
fails with `db.sock not found`.
|
|
||||||
- **OVN service enable API call**:
|
|
||||||
```bash
|
|
||||||
incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
|
|
||||||
"config": {
|
|
||||||
"database": "tcp:<SB_HOST>:6642",
|
|
||||||
"enabled": true,
|
|
||||||
"tunnel_address": "<THIS_NODE_LAN_IP>",
|
|
||||||
"tunnel_protocol": "geneve"
|
|
||||||
},
|
|
||||||
"state": {}
|
|
||||||
}'
|
|
||||||
```
|
|
||||||
`database` is the **southbound** DB (port 6642), NOT northbound (6641).
|
|
||||||
- **OVN control plane as container**: deploy `ovn-central` package in a
|
|
||||||
Debian container on the cluster. Use proxy devices to expose NB (6641)
|
|
||||||
and SB (6642) ports on the host's LAN IP so all nodes can reach it.
|
|
||||||
- **Setup sequence** (order matters):
|
|
||||||
1. Deploy OVN control plane container
|
|
||||||
2. Enable OVN services on ALL IncusOS nodes
|
|
||||||
3. `incus config set network.ovn.northbound_connection tcp:<host>:6641`
|
|
||||||
4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes)
|
|
||||||
5. Create physical uplink network (two-step cluster pattern)
|
|
||||||
6. Create OVN network with `--type=ovn network=UPLINK`
|
|
||||||
- **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC —
|
|
||||||
NOT `ens18`, which is the underlying device name but not exposed to Incus).
|
|
||||||
`ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses,
|
|
||||||
`ipv4.gateway` is the LAN gateway in CIDR format.
|
|
||||||
- **OVN network isolation**: multiple OVN networks are fully isolated.
|
|
||||||
Instances on different networks cannot communicate, even on the same node.
|
|
||||||
Network peering (`incus network peer create`) enables cross-network routing.
|
|
||||||
- **OVN features tested**: cross-node connectivity, network isolation,
|
|
||||||
ACLs (per-source blocking), network peering, L4 load balancers (connection-
|
|
||||||
based hashing, not round-robin), network forwards (port forwarding to LAN
|
|
||||||
IPs), DNS resolution (per-network, hostname.incus domain).
|
|
||||||
- **LB/forward backends require IP addresses**: `incus network load-balancer
|
|
||||||
backend add` and `incus network forward port add` require the target
|
|
||||||
instance's IP address, NOT its name. Using instance names fails with
|
|
||||||
"Invalid target address".
|
|
||||||
- See `notes/networking-guide.md` for full tutorial with test results.
|
|
||||||
|
|
||||||
### Shared storage (iSCSI + lvmcluster)
|
| Rule file | Loads when editing |
|
||||||
|
|---|---|
|
||||||
|
| `incusos-scripts.md` | incusos-iso, incusos-seed, incusos-proxmox, helpers |
|
||||||
|
| `proxmox-deployment.md` | incusos-proxmox, observe-deploy, helpers, examples |
|
||||||
|
| `clustering.md` | lab-test, incusos-proxmox, clustering/production guides |
|
||||||
|
| `operations-center.md` | OC guide, incusos-proxmox, OC example configs |
|
||||||
|
| `networking-storage.md` | networking, storage, migration, UTM guides |
|
||||||
|
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
||||||
|
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||||
|
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||||
|
|
||||||
- **iSCSI + lvmcluster** is the IncusOS-native path to shared storage.
|
For deep reference, see the guides in `notes/`.
|
||||||
All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS
|
|
||||||
and enabled via the REST API — no packages to install.
|
|
||||||
- **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN
|
|
||||||
field is `"target"` (NOT `"iqn"` — using the wrong field silently fails).
|
|
||||||
The config lives under `config.targets[]`, and `state` returns the
|
|
||||||
auto-generated `initiator_name`.
|
|
||||||
- **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique
|
|
||||||
`system_id` (1-2000) per node for sanlock host identification. Using 0
|
|
||||||
or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool
|
|
||||||
creation.
|
|
||||||
- **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on
|
|
||||||
custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately.
|
|
||||||
~256 MiB overhead for LVM metadata + sanlock lease area.
|
|
||||||
- **Pool creation**: two-step cluster pattern (`--target` per member, then
|
|
||||||
finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`,
|
|
||||||
and `vgchange --lock-start` automatically.
|
|
||||||
- **Migration performance** (tested on 1GbE, 2026-02-23):
|
|
||||||
- Container stop/move/start: **0.12-0.15s** (metadata only)
|
|
||||||
- VM non-live (stop/move/start): **1.8s** (LVM metadata update)
|
|
||||||
- VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer)
|
|
||||||
- Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data)
|
|
||||||
- **Proxy devices don't work for iSCSI**: SendTargets discovery returns
|
|
||||||
container IP in TargetAddress, causing portal mismatch. Use direct network
|
|
||||||
paths (bridge for same-node, macvlan for cross-node).
|
|
||||||
- **Lab target container**: Debian container with `tgt` (userspace iSCSI
|
|
||||||
target) on the cluster — no external hardware needed. Uses dual network:
|
|
||||||
bridge IP for same-node access, macvlan on `mgmt` for cross-node access.
|
|
||||||
- **First live migration after stop/start may fail**: QEMU on destination
|
|
||||||
fails to start (transient sanlock lease issue). Retry succeeds. Non-live
|
|
||||||
migration always works as fallback.
|
|
||||||
- **Hybrid architecture recommended**: local ZFS (`local` pool) for general
|
|
||||||
workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant
|
|
||||||
migration.
|
|
||||||
- See `notes/shared-storage-guide.md` for the full tested walkthrough.
|
|
||||||
|
|
||||||
### Migration into Incus
|
|
||||||
|
|
||||||
- **`incus-migrate`**: official tool for importing disk images, running
|
|
||||||
instances, or physical machines into Incus.
|
|
||||||
- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2,
|
|
||||||
raw, vdi, vhd formats. Incus accepts raw and qcow2.
|
|
||||||
- **Import workflow**: convert disk → `incus storage volume import` →
|
|
||||||
`incus init --empty --vm` → attach disk → start.
|
|
||||||
- **Container migration**: `docker export` → `incus import` for filesystem-
|
|
||||||
level container migration. Docker volumes must be copied separately.
|
|
||||||
- See `notes/migration-guide.md` for full procedures per source hypervisor.
|
|
||||||
|
|
||||||
### UTM support (future)
|
|
||||||
|
|
||||||
- Design document at `notes/utm-support.md`.
|
|
||||||
- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation
|
|
||||||
(requires AppleScript or `.utm` bundle generation).
|
|
||||||
- No `blockstat` equivalent -- install detection must use timeout + port polling.
|
|
||||||
- Seed generation already works cross-platform (Phase 3 macOS compatibility).
|
|
||||||
|
|
||||||
### AWX integration (Ansible automation for Aether)
|
|
||||||
|
|
||||||
- **AWX** is the open-source Ansible automation platform (upstream of Ansible
|
|
||||||
Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
|
|
||||||
- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
|
|
||||||
to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
|
|
||||||
- **AWX URL**: `http://192.168.102.161:30080` — exposed via K3s NodePort.
|
|
||||||
Traefik ingress returns 404 for IP-based access; use NodePort directly.
|
|
||||||
- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
|
|
||||||
`--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
|
|
||||||
- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
|
|
||||||
- **Ansible playbooks** in `ansible/` directory:
|
|
||||||
- `playbooks/post-deploy.yml` -- runs after Aether creates an instance
|
|
||||||
- `playbooks/decommission.yml` -- runs before Aether deletes an instance
|
|
||||||
- **Aether extra vars use `ffsdn_` prefix**: Aether passes `ffsdn_instance_name`,
|
|
||||||
`ffsdn_instance_ip`, `ffsdn_cluster_id`, `ffsdn_cluster_name`,
|
|
||||||
`ffsdn_deployed_by`, `ffsdn_image_os`, `ffsdn_image_release`,
|
|
||||||
`ffsdn_image_alias`. It does NOT pass `vm_name`, `vm_ip`, `environment`,
|
|
||||||
`owner`, or `cost_center` (the original plan assumed these).
|
|
||||||
- **Playbook pattern — Incus REST API, not SSH**: AWX cannot SSH to containers
|
|
||||||
on incusbr0 (bridge subnet not routable from management VLAN, IncusOS
|
|
||||||
nftables blocks inbound forwarding). Playbooks use the Incus REST API
|
|
||||||
(`uri` module with client cert) for file push + exec. The cert is at
|
|
||||||
`/runner/project/incus-client.crt` during job execution (AWX EE mounts
|
|
||||||
projects at `/runner/project/`, NOT `/var/lib/awx/projects/`).
|
|
||||||
- **Manual project (local_path)**: AWX EE containers cannot reach the private
|
|
||||||
git repo. Use `scm_type: ""` with `local_path: "incus-contrib"` and push
|
|
||||||
playbooks directly to the AWX task pod at
|
|
||||||
`/var/lib/awx/projects/incus-contrib/playbooks/`.
|
|
||||||
- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
|
|
||||||
deleted). Decommission failure does NOT block deletion.
|
|
||||||
- **Aether cluster AWX config API bug**: `PUT /api/clusters/{id}/awx-config`
|
|
||||||
returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL
|
|
||||||
UPDATE on the `clusters` table from within the Aether container.
|
|
||||||
- **Self-referencing vars cause infinite recursion in Ansible**: patterns like
|
|
||||||
`vm_ip: "{{ vm_ip | default('') }}"` cause `AnsibleUndefinedVariable`
|
|
||||||
recursive loop when the variable is not provided. Use `ffsdn_*` vars
|
|
||||||
directly without redefining them.
|
|
||||||
- **Ansible `environment` is a reserved keyword**: using it as an extra var
|
|
||||||
resolves to `[]` instead of the string value.
|
|
||||||
- See `notes/awx-guide.md` for full deployment guide and troubleshooting.
|
|
||||||
|
|
||||||
## Git workflow
|
## Git workflow
|
||||||
|
|
||||||
- Main branch: `main`
|
- **Main branch**: `master`
|
||||||
|
- **Remotes**:
|
||||||
|
- `origin`: `ssh://git@192.168.1.200:2222/maarten/incus-contrib.git` (private Gitea)
|
||||||
|
- `aether`: `_gitea@code.sovereignprivatecloud.nl:maarten/incus-contrib.git`
|
||||||
|
- Push to **both remotes** when committing
|
||||||
- Development happens on feature branches
|
- Development happens on feature branches
|
||||||
- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git`
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,175 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# proxmox-api -- Make authenticated Proxmox API calls
|
||||||
|
#
|
||||||
|
# Usage: proxmox-api METHOD PATH [CURL_ARGS...]
|
||||||
|
#
|
||||||
|
# Handles the ! in automation@pve!deploy safely (no bash history expansion).
|
||||||
|
# Config: proxmox.yaml (host, api_token_id), env file (PROXMOX_TOKEN_SECRET)
|
||||||
|
#
|
||||||
|
# Outputs JSON response on stdout.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Usage
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'EOF'
|
||||||
|
Usage: proxmox-api METHOD PATH [CURL_ARGS...]
|
||||||
|
|
||||||
|
Make authenticated Proxmox API calls with proper token handling.
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
METHOD HTTP method (GET, POST, PUT, DELETE)
|
||||||
|
PATH API path (e.g., /nodes/pve/qemu)
|
||||||
|
CURL_ARGS Additional curl arguments (e.g., -d 'key=val')
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--json Pretty-print JSON output
|
||||||
|
--help, -h Show this help
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
PROXMOX_TOKEN_SECRET API token secret (from env file)
|
||||||
|
|
||||||
|
Config:
|
||||||
|
proxmox.yaml Proxmox host + api_token_id (auto-discovered)
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
proxmox-api GET /nodes/pve/qemu
|
||||||
|
proxmox-api GET /nodes/pve/qemu/900/status/current --json
|
||||||
|
proxmox-api POST /nodes/pve/qemu/900/status/stop
|
||||||
|
proxmox-api GET /cluster/resources --json
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||||
|
usage
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Parse arguments
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
PRETTY=false
|
||||||
|
ARGS=()
|
||||||
|
for arg in "$@"; do
|
||||||
|
if [[ "$arg" == "--json" ]]; then
|
||||||
|
PRETTY=true
|
||||||
|
else
|
||||||
|
ARGS+=("$arg")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ ${#ARGS[@]} -lt 2 ]]; then
|
||||||
|
echo "Error: METHOD and PATH required. Usage: proxmox-api METHOD PATH [CURL_ARGS...]" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
METHOD="${ARGS[0]}"
|
||||||
|
API_PATH="${ARGS[1]}"
|
||||||
|
EXTRA_ARGS=("${ARGS[@]:2}")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load environment
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
load_env() {
|
||||||
|
local dir="$SCRIPT_DIR"
|
||||||
|
while [[ "$dir" != "/" ]]; do
|
||||||
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "${dir}/env"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
dir="$(dirname "$dir")"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
load_env
|
||||||
|
|
||||||
|
if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then
|
||||||
|
echo "Error: PROXMOX_TOKEN_SECRET not set. Create an env file or export it." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load proxmox.yaml
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
yaml_get() {
|
||||||
|
local file="$1" key="$2" default="${3:-}"
|
||||||
|
local val
|
||||||
|
val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val=""
|
||||||
|
val="${val#\"}" ; val="${val%\"}"
|
||||||
|
val="${val#\'}" ; val="${val%\'}"
|
||||||
|
echo "${val:-$default}"
|
||||||
|
}
|
||||||
|
|
||||||
|
find_proxmox_yaml() {
|
||||||
|
local search_dirs=(
|
||||||
|
"${SCRIPT_DIR}/.."
|
||||||
|
"${SCRIPT_DIR}/../.."
|
||||||
|
"$(pwd)"
|
||||||
|
)
|
||||||
|
for dir in "${search_dirs[@]}"; do
|
||||||
|
if [[ -f "${dir}/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/incusos/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
PROXMOX_YAML=$(find_proxmox_yaml)
|
||||||
|
if [[ -z "$PROXMOX_YAML" ]]; then
|
||||||
|
echo "Error: proxmox.yaml not found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
PVE_HOST=$(yaml_get "$PROXMOX_YAML" "host")
|
||||||
|
PVE_TOKEN_ID=$(yaml_get "$PROXMOX_YAML" "api_token_id")
|
||||||
|
|
||||||
|
if [[ -z "$PVE_HOST" ]]; then
|
||||||
|
echo "Error: proxmox.yaml: 'host' is required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [[ -z "$PVE_TOKEN_ID" ]]; then
|
||||||
|
echo "Error: proxmox.yaml: 'api_token_id' is required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Make API call
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Build the full auth header in a variable (avoids ! expansion issues)
|
||||||
|
AUTH_HEADER="Authorization: PVEAPIToken=${PVE_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
|
||||||
|
URL="https://${PVE_HOST}:8006/api2/json${API_PATH}"
|
||||||
|
|
||||||
|
CURL_CMD=(curl -fsSk -X "$METHOD" -H "$AUTH_HEADER")
|
||||||
|
|
||||||
|
if [[ ${#EXTRA_ARGS[@]} -gt 0 ]]; then
|
||||||
|
CURL_CMD+=("${EXTRA_ARGS[@]}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
CURL_CMD+=("$URL")
|
||||||
|
|
||||||
|
RESPONSE=$("${CURL_CMD[@]}" 2>&1) || {
|
||||||
|
echo "Error: API call failed: $METHOD $API_PATH" >&2
|
||||||
|
echo "$RESPONSE" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "$PRETTY" == true ]]; then
|
||||||
|
echo "$RESPONSE" | python3 -m json.tool 2>/dev/null || echo "$RESPONSE"
|
||||||
|
else
|
||||||
|
echo "$RESPONSE"
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,172 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# proxmox-screenshot -- Take a console screenshot of a Proxmox VM
|
||||||
|
#
|
||||||
|
# Usage: proxmox-screenshot VMID [OUTPUT.png]
|
||||||
|
#
|
||||||
|
# Requires: sshpass, python3-pil (for PNG conversion)
|
||||||
|
# Config: proxmox.yaml (host), env file (PROXMOX_ROOT_PASSWORD)
|
||||||
|
#
|
||||||
|
# Outputs the path to the PNG file on stdout.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Usage
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'EOF'
|
||||||
|
Usage: proxmox-screenshot VMID [OUTPUT.png]
|
||||||
|
|
||||||
|
Take a console screenshot of a Proxmox VM via QEMU monitor screendump.
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
VMID VM ID (must be in allowed range: 400-499, 800-939)
|
||||||
|
OUTPUT.png Output file path (default: /tmp/vm-VMID-screen.png)
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
PROXMOX_ROOT_PASSWORD SSH root password (from env file)
|
||||||
|
|
||||||
|
Config:
|
||||||
|
proxmox.yaml Proxmox host (auto-discovered)
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
proxmox-screenshot 900
|
||||||
|
proxmox-screenshot 900 /tmp/my-screenshot.png
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||||
|
usage
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Validate arguments
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "Error: VMID required. Usage: proxmox-screenshot VMID [OUTPUT.png]" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
VMID="$1"
|
||||||
|
OUTPUT="${2:-/tmp/vm-${VMID}-screen.png}"
|
||||||
|
|
||||||
|
# VMID safety check: only allowed ranges
|
||||||
|
if ! [[ "$VMID" =~ ^[0-9]+$ ]]; then
|
||||||
|
echo "Error: VMID must be a number" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! (( (VMID >= 400 && VMID <= 499) || (VMID >= 800 && VMID <= 939) )); then
|
||||||
|
echo "Error: VMID $VMID not in allowed range (400-499, 800-939)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load environment
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
load_env() {
|
||||||
|
local dir="$SCRIPT_DIR"
|
||||||
|
while [[ "$dir" != "/" ]]; do
|
||||||
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "${dir}/env"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
dir="$(dirname "$dir")"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
load_env
|
||||||
|
|
||||||
|
if [[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]]; then
|
||||||
|
echo "Error: PROXMOX_ROOT_PASSWORD not set. Add it to the env file." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load proxmox.yaml for host
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
yaml_get() {
|
||||||
|
local file="$1" key="$2" default="${3:-}"
|
||||||
|
local val
|
||||||
|
val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val=""
|
||||||
|
val="${val#\"}" ; val="${val%\"}"
|
||||||
|
val="${val#\'}" ; val="${val%\'}"
|
||||||
|
echo "${val:-$default}"
|
||||||
|
}
|
||||||
|
|
||||||
|
find_proxmox_yaml() {
|
||||||
|
# Search: script's grandparent (repo/incusos -> repo), script parent, cwd
|
||||||
|
local search_dirs=(
|
||||||
|
"${SCRIPT_DIR}/.."
|
||||||
|
"${SCRIPT_DIR}/../.."
|
||||||
|
"$(pwd)"
|
||||||
|
)
|
||||||
|
for dir in "${search_dirs[@]}"; do
|
||||||
|
if [[ -f "${dir}/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/incusos/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
PROXMOX_YAML=$(find_proxmox_yaml)
|
||||||
|
if [[ -z "$PROXMOX_YAML" ]]; then
|
||||||
|
echo "Error: proxmox.yaml not found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
PVE_HOST=$(yaml_get "$PROXMOX_YAML" "host")
|
||||||
|
if [[ -z "$PVE_HOST" ]]; then
|
||||||
|
echo "Error: proxmox.yaml: 'host' is required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Take screenshot
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -o LogLevel=ERROR)
|
||||||
|
REMOTE_PPM="/tmp/vm-${VMID}-screen.ppm"
|
||||||
|
LOCAL_PPM="/tmp/vm-${VMID}-screen.ppm"
|
||||||
|
|
||||||
|
# Take screenshot via QEMU monitor
|
||||||
|
if ! sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}" \
|
||||||
|
"echo 'screendump ${REMOTE_PPM}' | qm monitor ${VMID}" &>/dev/null; then
|
||||||
|
echo "Error: screendump failed (VM $VMID not running or SSH error)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Retrieve PPM file
|
||||||
|
if ! sshpass -p "$PROXMOX_ROOT_PASSWORD" scp "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}:${REMOTE_PPM}" "$LOCAL_PPM" &>/dev/null; then
|
||||||
|
echo "Error: failed to retrieve screenshot from Proxmox host" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Cleanup remote file
|
||||||
|
sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" \
|
||||||
|
"root@${PVE_HOST}" "rm -f ${REMOTE_PPM}" &>/dev/null || true
|
||||||
|
|
||||||
|
# Convert PPM -> PNG
|
||||||
|
if python3 -c "from PIL import Image; Image.open('${LOCAL_PPM}').save('${OUTPUT}')" 2>/dev/null; then
|
||||||
|
rm -f "$LOCAL_PPM"
|
||||||
|
else
|
||||||
|
# PIL not available, keep as PPM
|
||||||
|
OUTPUT="${OUTPUT%.png}.ppm"
|
||||||
|
mv "$LOCAL_PPM" "$OUTPUT" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$OUTPUT"
|
||||||
Loading…
Reference in New Issue