diff --git a/.claude/rules/awx-integration.md b/.claude/rules/awx-integration.md new file mode 100644 index 0000000..dd258a9 --- /dev/null +++ b/.claude/rules/awx-integration.md @@ -0,0 +1,56 @@ +--- +paths: + - "ansible/**" + - "incusos/deploy-awx" + - "incusos/awx-manifests/*" + - "notes/awx-guide.md" +--- + +# AWX Integration (Ansible for Aether) + +## Overview + +- AWX: open-source Ansible platform. Debian 12 VM with K3s + AWX Operator. +- Lab VM: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69). +- **AWX URL**: `http://192.168.102.161:30080` (K3s NodePort, not Traefik ingress). +- `deploy-awx` script: `--deploy`, `--status`, `--heal`, `--configure`, + `--join-aether`, `--cleanup`, `--doctor`. + +## Live job output + +Capture AWX job output while running (don't just poll status): +```bash +curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \ + -H "Authorization: Bearer $AWX_TOKEN" +``` + +## Aether extra vars + +Aether passes `ffsdn_` prefixed vars: `ffsdn_instance_name`, `ffsdn_instance_ip`, +`ffsdn_cluster_id`, `ffsdn_cluster_name`, `ffsdn_deployed_by`, `ffsdn_image_os`, +`ffsdn_image_release`, `ffsdn_image_alias`. + +It does NOT pass: `vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`. + +## Playbook pattern: Incus REST API, not SSH + +- AWX cannot SSH to containers on incusbr0 (not routable, nftables blocks). +- Use `uri` module with client cert for file push + exec. +- Cert at `/runner/project/incus-client.crt` (EE mounts at `/runner/project/`, + NOT `/var/lib/awx/projects/`). + +## Manual project (local_path) + +- EE containers can't reach private git repo. +- Use `scm_type: ""`, `local_path: "incus-contrib"`. +- Push playbooks to AWX task pod at `/var/lib/awx/projects/incus-contrib/playbooks/`. + +## Gotchas + +- **Self-referencing vars**: `vm_ip: "{{ vm_ip | default('') }}"` causes + infinite recursion. Use `ffsdn_*` vars directly. +- **`environment` is reserved**: resolves to `[]` as extra var. +- **AWX config API bug**: `PUT /api/clusters/{id}/awx-config` returns + "Invalid cluster ID". Workaround: direct PostgreSQL UPDATE. +- **Lifecycle hooks**: post-deploy failure -> auto-rollback (instance deleted). + Decommission failure does NOT block deletion. diff --git a/.claude/rules/clustering.md b/.claude/rules/clustering.md new file mode 100644 index 0000000..0a3ff47 --- /dev/null +++ b/.claude/rules/clustering.md @@ -0,0 +1,97 @@ +--- +paths: + - "incusos/lab-test" + - "incusos/incusos-proxmox" + - "notes/clustering-guide.md" + - "notes/production-lab-guide.md" +--- + +# Incus Clustering + +## Overview + +- Cluster formation via `incus` CLI using remotes. No SSH needed (IncusOS is immutable). +- No VIP needed: each node advertises its own IP. Requests forwarded internally. + +## Pre-clustering: fix core.https_address + +- Default `:8443` (wildcard). Clustering needs specific routable IP. +- Set on every node BEFORE clustering: + ```bash + incus config set : core.https_address :8443 + ``` +- Get routable IP: + ```bash + incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ + [print(a) for a in d['environment']['addresses'] \ + if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" + ``` + +## Cluster enable (init node) + +- ```bash + incus cluster enable : + ``` + TWO arguments: `:` (trailing colon) and ``. +- Generates new TLS cert. Fix remote: + ```bash + incus remote switch local + incus remote remove + incus remote add https://:8443 --accept-certificate + ``` + +## apply_defaults for clusters + +- **Bootstrap node**: `apply_defaults: true` (needs pool + network) +- **Joining nodes**: `apply_defaults: false` (recommended, clean join) +- **Standalone**: `apply_defaults: true` +- If `apply_defaults: true` on joiners, 8-command cleanup needed before join + (delete pool, network, volumes, profile devices). + +## Join workflow + +- **Token**: `incus cluster add :` (ONE arg) +- **Join**: `incus cluster join : :` (TWO args) +- **Automated**: + ```bash + printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : + ``` +- After join: fix remote (same as init). + +## Command syntax gotchas + +- `cluster enable remote: member-name` -- TWO args +- `cluster add remote:member-name` -- ONE arg +- `cluster remove remote:member-name --force` -- ONE arg, prompts "yes/no" +- `cluster evacuate remote:member-name` -- ONE arg +- `cluster restore remote:member-name` -- ONE arg +- `cluster join init-remote: joining-remote:` -- TWO args +- `config set remote: key value` -- remote with trailing colon + space +- General: `remote:resource` for resource, `remote:` for server itself + +## Workload placement and migration + +- **Targeted launch**: `incus launch images:debian/12 :name --target ` +- **Container migration**: stop/move/start only (no CRIU). +- **VM live migration**: requires `migration.stateful=true` (set while stopped). +- **limits.cpu MUST be a range** (e.g., `0-1`), not integer. Integer causes + `maxcpus` mismatch -> `Missing section footer for ICH9LPC`. +- **size.state required**: `size.state=2GiB` on root disk for stateful ops. +- **Agent reconnect**: sleep 4s after migration before `incus exec`. +- **Evacuation**: `incus cluster evacuate : --force` + Use `--action stop` without limits.cpu range fix. + +## Cluster rebalancing + +```bash +incus config set : cluster.rebalance.interval=1 +incus config set : cluster.rebalance.threshold=10 +incus config set : cluster.rebalance.batch=2 +incus config set : cluster.rebalance.cooldown=5m +``` + +## Lab validation (lab-test) + +- Reads same YAML config as `incusos-proxmox`. +- Phases: deploy, single, cluster, workload, migrate. +- Test instances: `test-*` prefix. Reports PASS/FAIL/SKIP. diff --git a/.claude/rules/incusos-scripts.md b/.claude/rules/incusos-scripts.md new file mode 100644 index 0000000..cff7f16 --- /dev/null +++ b/.claude/rules/incusos-scripts.md @@ -0,0 +1,50 @@ +--- +paths: + - "incusos/incusos-iso" + - "incusos/incusos-seed" + - "incusos/incusos-proxmox" + - "incusos/helpers/*" + - "incusos/examples/*" +--- + +# IncusOS Scripts Context + +## Incus version differences + +- **Debian stable ships Incus 6.0 LTS** (behind upstream). + Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu. +- **macOS (Homebrew)** and **Arch Linux** track latest upstream (currently 6.21). + macOS is client-only; Arch has no client-only split package. +- `incus remote get-client-certificate` added in **6.3+** -- does not exist + in 6.0 LTS. Scripts must never depend on it as the only cert path. +- Always prefer reading `~/.config/incus/client.crt` from disk first. +- See `notes/incus-version-compatibility.md` for full matrix. + +## IncusOS flasher-tool + +- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest` +- Flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version` +- There is NO `--seed-tar` flag -- it's `--seed` (or `-s`). +- There is NO `--arch` flag -- architecture from downloaded image. +- CDN index: `https://images.linuxcontainers.org/os/index.json` +- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz` + +## Seed archives + +- Tar archives with YAML files at root level. +- Written to byte offset 2148532224 (seed partition) in the image. +- External boot media labeled `SEED_DATA` as CD-ROM: + - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM. + - **FAT image**: works for USB but NOT CD-ROM (kernel sr_mod doesn't expose + FAT labels). +- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, + `operations-center.yaml`, `network.yaml`, `update.yaml`. + +## Client certificates + +- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. +- `incus remote list` triggers auto-generation if no keypair exists. +- Incus seed: `preseed.certificates[]` (NOT `preseed.server.certificates[]`). + `InitPreseed.Server` uses `yaml:",inline"` so fields are promoted to top level. +- OC seed: `trusted_client_certificates[]`. +- OC **requires** at least one trusted cert or you're locked out. diff --git a/.claude/rules/lab-infrastructure.md b/.claude/rules/lab-infrastructure.md new file mode 100644 index 0000000..63e996e --- /dev/null +++ b/.claude/rules/lab-infrastructure.md @@ -0,0 +1,35 @@ +--- +paths: + - "incusos/incusos-proxmox" + - "incusos/lab-test" + - "incusos/examples/*" +--- + +# Lab Infrastructure + +## Multi-lab coexistence + +- `--lab-down config.yaml`: stops VMs (stay on disk). +- `--lab-up config.yaml`: starts stopped VMs (refuses if media attached). +- `--cleanup`: destroys permanently. These are distinct operations. +- `--resources`: host RAM/CPU/storage + per-pool allocation (API only). +- `--labs`: scans pool for `[incusos-lab:managed]` VMs, groups by config. +- `--lab-up` auto-deploy: offers full pipeline if no VMs exist yet. + +## VMID range convention + +| Range | Lab | +|---------|--------------------------------| +| 400-499 | OC-managed nodes | +| 800-809 | Single-node labs | +| 900-909 | Basic cluster | +| 910-919 | OC combined (server + nodes) | +| 920-929 | OC server standalone | +| 930-939 | Advanced / heterogeneous | + +## Resource constraints + +- **RAM is the bottleneck**: 4 GiB/VM minimum. 3-node = 12 GiB, OC lab = 28 GiB. +- **Storage**: thin provisioned (ZFS sparse). 3x 50 GiB = ~7-8 GiB actual. +- **CPU**: 4 cores/VM, 20 on host. Not a constraint. +- Pre-deploy checks warn if RAM exceeds available (API method). diff --git a/.claude/rules/networking-storage.md b/.claude/rules/networking-storage.md new file mode 100644 index 0000000..3aa7af2 --- /dev/null +++ b/.claude/rules/networking-storage.md @@ -0,0 +1,58 @@ +--- +paths: + - "notes/networking-guide.md" + - "notes/shared-storage-guide.md" + - "notes/migration-guide.md" + - "notes/utm-support.md" +--- + +# Networking, Storage, and Migration + +## OVN networking + +- **Bridge networks are node-local**: same subnet but separate L2 domains. +- **OVN**: cross-node L2 overlay via Geneve tunnels. Sub-ms latency. +- **IncusOS OVN services disabled by default**: enable via `/os/1.0/services/ovn` on EVERY node. +- OVN service enable: + ```bash + incus query :/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp::6642", + "enabled": true, + "tunnel_address": "", + "tunnel_protocol": "geneve" + }, "state": {} + }' + ``` + `database` is **southbound** (6642), NOT northbound (6641). +- **Setup sequence**: deploy OVN container -> enable services -> set northbound + connection -> add ovn-chassis roles -> create uplink -> create OVN network. +- **Uplink**: `parent=mgmt` (NOT `ens18`). `ipv4.ovn.ranges` + `ipv4.gateway`. +- **LB/forward backends need IP addresses**, not instance names. +- See `notes/networking-guide.md` for full tutorial. + +## Shared storage (iSCSI + lvmcluster) + +- IncusOS-native: iSCSI initiator, lvmlockd, sanlock built in. +- **iSCSI API**: `/os/1.0/services/iscsi`. Field is `"target"` (NOT `"iqn"`). +- **LVM API**: `/os/1.0/services/lvm`. `system_id` must be 1-2000 (not 0). +- **lvmcluster**: thick provisioning. Two-step cluster pattern for pool creation. +- **Migration perf** (1GbE): container 0.15s, VM non-live 1.8s, VM live ~6s. +- **Proxy devices don't work for iSCSI** (portal mismatch). Use direct paths. +- **First live migration after stop/start may fail** (transient sanlock). Retry works. +- **Hybrid recommended**: local ZFS + shared lvmcluster for HA VMs. +- See `notes/shared-storage-guide.md` for full walkthrough. + +## Migration into Incus + +- `incus-migrate`: official import tool. `qemu-img convert` for format conversion. +- Workflow: convert disk -> `incus storage volume import` -> `incus init --empty --vm` + -> attach -> start. +- Docker: `docker export` -> `incus import`. +- See `notes/migration-guide.md` for per-hypervisor procedures. + +## UTM support (future) + +- Design doc at `notes/utm-support.md`. +- `utmctl` for start/stop but not VM creation (AppleScript or `.utm` bundle). +- No `blockstat` equivalent -- timeout + port polling for install detection. diff --git a/.claude/rules/operations-center.md b/.claude/rules/operations-center.md new file mode 100644 index 0000000..669f1dc --- /dev/null +++ b/.claude/rules/operations-center.md @@ -0,0 +1,64 @@ +--- +paths: + - "notes/operations-center-guide.md" + - "incusos/incusos-proxmox" + - "incusos/examples/*oc*" +--- + +# Operations Center + +## Basics + +- **CLI**: `operations-center` (from GitHub releases or `github.com/FuturFusion/operations-center`). +- **Config**: `~/.config/operations-center/` (copy certs from `~/.config/incus/`). +- **Port**: 8443. Browser needs PKCS#12 client cert (`client.pfx`). Web UI at `/ui/`. +- **v0.3.0** (active dev). Use `--version` to check. +- **No `remote:` suffix**: requires `operations-center remote switch NAME`. + +## Provisioning workflow + +- **No brownfield**: nodes must boot from OC-provisioned ISO. +- **Token seeds**: `provisioning token seed add/get-image`. No `force_reboot` for Proxmox. + ```bash + operations-center provisioning token seed add proxmox-preseed \ + /tmp/preseed.yaml --description "No force_reboot for Proxmox" + operations-center provisioning token seed get-image proxmox-preseed \ + /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable + ``` +- **Hybrid deploy** (recommended): + `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml` +- **Self-registration**: nodes register with OC in ~30s after first boot. + +## needs_update blocker (critical) + +- OC requires `needs_update: false` for `provisioning cluster add`. +- Nodes from latest ISO are `needs_update: true` (never went through OC pipeline). +- **Fix**: use `--channel old-stable` for ISO. OC pushes update, clears flag. + +## Cluster formation + +```bash +echo '{}' > /tmp/oc-app-config.yaml +operations-center provisioning cluster add oc-cluster \ + https://:8443 \ + --server-names oc-node-01,oc-node-02,oc-node-03 \ + --server-type incus \ + --application-seed-config /tmp/oc-app-config.yaml +``` +- Use `apply_defaults: false` for OC nodes (recommended). +- Empty `{}` app config if cert already in trust store from SEED_DATA. + +## Tested limitations + +- Inventory NOT real-time -- requires `cluster resync` +- OC reboot breaks OC-managed Proxmox nodes (daemon crash from invalid state.txt) + - Fix: destroy and redeploy. Proxmox stop/start is safe. +- No cluster member state tracking (always shows `ready`) +- Stale entries from out-of-band changes persist +- Server removal blocked if part of OC cluster +- `needs_update` is pipeline-based, not version-based +- OVN LB has no health checks + +## ISO upload skip bug (fixed) + +- `--iso` now deletes existing ISO before re-upload (prevents stale reuse). diff --git a/.claude/rules/proxmox-deployment.md b/.claude/rules/proxmox-deployment.md new file mode 100644 index 0000000..c8683e2 --- /dev/null +++ b/.claude/rules/proxmox-deployment.md @@ -0,0 +1,104 @@ +--- +paths: + - "incusos/incusos-proxmox" + - "incusos/observe-deploy" + - "incusos/helpers/*" + - "incusos/examples/*" + - "incusos/TESTING.md" +--- + +# Proxmox VE Deployment + +## incusos-proxmox overview + +- Reads YAML config, generates per-VM SEED_DATA via `incusos-seed --format iso`, + uploads ISO + seeds to Proxmox, creates VMs, boots through installation. +- **Config merge**: `proxmox.yaml` base -> lab config `proxmox:` overlay -> CLI flags. + Auto-discovers `proxmox.yaml` in script dir then cwd; override with `--proxmox FILE`. +- **API token**: `env` file at repo root (gitignored). Auto-loaded on startup. +- **Methods**: SSH (default) or API (`curl -k https://host:8006/api2/json/...`). + +## Required VM settings (wrong = install failure) + +- `bios=ovmf`, `machine=q35` -- UEFI required +- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls own Secure Boot keys +- `tpmstate0`: `version=v2.0` -- required for disk encryption +- `cpu=host` -- x86_64_v3 instruction set requirement +- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk broken with IncusOS +- `balloon=0` -- IncusOS manages memory internally +- `ide3` -- SEED_DATA ISO 9660 as second CD-ROM +- Minimum 50 GiB disk, 4096 MiB RAM + +## Install flow (automated) + +1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed +2. IncusOS installs to scsi0 (~876 MiB), sits at "remove media" prompt +3. Detect completion: poll `blockstat.scsi0.wr_bytes` -- writes start then stop +4. Stop VM (Proxmox stop) +5. **Delete ide2 and ide3** -- IncusOS refuses to start if media present +6. Set `boot: order=scsi0`, start from disk + +## IP detection + +- **Static IP** (preferred): `ip: ADDR/PREFIX` per VM + `gateway:` + `dns:`. +- **ARP-based** (DHCP fallback): MAC from VM config -> ARP table. Same L2 only. + +## VLAN tagging + +- `vlan:` config key adds tag to NIC (`tag=69`). Current lab: VLAN 69 (192.168.100.0/22). + +## Disk target + +- Do NOT specify `disk-target` for Proxmox VMs. IncusOS does **literal** string + matching. `scsi-*` does NOT match the actual device ID. Omit and let auto-detect. + +## Resource pool isolation + +- `proxmox.pool` scopes VM operations to a Proxmox pool. API token ACL on `/pool/`. + +## Reconcile and idempotency + +- Re-runs with existing VMs: interactive menu (status/continue/destroy/abort). +- `--yes` defaults to status checks (never auto-destroys). +- `phase_install` checks each VM state before acting. + +## Doctor and cleanup + +- `--doctor`: environment check, no config needed. +- `--cleanup`: destroys config VMs. `--cleanup --deep`: also ISOs + remotes. +- `--cleanup-all`: pool-wide. `--cleanup-all --deep`: all IncusOS ISOs. +- `--verbose`/`-v`, `--quiet`, `--retries N` (default 3). + +## Boot timeout + +- **180s** (60s sleep + 120s poll). First boot downloads sysext (~30-120s). + Do NOT reduce -- premature retries can corrupt TPM encryption key permanently. + +## Disk resize + +- Proxmox `qm resize` only grows. Partition 11 auto-expands via `systemd-repart`. +- ZFS pool does NOT auto-expand. Manual expansion via privileged container needed. + See `notes/aether-guide.md` for procedure. + +## First-boot sequence + +1. ISO boot + install (~60-85s) +2. Transition: detect complete, stop, remove media, start (~15s) +3. First disk boot: encryption key, sysext download, "System ready" (~50s) +4. **Total: ~130-215s** to port 8443 + +## Crontab bug (issue #843) + +- **Eliminated** by omitting `force_reboot`. Upstream bug only with `force_reboot: true`. +- Empty `ScrubSchedule` crashes daemon. `fix_scrub_schedule()` auto-heals. +- **TPM corruption risk**: hard-stop during first boot permanently corrupts. + 180s timeout prevents this. + +## API privileges (role: IncusOSDeployer) + +``` +VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory +VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType +VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate +Datastore.Audit SDN.Use Sys.Audit +``` diff --git a/.claude/rules/proxmox-ssh-rules.md b/.claude/rules/proxmox-ssh-rules.md new file mode 100644 index 0000000..5aa54fe --- /dev/null +++ b/.claude/rules/proxmox-ssh-rules.md @@ -0,0 +1,39 @@ +# Proxmox SSH Root Access -- STRICT SAFETY RULES + +Root SSH to Proxmox host is for **diagnostics only**. +Password: `PROXMOX_ROOT_PASSWORD` in `env` file. + +## MANDATORY RULES -- violation is unacceptable + +1. **Screenshots only**: ONLY use SSH for `qm monitor screendump`. + No other commands without explicit user instruction. +2. **Test VMs only**: screenshot VMs in VMID range 400-499 or 800-939, + or VMs in the IncusLab pool created by our scripts. +3. **No modifications**: NEVER `qm set/stop/start/destroy`, `pct`, + `zfs`, `systemctl`, or ANY write operation on the host. +4. **Do not touch the dev VM**: user's dev VM is on the same host. + Do not interact with it in any way. +5. **No config access**: do not read/modify `/etc/pve/`, storage/network + configs, user/ACL settings, or host-level configuration. +6. **Transparency**: every SSH command is visible. If user rejects, stop. +7. **Only during active tests**: only when test VMs are deployed. + +## Permitted commands (exhaustive) + +Use `incusos/helpers/proxmox-screenshot VMID` instead of raw SSH. + +Manual fallback (only if helper unavailable): +```bash +sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ + root@ "echo 'screendump /tmp/vm--screen.ppm' | qm monitor " +sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \ + root@:/tmp/vm--screen.ppm /tmp/ +sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ + root@ "rm -f /tmp/vm--screen.ppm" +``` + +## Technical notes + +- `screendump` is root-only in PVE 9 HMP. API tokens cannot do it. +- Use `.ppm` format -- PVE 9.1 lacks libpng for PNG screendump. +- PPM ~3 MB -> PNG ~20 KB via `python3 -c "from PIL import Image; ..."`. diff --git a/.claude/skills/proxmox-api.md b/.claude/skills/proxmox-api.md new file mode 100644 index 0000000..82a5e61 --- /dev/null +++ b/.claude/skills/proxmox-api.md @@ -0,0 +1,34 @@ +--- +name: proxmox-api +description: Make an authenticated Proxmox API call +user_invocable: true +arguments: + - name: args + description: "METHOD PATH [--json]" + required: true +--- + +# Proxmox API Skill + +Make an authenticated Proxmox API call with proper token handling. + +## Steps + +1. Run the helper script: + ```bash + /home/maarten/dev/incus-contrib/incusos/helpers/proxmox-api {{args}} + ``` +2. Parse the JSON response +3. Present the results in a readable format + +## Notes + +- The helper handles the `!` in `automation@pve!deploy` safely +- Reads host and token ID from `proxmox.yaml`, secret from env file +- Add `--json` for pretty-printed output +- Common paths: + - `/nodes/pve/qemu` -- list all VMs + - `/nodes/pve/qemu/{vmid}/status/current` -- VM status + - `/nodes/pve/status` -- host resources + - `/cluster/resources` -- cluster-wide resources + - `/pools/{pool}` -- pool members diff --git a/.claude/skills/screenshot.md b/.claude/skills/screenshot.md new file mode 100644 index 0000000..ad74984 --- /dev/null +++ b/.claude/skills/screenshot.md @@ -0,0 +1,30 @@ +--- +name: screenshot +description: Take a console screenshot of a Proxmox VM and view it +user_invocable: true +arguments: + - name: vmid + description: The VM ID to screenshot + required: true +--- + +# Screenshot Skill + +Take a console screenshot of a Proxmox VM and display it. + +## Steps + +1. Run the helper script to capture the screenshot: + ```bash + /home/maarten/dev/incus-contrib/incusos/helpers/proxmox-screenshot {{vmid}} + ``` +2. Read the output path from stdout +3. Use the Read tool to view the PNG image +4. Describe what the VM console shows (boot stage, errors, prompts, etc.) + +## Notes + +- The helper handles SSH, PPM capture, and PNG conversion automatically +- Requires `PROXMOX_ROOT_PASSWORD` in env file and `proxmox.yaml` for host +- Only works for VMIDs in allowed ranges (400-499, 800-939) +- Use this PROACTIVELY during deploy scenarios to monitor boot progress diff --git a/CLAUDE.md b/CLAUDE.md index c3ff078..d657382 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,858 +9,104 @@ Primarily targeting home lab environments but aiming for production-quality scri ## Repository structure ``` -incu-contrib/ -├── CLAUDE.md # This file -- project context +incus-contrib/ +├── CLAUDE.md # This file -- behavioral rules + project overview +├── .claude/rules/ # Topic-specific context (auto-loaded by paths:) +├── .claude/skills/ # Slash commands: /screenshot, /proxmox-api ├── README.md # Main overview ├── .gitignore ├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks -│ ├── ansible.cfg # Project-level Ansible config +│ ├── ansible.cfg │ └── playbooks/ -│ ├── post-deploy.yml # Runs after Aether creates an instance (Incus API) +│ ├── post-deploy.yml # Runs after Aether creates an instance │ └── decommission.yml # Runs before Aether deletes an instance ├── incusos/ # IncusOS installation tooling -│ ├── README.md # Detailed usage docs +│ ├── README.md │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) -│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) +│ ├── incusos-seed # Seed archive generator (Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── deploy-awx # AWX deployment + management on Incus cluster │ ├── awx-manifests/ # K8s manifests for AWX Operator + instance +│ ├── helpers/ +│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot +│ │ └── proxmox-api # Authenticated API calls (handles ! in token) │ ├── lab-test # Guided lab validation (12 test phases) -│ ├── observe-deploy # Single-VM deploy with rapid console screenshots -│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) -│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) -│ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env -│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test +│ ├── observe-deploy # Single-VM deploy with console screenshots +│ ├── proxmox.yaml # Proxmox connection (gitignored) +│ ├── TESTING.md │ └── examples/ # Example seed + Proxmox YAML files -└── notes/ # Research notes and reference material - ├── clustering-guide.md # Detailed Incus clustering walkthrough - ├── operations-center-guide.md # Operations Center provisioning & management - ├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN) - ├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested) - ├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end) - ├── migration-guide.md # Migration paths into Incus from other hypervisors - ├── aether-guide.md # Aether management platform (deploy, blueprints, API) - ├── awx-guide.md # AWX + Aether Ansible automation - ├── incus-version-compatibility.md # Incus versions across platforms - ├── iso-download-methods.md # ISO download/customization research - └── utm-support.md # UTM support design document (future) +├── env # PROXMOX_TOKEN_SECRET + PROXMOX_ROOT_PASSWORD (gitignored) +└── notes/ # Reference guides (clustering, networking, OC, AWX, etc.) ``` -## Key technical context +## Capabilities you have -### Incus version differences +### Proxmox VM screenshots +Take console screenshots during deploys to see boot progress or errors: +``` +incusos/helpers/proxmox-screenshot VMID [/tmp/output.png] +``` +Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during +deploy scenarios to monitor boot progress, diagnose hangs, and verify installs. -- **Debian stable ships Incus 6.0 LTS** which is significantly behind upstream. - The Zabbly repo (https://github.com/zabbly/incus) provides latest on Debian/Ubuntu. -- **macOS (Homebrew)** and **Arch Linux** both track latest upstream (currently 6.21). - macOS is client-only by design; Arch has no client-only split package. -- `incus remote get-client-certificate` was added in **Incus 6.3+** and does - not exist in 6.0 LTS. Scripts must never depend on it as the only cert path. -- Always prefer reading `~/.config/incus/client.crt` directly from disk. - Fall back to the CLI command only as a secondary option. -- See `notes/incus-version-compatibility.md` for full platform matrix and - install instructions. +### Proxmox API calls +Make authenticated API calls (handles the `!` in `automation@pve!deploy`): +``` +incusos/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json +incusos/helpers/proxmox-api GET /nodes/pve/status --json +incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop +``` -### IncusOS flasher-tool - -- Install: `go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest` -- Actual CLI flags: `-f/--format`, `-s/--seed`, `-c/--channel`, `-i/--image`, `-v/--version` -- There is NO `--seed-tar` flag -- it's just `--seed` (or `-s`). -- There is NO `--arch` flag -- architecture is determined by the downloaded image. - For cross-arch builds, download the image manually and pass via `--image`. -- CDN index: `https://images.linuxcontainers.org/os/index.json` -- CDN images: `https://images.linuxcontainers.org/os/{version}/{arch}/IncusOS_{version}.{format}.gz` - -### Seed archives - -- Tar archives containing YAML files at the root level. -- Written to byte offset 2148532224 (the seed partition) in the image. -- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM. - - **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM - devices. Volume labels are correctly detected by the kernel on `/dev/sr*`. - - **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but - **NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose - FAT filesystem labels, so IncusOS cannot find the seed. -- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, - `operations-center.yaml`, `network.yaml`, `update.yaml`. - -### Client certificates - -- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. -- Running `incus remote list` triggers auto-generation if no keypair exists. -- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`). - The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including - `certificates`) are promoted to the top level of the `preseed` object. -- For Operations Center seed: injected under `trusted_client_certificates[]`. -- Operations Center **requires** at least one trusted certificate -- without it, - you are locked out after installation. - -### Proxmox VE deployment - -- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images - via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates - VMs with IncusOS-correct settings, and boots them through installation. -- **Config separation**: Proxmox connection settings (host, credentials, pool) - live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`, - etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config - `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks - for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`. -- **API token secret**: stored in `env` file at the repo root (gitignored). - Scripts auto-load it on startup by searching for `env` up the directory - tree from the script location. No manual `source env` needed. If the file - is missing and `PROXMOX_TOKEN_SECRET` is not exported, API commands fail - with a clear error message. -- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API - (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). -- **Minimum API privileges** for token-based access (role: `IncusOSDeployer`): - ``` - VM.Allocate VM.Config.Disk VM.Config.CPU VM.Config.Memory - VM.Config.Network VM.Config.CDROM VM.Config.Options VM.Config.HWType - VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.AllocateTemplate - Datastore.Audit SDN.Use Sys.Audit - ``` - `Sys.Audit` is needed for `--resources` (host RAM/CPU/uptime via - `/nodes//status`). It requires a separate ACL on `/nodes/` - since the pool-scoped ACL doesn't cover node-level endpoints. -- **Required VM settings** (getting any wrong causes IncusOS install failure): - - `bios=ovmf`, `machine=q35` -- UEFI boot required - - `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys - - `tpmstate0`: `version=v2.0` -- required for disk encryption - - `cpu=host` -- needed for x86_64_v3 instruction set requirement - - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - - `balloon=0` -- IncusOS manages memory internally - - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - - Minimum 50 GiB disk, minimum 4096 MiB RAM -- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag - to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM - on the tagged VLAN instead of the native/untagged network. The current lab - uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, - VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the - Proxmox VM level only — IncusOS and Incus instances inside the VM are - unaware of it. The setting is optional: omit `vlan:` for untagged access. -- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. - IncusOS does **literal** string matching (not glob) on disk device IDs. - `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit - `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). -- **Install flow** (automated by `incusos-proxmox`): - 1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed - 2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), - then sits at "please remove installation media" prompt - 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- - when disk writes start then stop for 15s (3 stable polls), install is done - 4. Stop the VM (Proxmox stop, not guest shutdown) - 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot - and refuses to start if found, regardless of boot order - 6. Set boot order to `order=scsi0` and start from disk -- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two - strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM - in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's - `network.yaml` configures the VM's interface at boot. No ARP scan or - SSH needed — the IP is known at deploy time. Works across VLANs. - (2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM - config → flush stale ARP → ping broadcast → look up MAC in ARP table. - Only works on the same L2 domain (native LAN, not across VLANs). -- **force_reboot is NOT used on Proxmox**: the seed omits `force_reboot`. - IncusOS sits at "please remove installation media" after install. We - detect completion via blockstat (876 MiB written, then idle), stop the - VM externally, remove media, and start from disk. This avoids the - crontab race condition (issue #843). On physical hardware, - `force_reboot: true` is still needed (no external orchestrator). -- **Resource pool isolation**: the optional `proxmox.pool` config field scopes - all VM operations to a Proxmox resource pool. When set, the script only - "sees" VMs in that pool (for collision detection and cleanup), and the API - token ACL can be scoped to `/pool/` instead of `/`. Setup: - ``` - pveum pool add IncusLab --comment "IncusOS Lab VMs" - pveum pool modify IncusLab --storage local-lvm,local - pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer - pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0 - ``` -- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM - deployment status (Proxmox state, install status, IP, port 8443, incus - remote). Also runs post-deployment checks (Incus connectivity, Operations - Center URL). -- **Reconcile on re-runs**: when `--phase all` detects existing VMs from - config, an interactive menu offers: (1) run status checks, (2) continue - install for incomplete VMs, (3) destroy and redeploy, (4) abort. With - `--yes`, defaults to option 1 (safe -- never auto-destroys). -- **Install idempotency**: `phase_install` checks each VM's state before - acting -- already-running VMs are skipped, stopped-but-installed VMs are - started from disk, only VMs with install media proceed through installation. - -### Multi-lab coexistence - -- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop, - VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk - (refuses to start VMs with install media still attached). These are - distinct from `--cleanup` (which destroys VMs permanently). -- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage - usage, and per-pool allocation. Requires API method. -- **Lab inventory**: `--labs` scans the pool for managed VMs (by - `[incusos-lab:managed]` marker), groups by config file, and shows - per-lab status, VM count, RAM, and disk. -- **VMID range convention** (to avoid collisions between coexisting labs): - - | Range | Lab | - |-------|-----| - | 400-499 | OC-managed nodes | - | 800-809 | Single-node labs | - | 900-909 | Basic cluster | - | 910-919 | OC combined (server + nodes) | - | 920-929 | OC server standalone | - | 930-939 | Advanced / heterogeneous | - -### Resource constraints for multi-lab - -- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node - cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource - where you can actually run out. -- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default - (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. - LZ4 compression provides ~1.5-1.8x ratio on OS data. -- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs - can share CPUs without contention. -- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if - requested RAM exceeds available host RAM (API method only). -- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to - run the full deploy pipeline (auto-accepts with `--yes`). -- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs - (via storage content API) and notes thin provisioning for ZFS/LVM-thin. - -### Incus clustering via remotes - -- **Cluster formation** is done entirely through the `incus` CLI using remotes. - No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access). -- **No VIP needed**: each node advertises its own IP as its cluster address. - Clients can connect to any cluster member; requests are forwarded internally. - -#### Pre-clustering: fix core.https_address - -- IncusOS nodes default to `core.https_address: :8443` (wildcard / all - interfaces). Clustering requires a **specific routable IP** so nodes can - address each other. -- **Set the IP on every node BEFORE enabling clustering:** - ```bash - incus config set : core.https_address :8443 - ``` -- Get each node's routable IP via the API: - ```bash - incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ - [print(a) for a in d['environment']['addresses'] \ - if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" - ``` -- This is safe to do while remotes are connected -- the remote already points - to the specific IP; we're just narrowing the bind address. Certificate trust - is fingerprint-based, not address-based. - -#### Cluster enable (init node) - -- ```bash - incus cluster enable : - ``` - Note: this is TWO arguments: `:` (trailing colon) and ``. - The help text shows `[:] ` — NOT `remote:name` as a single arg. -- **TLS certificate regeneration**: enabling clustering causes the server to - generate a new TLS certificate (cluster cert). The new cert may only have - SANs for `127.0.0.1` and `::1`, breaking the existing remote. -- **Fix**: remove and re-add the remote to pin the new certificate: - ```bash - incus remote switch local # if init remote is current default - incus remote remove - incus remote add https://:8443 --accept-certificate - ``` -- The cert trust on the server side (client → server) is unaffected -- it's - stored by fingerprint in the Incus database, independent of listen address. - -#### Joining nodes: apply_defaults and the storage pool conflict - -- **Upstream recommendation**: use `apply_defaults: false` for nodes destined - to join a cluster. The official IncusOS clustering tutorial states joining - servers "cannot have preexisting networks or storage pools defined." With - `apply_defaults: false`, the node still listens on port 8443, still trusts - preseed certificates, and the underlying ZFS dataset (`local/incus`) still - exists -- but no Incus storage pool or network metadata is created, so - the join process works cleanly. -- **`apply_defaults: true` on joining nodes** is also functional but requires - an 8-command cleanup per node before join (delete pool, network, volumes, - profile devices). This is automated in `lab-test` but adds complexity. -- **Recommended seed pattern for clusters**: - - Bootstrap/init node: `apply_defaults: true` (needs pool and network) - - Joining nodes: `apply_defaults: false` (join process creates member-specific entries) - - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional) -- **If `apply_defaults: true` was used**, the cleanup before join is: - ```bash - # 1. Remove config references - incus config unset : storage.backups_volume - incus config unset : storage.images_volume - # 2. Delete volumes - incus storage volume delete :local backups - incus storage volume delete :local images - # 3. Clear default profile references (pool is "in use" otherwise) - incus profile device remove :default root - incus profile device remove :default eth0 - # 4. Delete pool and network - incus storage delete :local - incus network delete :incusbr0 - ``` - -#### Join workflow - -- **Generate token** (on init node, single argument `remote:member-name`): - ```bash - incus cluster add : - ``` -- **Join** (interactive -- prompts for 5 values): - ```bash - incus cluster join : : - ``` - Interactive prompts and correct answers: - 1. IP address → accept default (node's IP, already set via core.https_address) - 2. Member name → accept default (matches the token) - 3. "All existing data is lost" → `yes` - 4. `source` property for storage pool "local" → `local/incus` - 5. `zfs.pool_name` property for storage pool "local" → `local/incus` -- **Automated (non-interactive):** - ```bash - printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : - ``` -- **After join**: the joining node gets a new cluster certificate. Fix the - remote (same as init node): - ```bash - incus remote remove - incus remote add https://:8443 --accept-certificate - ``` - -#### Command syntax gotchas - -- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name) -- `incus cluster add remote:member-name` -- ONE argument (no space) -- `incus cluster remove remote:member-name --force` -- ONE argument; prompts - "yes/no" even with `--force`, pipe `printf "yes\n"` for automation -- `incus cluster evacuate remote:member-name` -- ONE argument (no space) -- `incus cluster restore remote:member-name` -- ONE argument (no space) -- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space) -- `incus storage show remote:pool` -- ONE argument (no space) -- `incus storage show remote:pool --target member` -- target flag for - member-specific config -- `incus config set remote: key value` -- remote with trailing colon + space -- General rule: `remote:resource` for targeting a resource, `remote:` (trailing - colon) for targeting the server itself - -#### Post-join state - -- After joining, the cluster is managed through the init node's remote. The - individual node remotes still work for node-specific operations. -- **`lab-test`** automates cluster formation, workload testing, and migration. - -#### Workload placement and migration - -- **Targeted launch**: `incus launch images:debian/12 :name --target ` -- **Cluster-wide visibility**: `incus list` on any member shows all instances. -- **Container migration**: stop/move/start only (CRIU live migration is - unreliable). Data persists, processes do not. - ```bash - incus stop : - incus move : --target - incus start : - ``` -- **VM live migration**: requires `migration.stateful=true` (must be set - while VM is stopped). Preserves running state with no downtime. - ```bash - incus move : --target - ``` -- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`), - not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's - CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`). - Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays - differently, causing `Missing section footer for ICH9LPC` on restore. - Using a range (pinning syntax) eliminates `maxcpus` entirely and uses - fixed `sockets/cores/threads` topology — portable across all hosts. - ```bash - # WRONG: integer → maxcpus varies by host → migration fails - incus config set limits.cpu=2 - # RIGHT: range → fixed topology → migration works everywhere - incus config set limits.cpu=0-1 - ``` -- **VM live migration works in nested virtualization** (IncusOS inside - Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU - 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). -- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that - appears during migration is cosmetic. It fires from QEMU's feature - dependency checker before KVM filters out unsupported features and does - not affect migration. -- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also - requires the `limits.cpu` range fix. Use `incus start --stateless` to - discard a saved state file that cannot be restored. -- **VM `size.state` config**: stateful operations require `size.state` on - the root disk (`incus config device add root disk path=/ - pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails. -- **Cluster evacuation**: `incus cluster evacuate : --force` - (ONE argument, like `cluster enable` and `cluster add`). - Use `--action stop` if VMs lack the `limits.cpu` range fix. - Restore with `incus cluster restore : --force`. -- **VM agent reconnect**: after live migration, the incus agent inside the VM - needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately - after migration may fail with "VM agent isn't currently running". Scripts - should `sleep 4` after migration before running `incus exec`. -- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across - heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`) - work identically to even counts. A 4-vCPU VM on a 4-core host (100% core - usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs. -- **Concurrent migrations**: migrating multiple VMs simultaneously from - different source nodes works without interference. ~140 MB/s per migration. -- **Active I/O during migration**: disk writes and network activity survive - live migration transparently. File integrity verified after migration. -- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node - joins. Only moves VMs with `migration.stateful=true`. Containers are NOT - auto-rebalanced. - ```bash - incus config set : cluster.rebalance.interval=1 # minutes - incus config set : cluster.rebalance.threshold=10 # imbalance % - incus config set : cluster.rebalance.batch=2 # max VMs/run - incus config set : cluster.rebalance.cooldown=5m # wait between runs - ``` -- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh → - join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md` - for step-by-step instructions. -- **`incus cluster remove` requires confirmation**: even with `--force`, it - prompts "Are you really sure?". Pipe `yes` for automation: - ```bash - printf "yes\n" | incus cluster remove : --force - ``` -- See `notes/clustering-guide.md` for full details and references. - -### Lab validation (lab-test) - -- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates - on the VM names defined there (expects `incus` remotes to exist). -- **Phases:** deploy, single (workloads), cluster, workload (on cluster), - migrate (stop/move + evacuate/restore). -- **Test instances** use names starting with `test-` for easy cleanup. -- The script reports PASS/FAIL/SKIP for each test and prints a summary. - -### Operations Center - -- **CLI binary:** `operations-center` (installed from GitHub releases or - built from source at `github.com/FuturFusion/operations-center`). -- **Config directory:** `~/.config/operations-center/` (uses same cert format - as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). -- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI. -- **Browser access**: requires PKCS#12 client certificate (`client.pfx`) - imported into the browser. Web UI is a React SPA at `/ui/`. -- **OC is under active development** (v0.3.0). Commands and APIs may change. - Use `operations-center --version` (not `version` subcommand) to check. -- **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI, - the OC CLI requires `operations-center remote switch NAME` before running - commands. `operations-center admin os show oc-lab:` fails with "Invalid - number of arguments". -- **v0.3.0 new commands**: `provisioning channel` (add/list/show), - `cluster update/update-certificate/rename`, expanded `inventory` - (network-acl, address-sets, load-balancers, peers, zones, integrations, - forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource - tree view with filters and Go templates), `system certificate set`. -- The `--doctor` command on `incusos-proxmox` reports whether the CLI is - installed. - -#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster) - -- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO. -- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML - must use structured format with section keys (`install:`, not flat). - **No force_reboot** in token seeds for Proxmox (same fix as standard deploys). - ```bash - operations-center provisioning token seed add proxmox-preseed \ - /tmp/preseed.yaml --description "No force_reboot for Proxmox" - operations-center provisioning token seed get-image proxmox-preseed \ - /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable - ``` -- **Hybrid deployment** (tested, recommended): - `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml` - combines OC auto-registration (from boot ISO token) with `incusos-proxmox` - VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, - and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist. -- **Self-registration**: nodes auto-register with OC within ~30s of first - boot. Hostname from SEED_DATA is used as the server name. -- **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires - all nodes to have `needs_update: false` before `provisioning cluster add` - succeeds. Nodes deployed from an ISO matching the latest OC update version - are tracked as `needs_update: true` because the OS was never delivered - through OC's update pipeline. The `needs_update` flag is server-side - computed (not a simple version comparison) and cannot be overridden via - REST API PUT. **Solution**: generate the ISO from an older channel - (`--channel old-stable`) so nodes start with an older version. OC then - pushes the latest update through its pipeline, clearing the flag. Use - `provisioning update assign-channels` to control which versions are in - which channels. -- **Cluster formation** (tested, 2026-02-23): - ```bash - # Use empty app config if cert already injected via SEED_DATA - echo '{}' > /tmp/oc-app-config.yaml - operations-center provisioning cluster add oc-cluster \ - https://:8443 \ - --server-names oc-node-01,oc-node-02,oc-node-03 \ - --server-type incus \ - --application-seed-config /tmp/oc-app-config.yaml - ``` - OC handles: `core.https_address` → cluster enable → joins → storage pool - + network creation → Terraform config. Adds `meshbr0` network. If the - client cert is already in the trust store (from SEED_DATA), use an empty - `{}` app config to avoid "Certificate already in trust store" Terraform - error. The cluster forms successfully either way. -- **apply_defaults: false is recommended** (tested): use `apply_defaults: false` - for OC-managed nodes. OC's Terraform handles storage pool, network, and cert - creation cleanly. With `apply_defaults: true`, nodes already have these - resources and OC's Terraform fails with "already exists" errors (cluster still - forms, but Terraform artifacts are empty). -- **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster - formation via OC, OVN overlay networking works identically to manual clusters. - Deploy ovn-central container, enable OVN services, create UPLINK + OVN - network, add ovn-chassis roles. HA nginx workload with OVN load balancer - tested and working. See `notes/operations-center-guide.md` for full guide. -- **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously - skipped uploading an ISO if one with the same filename existed on Proxmox. - This caused stale ISOs from previous deployments to be silently reused. - Fixed: when `--iso` is explicitly provided, the script now deletes the - existing ISO and re-uploads the new one. - -#### Tested limitations - -- **Inventory is NOT real-time** -- requires explicit `cluster resync` -- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe - on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in - ~50s with data intact). The failure is OC-specific: the OC agent pushes - config via the IncusOS REST API that gets persisted to `state.txt`. On - reboot, invalid values (e.g., cron expression where Go duration is expected) - crash the daemon. See "IncusOS boot failure" section below for full analysis. - Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. -- **No cluster member state tracking** -- OC always shows `ready` even for - EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat - detects failure in ~40s; OC has no equivalent). -- **Stale entries** from out-of-band cluster changes persist after resync -- **Server removal blocked** if server is part of an OC cluster -- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart, - node auto-rejoins cluster in ~60s. Containers auto-start. -- **`needs_update` tracking is pipeline-based**: OC tracks whether an update - was delivered through its pipeline, not just whether versions match. Nodes - deployed from the latest ISO are tracked as needing updates even when - `version == available_version`. Must deploy from older ISO to work around. -- **OVN LB has no health checks**: connection-based hashing distributes - traffic to dead backends. Requests to stopped instances return empty. - -- See `notes/operations-center-guide.md` for full tested OC reference. - -### incusos-proxmox doctor and cleanup - -- **`--doctor`**: standalone environment check. No config file required. - Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally - Proxmox connectivity (from proxmox.yaml or config file). -- **`--cleanup`**: destroys VMs defined in the config file. -- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this - deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete - all IncusOS ISOs (unlike the old behavior). -- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab - config required). Destroys all VMs with `[incusos-lab:managed]` marker. -- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso` - and `seed-*.iso` from storage + remotes + cache. -- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default - output is concise (step names + results). `--quiet` suppresses everything - except warnings and errors. -- **`--retries N`**: number of stop+start retries for VMs that fail to boot - (port 8443 not reachable). Default: 3. `--retries 0` disables retries. - Retries are rarely needed since omitting `force_reboot`. After boot, - `fix_scrub_schedule()` proactively heals any scrub_schedule issues. -- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot - downloads application sysext (~30-120s depending on CDN speed). Do not - reduce — premature retries can corrupt the TPM encryption key permanently. - -### IncusOS disk resize (Proxmox VMs) - -- **Disk grow is non-destructive**: Proxmox `qm resize` only grows disks. -- **Partition 11 auto-expands**: `systemd-repart` runs in initrd on every - boot. The `local-data` partition has no `SizeMaxBytes`, so it fills all - remaining space automatically. -- **ZFS pool does NOT auto-expand**: IncusOS creates the pool without - `autoexpand=on`. After partition grow, `zpool list` shows the correct - `EXPANDSZ` but the pool stays at its original size. -- **Manual expansion via privileged container**: create a container with - `security.privileged=true`, pass `/dev/zfs` (unix-char) and `/dev/sda11` - (unix-block), install `zfsutils-linux`, create a symlink at - `/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11` → - `/dev/sda11`, then run `zpool online -e local - scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11`. Clean up the container - after expansion. Full procedure in `notes/aether-guide.md`. -- **Warning**: running `zpool online -e` with a non-existent device path - inside the container causes pool SUSPENSION. Recovery: Proxmox stop/start - (ZFS re-imports cleanly, no data loss). - -### IncusOS first-boot sequence - -Complete lifecycle from Proxmox VM start to port 8443 ready: - -**Phase 1: ISO boot and installation (~60-85s)** -1. UEFI firmware → IncusOS boot menu → "Starting install of IncusOS to local disk" -2. "Cloning GPT partitions" → progress bar → complete (~876 MiB written) -3. Without `force_reboot`, installer sits at "please remove installation media" - -**Phase 2: Transition (~15s)** -1. `incusos-proxmox` detects install complete via blockstat (876 MiB written, then idle) -2. Proxmox stops VM, removes ide2 (ISO) and ide3 (seed), sets `boot: order=scsi0` -3. Start VM from disk - -**Phase 3: First boot from disk (~50s to port 8443)** -1. UEFI → "IncusOS is starting..." -2. `state.LoadOrCreate()` creates `state.txt` with defaults -3. "Auto-generating encryption recovery key" (~10s) -4. "Downloading SecureBoot update" + "Downloading application update" (~33-38s) -5. "Starting application" → "System is ready" — port 8443 now reachable - -**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable -is the application sysext download (~33-38s on first boot; skipped if cached). - -### IncusOS crontab bug (upstream issue #843) - -**Status:** Eliminated in our pipeline by omitting `force_reboot` from seeds. -The upstream bug still exists but only triggers with `force_reboot: true`. - -- **What it is**: a race condition in IncusOS where `ScrubSchedule` ends up - empty in `state.txt`, causing `registerJobs()` → `gocron.IsValid("")` to - fail and the daemon to exit. Port 8443 never opens (or opens briefly - before the daemon crashes). -- **Our fix**: omit `force_reboot` from the seed. Without it, there is no - SysRq-B intermediate boot, and the race condition does not trigger. - Result: **100% success rate** (vs ~50% with `force_reboot` on 4-core VMs). -- **Detection**: port 8443 reachable but `scrub_schedule` is empty in - `GET /os/1.0/system/storage` → bug hit. `incusos-proxmox` checks this - automatically. -- **Auto-heal**: `incusos-proxmox` includes `fix_scrub_schedule()` which - sets `scrub_schedule` to `"0 4 * * 0"` via `PUT /os/1.0/system/storage` - on every deployed node as a safety net. -- **TPM corruption risk**: hard-stopping a VM during first boot (while the - encryption key is being written) can permanently corrupt the TPM. Error: - "zfs load-key: Raw key too short (expected 32)". Only fix is VM destruction - and redeployment. The 180s boot timeout in `incusos-proxmox` prevents this. -- **On physical hardware**: `force_reboot: true` is still needed (no external - orchestrator to remove install media). The bug may occur; recovery is a - manual power cycle. - -### Proxmox SSH root access — strict rules - -Root SSH access to the Proxmox host is available for **diagnostics only**. -The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`. - -**MANDATORY RULES — violation of any rule is unacceptable:** - -1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM - console screenshots via `qm monitor screendump`. No other use - without explicit user instruction. -2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869) - or VMs in the IncusLab pool that were created by our scripts. -3. **No modifications**: NEVER run any command that modifies, stops, starts, - or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`, - `pct` commands, `zfs` commands, `systemctl` commands, or ANY write - operation on the host. -4. **Do not touch the dev VM**: the user's dev VM runs on the same host. - Do not interact with it in any way — do not even `qm status` it. -5. **No config access**: do not read or modify `/etc/pve/`, storage configs, - network configs, user/ACL settings, or any host-level configuration. -6. **Transparency**: every SSH command is visible in tool output. If the - user rejects a command, do not retry it. -7. **Only during active tests**: only SSH during test runs where test VMs - have been deployed, and only to screenshot those test VMs. - -**Permitted commands (exhaustive list):** +### Live AWX job output +Capture output while a job is running (don't just poll status): ```bash -# Take screenshot of a test VM (VMID in 850-869 range) -sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ - root@ "echo 'screendump /tmp/vm--screen.ppm' | qm monitor " - -# Retrieve screenshot -sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \ - root@:/tmp/vm--screen.ppm /tmp/ - -# Cleanup screenshot on remote -sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ - root@ "rm -f /tmp/vm--screen.ppm" +curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \ + -H "Authorization: Bearer $AWX_TOKEN" ``` -**Technical notes:** -- QEMU's `screendump` is root-only in PVE 9's HMP permission model. - API tokens cannot execute it regardless of privileges. -- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng, - which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`). -- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`: - `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"` - PNG output is ~20 KB and can be read directly by Claude Code (multimodal). +## Critical safety rules -## Coding conventions for scripts +- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`. +- **Never hard-stop VMs during first boot** -- corrupts TPM permanently. +- **Boot timeout is 180s** -- do not reduce (sysext download takes 30-120s). +- **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition. +- **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters). + +## Coding conventions - **Shell**: bash with `set -euo pipefail` -- **Arithmetic**: use `var=$((var + 1))` instead of `((var++))` to avoid - false exits under `set -e` when the value is 0. -- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use setup_colors() pattern. -- **Flags**: support both short (`-d`) and long (`--defaults`) options. -- **Defaults**: sane defaults so the script does something useful with zero flags. -- **Dry run**: all scripts should support `--dry-run` to preview actions. -- **Cert detection order**: files on disk first, CLI command second. -- **Error messages**: include actionable remediation steps, not just "failed". -- **No hardcoded package managers**: say "install the Incus client" with a link, - not "sudo apt install incus". +- **Arithmetic**: `var=$((var + 1))` not `((var++))` (set -e safety) +- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use `setup_colors()` pattern +- **Flags**: both short (`-d`) and long (`--defaults`) +- **Defaults**: useful behavior with zero flags +- **Dry run**: all scripts support `--dry-run` +- **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second +- **Errors**: include actionable remediation steps +- **No hardcoded package managers**: say "install the Incus client" with a link -### Incus networking (OVN) +## Technical context loading -- **Bridge networks are node-local**: each cluster member has its own - independent bridge. Instances on the same bridge (same node) can - communicate; cross-node instances CANNOT. Each bridge has the same subnet - (e.g., 10.0.0.1/24) but they are separate L2 domains. -- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes. - Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane - + client services + physical uplink network. -- **IncusOS OVN services are disabled by default**: must be enabled via the - IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring - Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection` - fails with `db.sock not found`. -- **OVN service enable API call**: - ```bash - incus query :/os/1.0/services/ovn --request PUT --data '{ - "config": { - "database": "tcp::6642", - "enabled": true, - "tunnel_address": "", - "tunnel_protocol": "geneve" - }, - "state": {} - }' - ``` - `database` is the **southbound** DB (port 6642), NOT northbound (6641). -- **OVN control plane as container**: deploy `ovn-central` package in a - Debian container on the cluster. Use proxy devices to expose NB (6641) - and SB (6642) ports on the host's LAN IP so all nodes can reach it. -- **Setup sequence** (order matters): - 1. Deploy OVN control plane container - 2. Enable OVN services on ALL IncusOS nodes - 3. `incus config set network.ovn.northbound_connection tcp::6641` - 4. `incus cluster role add : ovn-chassis` (all nodes) - 5. Create physical uplink network (two-step cluster pattern) - 6. Create OVN network with `--type=ovn network=UPLINK` -- **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC — - NOT `ens18`, which is the underlying device name but not exposed to Incus). - `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, - `ipv4.gateway` is the LAN gateway in CIDR format. -- **OVN network isolation**: multiple OVN networks are fully isolated. - Instances on different networks cannot communicate, even on the same node. - Network peering (`incus network peer create`) enables cross-network routing. -- **OVN features tested**: cross-node connectivity, network isolation, - ACLs (per-source blocking), network peering, L4 load balancers (connection- - based hashing, not round-robin), network forwards (port forwarding to LAN - IPs), DNS resolution (per-network, hostname.incus domain). -- **LB/forward backends require IP addresses**: `incus network load-balancer - backend add` and `incus network forward port add` require the target - instance's IP address, NOT its name. Using instance names fails with - "Invalid target address". -- See `notes/networking-guide.md` for full tutorial with test results. +Detailed technical context loads automatically from `.claude/rules/` based on +which files are being edited: -### Shared storage (iSCSI + lvmcluster) +| Rule file | Loads when editing | +|---|---| +| `incusos-scripts.md` | incusos-iso, incusos-seed, incusos-proxmox, helpers | +| `proxmox-deployment.md` | incusos-proxmox, observe-deploy, helpers, examples | +| `clustering.md` | lab-test, incusos-proxmox, clustering/production guides | +| `operations-center.md` | OC guide, incusos-proxmox, OC example configs | +| `networking-storage.md` | networking, storage, migration, UTM guides | +| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide | +| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | +| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples | -- **iSCSI + lvmcluster** is the IncusOS-native path to shared storage. - All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS - and enabled via the REST API — no packages to install. -- **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN - field is `"target"` (NOT `"iqn"` — using the wrong field silently fails). - The config lives under `config.targets[]`, and `state` returns the - auto-generated `initiator_name`. -- **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique - `system_id` (1-2000) per node for sanlock host identification. Using 0 - or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool - creation. -- **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on - custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately. - ~256 MiB overhead for LVM metadata + sanlock lease area. -- **Pool creation**: two-step cluster pattern (`--target` per member, then - finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`, - and `vgchange --lock-start` automatically. -- **Migration performance** (tested on 1GbE, 2026-02-23): - - Container stop/move/start: **0.12-0.15s** (metadata only) - - VM non-live (stop/move/start): **1.8s** (LVM metadata update) - - VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer) - - Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data) -- **Proxy devices don't work for iSCSI**: SendTargets discovery returns - container IP in TargetAddress, causing portal mismatch. Use direct network - paths (bridge for same-node, macvlan for cross-node). -- **Lab target container**: Debian container with `tgt` (userspace iSCSI - target) on the cluster — no external hardware needed. Uses dual network: - bridge IP for same-node access, macvlan on `mgmt` for cross-node access. -- **First live migration after stop/start may fail**: QEMU on destination - fails to start (transient sanlock lease issue). Retry succeeds. Non-live - migration always works as fallback. -- **Hybrid architecture recommended**: local ZFS (`local` pool) for general - workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant - migration. -- See `notes/shared-storage-guide.md` for the full tested walkthrough. - -### Migration into Incus - -- **`incus-migrate`**: official tool for importing disk images, running - instances, or physical machines into Incus. -- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2, - raw, vdi, vhd formats. Incus accepts raw and qcow2. -- **Import workflow**: convert disk → `incus storage volume import` → - `incus init --empty --vm` → attach disk → start. -- **Container migration**: `docker export` → `incus import` for filesystem- - level container migration. Docker volumes must be copied separately. -- See `notes/migration-guide.md` for full procedures per source hypervisor. - -### UTM support (future) - -- Design document at `notes/utm-support.md`. -- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation - (requires AppleScript or `.utm` bundle generation). -- No `blockstat` equivalent -- install detection must use timeout + port polling. -- Seed generation already works cross-platform (Phase 3 macOS compatibility). - -### AWX integration (Ansible automation for Aether) - -- **AWX** is the open-source Ansible automation platform (upstream of Ansible - Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster. -- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent - to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk. -- **AWX URL**: `http://192.168.102.161:30080` — exposed via K3s NodePort. - Traefik ingress returns 404 for IP-based access; use NodePort directly. -- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`, - `--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`. -- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize). -- **Ansible playbooks** in `ansible/` directory: - - `playbooks/post-deploy.yml` -- runs after Aether creates an instance - - `playbooks/decommission.yml` -- runs before Aether deletes an instance -- **Aether extra vars use `ffsdn_` prefix**: Aether passes `ffsdn_instance_name`, - `ffsdn_instance_ip`, `ffsdn_cluster_id`, `ffsdn_cluster_name`, - `ffsdn_deployed_by`, `ffsdn_image_os`, `ffsdn_image_release`, - `ffsdn_image_alias`. It does NOT pass `vm_name`, `vm_ip`, `environment`, - `owner`, or `cost_center` (the original plan assumed these). -- **Playbook pattern — Incus REST API, not SSH**: AWX cannot SSH to containers - on incusbr0 (bridge subnet not routable from management VLAN, IncusOS - nftables blocks inbound forwarding). Playbooks use the Incus REST API - (`uri` module with client cert) for file push + exec. The cert is at - `/runner/project/incus-client.crt` during job execution (AWX EE mounts - projects at `/runner/project/`, NOT `/var/lib/awx/projects/`). -- **Manual project (local_path)**: AWX EE containers cannot reach the private - git repo. Use `scm_type: ""` with `local_path: "incus-contrib"` and push - playbooks directly to the AWX task pod at - `/var/lib/awx/projects/incus-contrib/playbooks/`. -- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance - deleted). Decommission failure does NOT block deletion. -- **Aether cluster AWX config API bug**: `PUT /api/clusters/{id}/awx-config` - returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL - UPDATE on the `clusters` table from within the Aether container. -- **Self-referencing vars cause infinite recursion in Ansible**: patterns like - `vm_ip: "{{ vm_ip | default('') }}"` cause `AnsibleUndefinedVariable` - recursive loop when the variable is not provided. Use `ffsdn_*` vars - directly without redefining them. -- **Ansible `environment` is a reserved keyword**: using it as an extra var - resolves to `[]` instead of the string value. -- See `notes/awx-guide.md` for full deployment guide and troubleshooting. +For deep reference, see the guides in `notes/`. ## Git workflow -- Main branch: `main` +- **Main branch**: `master` +- **Remotes**: + - `origin`: `ssh://git@192.168.1.200:2222/maarten/incus-contrib.git` (private Gitea) + - `aether`: `_gitea@code.sovereignprivatecloud.nl:maarten/incus-contrib.git` +- Push to **both remotes** when committing - Development happens on feature branches -- Remote: private Gitea at `ssh://git@192.168.1.200:2222/maarten/incu-contrib.git` diff --git a/incusos/helpers/proxmox-api b/incusos/helpers/proxmox-api new file mode 100755 index 0000000..518d3c2 --- /dev/null +++ b/incusos/helpers/proxmox-api @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# proxmox-api -- Make authenticated Proxmox API calls +# +# Usage: proxmox-api METHOD PATH [CURL_ARGS...] +# +# Handles the ! in automation@pve!deploy safely (no bash history expansion). +# Config: proxmox.yaml (host, api_token_id), env file (PROXMOX_TOKEN_SECRET) +# +# Outputs JSON response on stdout. + +set -euo pipefail + +# --------------------------------------------------------------------------- +# Usage +# --------------------------------------------------------------------------- + +usage() { + cat <<'EOF' +Usage: proxmox-api METHOD PATH [CURL_ARGS...] + +Make authenticated Proxmox API calls with proper token handling. + +Arguments: + METHOD HTTP method (GET, POST, PUT, DELETE) + PATH API path (e.g., /nodes/pve/qemu) + CURL_ARGS Additional curl arguments (e.g., -d 'key=val') + +Options: + --json Pretty-print JSON output + --help, -h Show this help + +Environment: + PROXMOX_TOKEN_SECRET API token secret (from env file) + +Config: + proxmox.yaml Proxmox host + api_token_id (auto-discovered) + +Examples: + proxmox-api GET /nodes/pve/qemu + proxmox-api GET /nodes/pve/qemu/900/status/current --json + proxmox-api POST /nodes/pve/qemu/900/status/stop + proxmox-api GET /cluster/resources --json +EOF + exit 0 +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage +fi + +# --------------------------------------------------------------------------- +# Parse arguments +# --------------------------------------------------------------------------- + +PRETTY=false +ARGS=() +for arg in "$@"; do + if [[ "$arg" == "--json" ]]; then + PRETTY=true + else + ARGS+=("$arg") + fi +done + +if [[ ${#ARGS[@]} -lt 2 ]]; then + echo "Error: METHOD and PATH required. Usage: proxmox-api METHOD PATH [CURL_ARGS...]" >&2 + exit 1 +fi + +METHOD="${ARGS[0]}" +API_PATH="${ARGS[1]}" +EXTRA_ARGS=("${ARGS[@]:2}") + +# --------------------------------------------------------------------------- +# Load environment +# --------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +load_env() { + local dir="$SCRIPT_DIR" + while [[ "$dir" != "/" ]]; do + if [[ -f "${dir}/env" ]]; then + # shellcheck disable=SC1091 + source "${dir}/env" + return + fi + dir="$(dirname "$dir")" + done +} +load_env + +if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then + echo "Error: PROXMOX_TOKEN_SECRET not set. Create an env file or export it." >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Load proxmox.yaml +# --------------------------------------------------------------------------- + +yaml_get() { + local file="$1" key="$2" default="${3:-}" + local val + val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val="" + val="${val#\"}" ; val="${val%\"}" + val="${val#\'}" ; val="${val%\'}" + echo "${val:-$default}" +} + +find_proxmox_yaml() { + local search_dirs=( + "${SCRIPT_DIR}/.." + "${SCRIPT_DIR}/../.." + "$(pwd)" + ) + for dir in "${search_dirs[@]}"; do + if [[ -f "${dir}/proxmox.yaml" ]]; then + echo "${dir}/proxmox.yaml" + return + fi + if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then + echo "${dir}/incusos/proxmox.yaml" + return + fi + done + echo "" +} + +PROXMOX_YAML=$(find_proxmox_yaml) +if [[ -z "$PROXMOX_YAML" ]]; then + echo "Error: proxmox.yaml not found" >&2 + exit 1 +fi + +PVE_HOST=$(yaml_get "$PROXMOX_YAML" "host") +PVE_TOKEN_ID=$(yaml_get "$PROXMOX_YAML" "api_token_id") + +if [[ -z "$PVE_HOST" ]]; then + echo "Error: proxmox.yaml: 'host' is required" >&2 + exit 1 +fi +if [[ -z "$PVE_TOKEN_ID" ]]; then + echo "Error: proxmox.yaml: 'api_token_id' is required" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Make API call +# --------------------------------------------------------------------------- + +# Build the full auth header in a variable (avoids ! expansion issues) +AUTH_HEADER="Authorization: PVEAPIToken=${PVE_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" + +URL="https://${PVE_HOST}:8006/api2/json${API_PATH}" + +CURL_CMD=(curl -fsSk -X "$METHOD" -H "$AUTH_HEADER") + +if [[ ${#EXTRA_ARGS[@]} -gt 0 ]]; then + CURL_CMD+=("${EXTRA_ARGS[@]}") +fi + +CURL_CMD+=("$URL") + +RESPONSE=$("${CURL_CMD[@]}" 2>&1) || { + echo "Error: API call failed: $METHOD $API_PATH" >&2 + echo "$RESPONSE" >&2 + exit 1 +} + +if [[ "$PRETTY" == true ]]; then + echo "$RESPONSE" | python3 -m json.tool 2>/dev/null || echo "$RESPONSE" +else + echo "$RESPONSE" +fi diff --git a/incusos/helpers/proxmox-screenshot b/incusos/helpers/proxmox-screenshot new file mode 100755 index 0000000..34a668d --- /dev/null +++ b/incusos/helpers/proxmox-screenshot @@ -0,0 +1,172 @@ +#!/usr/bin/env bash +# proxmox-screenshot -- Take a console screenshot of a Proxmox VM +# +# Usage: proxmox-screenshot VMID [OUTPUT.png] +# +# Requires: sshpass, python3-pil (for PNG conversion) +# Config: proxmox.yaml (host), env file (PROXMOX_ROOT_PASSWORD) +# +# Outputs the path to the PNG file on stdout. + +set -euo pipefail + +# --------------------------------------------------------------------------- +# Usage +# --------------------------------------------------------------------------- + +usage() { + cat <<'EOF' +Usage: proxmox-screenshot VMID [OUTPUT.png] + +Take a console screenshot of a Proxmox VM via QEMU monitor screendump. + +Arguments: + VMID VM ID (must be in allowed range: 400-499, 800-939) + OUTPUT.png Output file path (default: /tmp/vm-VMID-screen.png) + +Environment: + PROXMOX_ROOT_PASSWORD SSH root password (from env file) + +Config: + proxmox.yaml Proxmox host (auto-discovered) + +Examples: + proxmox-screenshot 900 + proxmox-screenshot 900 /tmp/my-screenshot.png +EOF + exit 0 +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage +fi + +# --------------------------------------------------------------------------- +# Validate arguments +# --------------------------------------------------------------------------- + +if [[ $# -lt 1 ]]; then + echo "Error: VMID required. Usage: proxmox-screenshot VMID [OUTPUT.png]" >&2 + exit 1 +fi + +VMID="$1" +OUTPUT="${2:-/tmp/vm-${VMID}-screen.png}" + +# VMID safety check: only allowed ranges +if ! [[ "$VMID" =~ ^[0-9]+$ ]]; then + echo "Error: VMID must be a number" >&2 + exit 1 +fi + +if ! (( (VMID >= 400 && VMID <= 499) || (VMID >= 800 && VMID <= 939) )); then + echo "Error: VMID $VMID not in allowed range (400-499, 800-939)" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Load environment +# --------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +load_env() { + local dir="$SCRIPT_DIR" + while [[ "$dir" != "/" ]]; do + if [[ -f "${dir}/env" ]]; then + # shellcheck disable=SC1091 + source "${dir}/env" + return + fi + dir="$(dirname "$dir")" + done +} +load_env + +if [[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]]; then + echo "Error: PROXMOX_ROOT_PASSWORD not set. Add it to the env file." >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Load proxmox.yaml for host +# --------------------------------------------------------------------------- + +yaml_get() { + local file="$1" key="$2" default="${3:-}" + local val + val=$(awk -F': ' -v k="$key" '$1 == k {print $2; exit}' "$file" 2>/dev/null) || val="" + val="${val#\"}" ; val="${val%\"}" + val="${val#\'}" ; val="${val%\'}" + echo "${val:-$default}" +} + +find_proxmox_yaml() { + # Search: script's grandparent (repo/incusos -> repo), script parent, cwd + local search_dirs=( + "${SCRIPT_DIR}/.." + "${SCRIPT_DIR}/../.." + "$(pwd)" + ) + for dir in "${search_dirs[@]}"; do + if [[ -f "${dir}/proxmox.yaml" ]]; then + echo "${dir}/proxmox.yaml" + return + fi + if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then + echo "${dir}/incusos/proxmox.yaml" + return + fi + done + echo "" +} + +PROXMOX_YAML=$(find_proxmox_yaml) +if [[ -z "$PROXMOX_YAML" ]]; then + echo "Error: proxmox.yaml not found" >&2 + exit 1 +fi + +PVE_HOST=$(yaml_get "$PROXMOX_YAML" "host") +if [[ -z "$PVE_HOST" ]]; then + echo "Error: proxmox.yaml: 'host' is required" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Take screenshot +# --------------------------------------------------------------------------- + +SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -o LogLevel=ERROR) +REMOTE_PPM="/tmp/vm-${VMID}-screen.ppm" +LOCAL_PPM="/tmp/vm-${VMID}-screen.ppm" + +# Take screenshot via QEMU monitor +if ! sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" \ + "root@${PVE_HOST}" \ + "echo 'screendump ${REMOTE_PPM}' | qm monitor ${VMID}" &>/dev/null; then + echo "Error: screendump failed (VM $VMID not running or SSH error)" >&2 + exit 1 +fi + +# Retrieve PPM file +if ! sshpass -p "$PROXMOX_ROOT_PASSWORD" scp "${SSH_OPTS[@]}" \ + "root@${PVE_HOST}:${REMOTE_PPM}" "$LOCAL_PPM" &>/dev/null; then + echo "Error: failed to retrieve screenshot from Proxmox host" >&2 + exit 1 +fi + +# Cleanup remote file +sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh "${SSH_OPTS[@]}" \ + "root@${PVE_HOST}" "rm -f ${REMOTE_PPM}" &>/dev/null || true + +# Convert PPM -> PNG +if python3 -c "from PIL import Image; Image.open('${LOCAL_PPM}').save('${OUTPUT}')" 2>/dev/null; then + rm -f "$LOCAL_PPM" +else + # PIL not available, keep as PPM + OUTPUT="${OUTPUT%.png}.ppm" + mv "$LOCAL_PPM" "$OUTPUT" 2>/dev/null || true +fi + +echo "$OUTPUT"