# CLAUDE.md - Project context for AI assistants ## What this repository is A collection of peripheral tools, scripts, and snippets for working with Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager). Primarily targeting home lab environments but aiming for production-quality scripts. ## Repository structure ``` incus-contrib/ ├── CLAUDE.md # This file -- behavioral rules + project overview ├── .claude/rules/ # Topic-specific context (auto-loaded by paths:) ├── .claude/skills/ # Slash commands: /screenshot, /proxmox-api ├── README.md # Main overview ├── .gitignore ├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks │ ├── ansible.cfg │ └── playbooks/ │ ├── post-deploy.yml # Runs after Aether creates an instance │ └── decommission.yml # Runs before Aether deletes an instance ├── bin/ # ALL reusable scripts │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator (Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── lab-test # Guided lab validation (12 test phases) │ ├── deploy-awx # AWX deployment + management on Incus cluster │ ├── deploy-haproxy # HAProxy LB deployment + management via Aether │ ├── deploy-observability # Prometheus + Grafana + Loki observability stack │ ├── manage-dashboards # Grafana dashboard management │ ├── pve-api # Python Proxmox API helper │ ├── helpers/ │ │ ├── proxmox-screenshot # VMID -> PNG console screenshot │ │ ├── proxmox-api # Authenticated API calls (handles ! in token) │ │ ├── aether-browser # Playwright browser automation for Aether web UI │ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace) │ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze) │ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all) │ ├── data/ # Static data consumed by scripts │ │ ├── awx-manifests/ # K8s manifests for AWX Operator + instance │ │ └── observability-dashboards/ # Grafana dashboard JSON exports │ ├── diagnostic/ # Investigation/one-off tools │ │ ├── observe-deploy # Single-VM deploy with console screenshots │ │ ├── investigate-boot # Boot investigation │ │ └── test-boot-reliability # Parallel boot testing │ ├── examples/ # Example seed + Proxmox YAML files │ └── TESTING.md ├── envs/ # Per-environment config + secrets │ ├── README.md # How environments work │ ├── beelink/ # Beelink mini PC target │ │ ├── env # Secrets (gitignored) │ │ ├── proxmox.yaml # Proxmox connection (gitignored) │ │ ├── proxmox.yaml.example │ │ ├── lab-cluster.yaml │ │ ├── awx.yaml │ │ ├── haproxy.yaml │ │ └── observability.yaml │ └── hetzner/ # Hetzner dedicated server target │ ├── env # Secrets (gitignored) │ ├── proxmox.yaml # Proxmox connection (gitignored) │ ├── proxmox.yaml.example │ ├── lab-cluster.yaml │ ├── lab-production.yaml │ ├── awx.yaml │ ├── haproxy.yaml │ ├── observability.yaml │ └── setup/ # Host provisioning │ ├── README.md │ ├── hetzner-setup.md │ ├── hetzner-lab-guide.md │ └── proxmox-setup ├── notes/ # Reference guides (clustering, networking, OC, AWX, etc.) └── sources/ # External software (gitignored) ``` ## Capabilities you have ### Proxmox VM screenshots Take console screenshots during deploys to see boot progress or errors: ``` bin/helpers/proxmox-screenshot VMID [/tmp/output.png] ``` Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during deploy scenarios to monitor boot progress, diagnose hangs, and verify installs. ### Proxmox API calls Make authenticated API calls (handles the `!` in `automation@pve!deploy`): ``` bin/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json bin/helpers/proxmox-api GET /nodes/pve/status --json bin/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop ``` ### Aether API Aether has a documented REST API at `https://192.168.102.160:8443/api/`. - **Swagger UI**: `https://192.168.102.160:8443/api/docs` - **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml` - **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}` Credentials in `envs/*/env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h. ```bash # Get JWT TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \ -H "Content-Type: application/json" \ -d "{\"username\":\"admin\",\"password\":\"$AETHER_ADMIN_PASSWORD\"}" \ | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") # Use it curl -sk https://192.168.102.160:8443/api/clusters \ -H "Authorization: Bearer $TOKEN" ``` Current API coverage (v6.4.317, API v1.0.0): - Authentication: `POST /api/auth/token` - Clusters: `GET /api/clusters` - Local ACLs: CRUD on `/api/clusters/{id}/rules` - Global ACLs: CRUD on `/api/global/rules` Not yet in the API: AWX endpoints, deploys, blueprints, instance management, health checks. These are available through the web UI only. The API is actively being developed — check `/api/swagger.yaml` for new endpoints in future Aether versions. ### Live AWX job output Capture output while a job is running (don't just poll status): ```bash curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \ -H "Authorization: Bearer $AWX_TOKEN" ``` ### Aether browser automation (Playwright) Many Aether features (HAProxy management, blueprints, deploys) are **not in the JWT API** — they use session-authenticated routes with CSRF protection that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions. **MCP server**: Configured in `.mcp.json` — provides `browser_navigate`, `browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when the Playwright MCP server is running. Prefer MCP tools when available. **Helper script**: `bin/helpers/aether-browser` provides standalone Playwright automation when MCP tools aren't loaded: ```bash source envs/beelink/env # loads AETHER_ADMIN_PASSWORD NODE_PATH=~/node_modules node bin/helpers/aether-browser [args] ``` Actions: - `login` — authenticate and save session cookies to `/tmp/aether-cookies.json` - `screenshot ` — navigate to path, take screenshot - `navigate ` — navigate and show page title/URL - `haproxy-status` — screenshot HAProxy management page - `haproxy-images` — list built HAProxy images via session API - `api-get ` — authenticated GET using browser session - `api-post ` — authenticated POST with CSRF handling - `eval ` — evaluate JavaScript on the current page **When to use what**: - JWT API (`/api/*` endpoints): use `curl` with bearer token - Session-authenticated routes (`/haproxy/*`, UI actions): use Playwright - Never use curl for session auth — CSRF protection requires browser cookies **Key technical detail**: Aether login is at `/` (root), NOT `/login`. The form POSTs to `/login`. CSRF token is in a hidden `` field, not in cookies. ## Critical safety rules - **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`. - **Never hard-stop VMs during first boot** -- corrupts TPM permanently. - **Boot timeout is 180s** -- do not reduce (sysext download takes 30-120s). - **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition. - **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters). ## Interacting with external systems (Aether, AWX, Incus, etc.) **Strict priority order** — follow this hierarchy in all guides, scripts, and automation flows: 1. **API first**: if the system has a documented REST API, use it. Authenticate as a regular user (JWT, PAT), never as root / DB superuser. 2. **Web UI fallback**: if the API doesn't cover a feature yet, document the UI steps (navigation path, form fields, expected result). 3. **Never use root / direct DB access in guides or automation.** Poking around in containers, editing databases directly, or using `incus exec` into Aether is acceptable **only for investigation and learning** — never as the documented path for users to follow. APIs improve over time. When a feature is UI-only today, note it and check `/api/swagger.yaml` (or equivalent) in future versions before assuming it's still UI-only. The API endpoint may have been added since the guide was written. ## Coding conventions - **Shell**: bash with `set -euo pipefail` - **Arithmetic**: `var=$((var + 1))` not `((var++))` (set -e safety) - **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use `setup_colors()` pattern - **Flags**: both short (`-d`) and long (`--defaults`) - **Defaults**: useful behavior with zero flags - **Dry run**: all scripts support `--dry-run` - **Config**: deploy scripts use `--config FILE` for environment-specific settings - **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second - **Errors**: include actionable remediation steps - **No hardcoded package managers**: say "install the Incus client" with a link ## Technical context loading Detailed technical context loads automatically from `.claude/rules/` based on which files are being edited: | Rule file | Loads when editing | |---|---| | `incusos-scripts.md` | incusos-iso, incusos-seed, incusos-proxmox, helpers | | `proxmox-deployment.md` | incusos-proxmox, observe-deploy, helpers, examples | | `clustering.md` | lab-test, incusos-proxmox, clustering/production guides | | `operations-center.md` | OC guide, incusos-proxmox, OC example configs | | `networking-storage.md` | networking, storage, migration, UTM guides | | `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide | | `haproxy-lb.md` | deploy-haproxy, HAProxy guide | | `observability.md` | deploy-observability, observability dashboards/guide | | `ovn-internals.md` | ovn-inspect, ovn-deep-dive | | `aether-api-payloads.md` | incus-mitm, api-interception | | `incusos-immutability.md` | incusos-health, incusos-break-fix | | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | | `lab-infrastructure.md` | incusos-proxmox, lab-test, examples | | `hetzner-setup.md` | envs/hetzner/, setup scripts | For deep reference, see the guides in `notes/`. ## Git workflow - **Main branch**: `master` - **Remotes**: - `origin`: `ssh://git@192.168.1.200:2222/maarten/incus-contrib.git` (private Gitea) - `aether`: `_gitea@code.sovereignprivatecloud.nl:maarten/incus-contrib.git` - Push to **both remotes** when committing - Development happens on feature branches