incus-contrib/CLAUDE.md

# CLAUDE.md - Project context for AI assistants

## What this repository is

A collection of peripheral tools, scripts, and snippets for working with
Incus, IncusOS, and the broader ecosystem (Operations Center, Migration Manager).
Primarily targeting home lab environments but aiming for production-quality scripts.

## Repository structure

```
incus-contrib/
├── CLAUDE.md              # This file -- behavioral rules + project overview
├── .claude/rules/         # Topic-specific context (auto-loaded by paths:)
├── .claude/skills/        # Slash commands: /screenshot, /proxmox-api
├── README.md              # Main overview
├── .gitignore
├── ansible/               # Ansible playbooks for AWX/Aether lifecycle hooks
│   ├── ansible.cfg
│   └── playbooks/
│       ├── post-deploy.yml    # Runs after Aether creates an instance
│       └── decommission.yml   # Runs before Aether deletes an instance
├── bin/                   # ALL reusable scripts
│   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
│   ├── incusos-seed       # Seed archive generator (Linux + macOS)
│   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
│   ├── lab-test           # Guided lab validation (12 test phases)
│   ├── deploy-awx         # AWX deployment + management on Incus cluster
│   ├── deploy-haproxy     # HAProxy LB deployment + management via Aether
│   ├── deploy-observability # Prometheus + Grafana + Loki observability stack
│   ├── manage-dashboards  # Grafana dashboard management
│   ├── pve-api            # Python Proxmox API helper
│   ├── helpers/
│   │   ├── proxmox-screenshot  # VMID -> PNG console screenshot
│   │   ├── proxmox-api         # Authenticated API calls (handles ! in token)
│   │   ├── aether-browser      # Playwright browser automation for Aether web UI
│   │   ├── ovn-inspect         # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
│   │   ├── incus-mitm         # API traffic capture and analysis (--capture/--live/--analyze)
│   │   └── incusos-health    # IncusOS health inspection (--status/--tpm/--partitions/--all)
│   ├── data/              # Static data consumed by scripts
│   │   ├── awx-manifests/     # K8s manifests for AWX Operator + instance
│   │   └── observability-dashboards/  # Grafana dashboard JSON exports
│   ├── diagnostic/        # Investigation/one-off tools
│   │   ├── observe-deploy     # Single-VM deploy with console screenshots
│   │   ├── investigate-boot   # Boot investigation
│   │   └── test-boot-reliability  # Parallel boot testing
│   ├── examples/          # Example seed + Proxmox YAML files
│   └── TESTING.md
├── envs/                  # Per-environment config + secrets
│   ├── README.md          # How environments work
│   ├── beelink/           # Beelink mini PC target
│   │   ├── env                # Secrets (gitignored)
│   │   ├── proxmox.yaml       # Proxmox connection (gitignored)
│   │   ├── proxmox.yaml.example
│   │   ├── lab-cluster.yaml
│   │   ├── awx.yaml
│   │   ├── haproxy.yaml
│   │   └── observability.yaml
│   └── hetzner/           # Hetzner dedicated server target
│       ├── env                # Secrets (gitignored)
│       ├── proxmox.yaml       # Proxmox connection (gitignored)
│       ├── proxmox.yaml.example
│       ├── lab-cluster.yaml
│       ├── lab-production.yaml
│       ├── awx.yaml
│       ├── haproxy.yaml
│       ├── observability.yaml
│       └── setup/             # Host provisioning
│           ├── README.md
│           ├── hetzner-setup.md
│           ├── hetzner-lab-guide.md
│           └── proxmox-setup
├── notes/                 # Reference guides (clustering, networking, OC, AWX, etc.)
└── sources/               # External software (gitignored)
```

## Capabilities you have

### Proxmox VM screenshots
Take console screenshots during deploys to see boot progress or errors:
```
bin/helpers/proxmox-screenshot VMID [/tmp/output.png]
```
Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during
deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.

### Proxmox API calls
Make authenticated API calls (handles the `!` in `automation@pve!deploy`):
```
bin/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
bin/helpers/proxmox-api GET /nodes/pve/status --json
bin/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
```

### Aether API
Aether has a documented REST API at `https://192.168.102.160:8443/api/`.

- **Swagger UI**: `https://192.168.102.160:8443/api/docs`
- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
  Credentials in `envs/*/env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.

```bash
# Get JWT
TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
  -H "Content-Type: application/json" \
  -d "{\"username\":\"admin\",\"password\":\"$AETHER_ADMIN_PASSWORD\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

# Use it
curl -sk https://192.168.102.160:8443/api/clusters \
  -H "Authorization: Bearer $TOKEN"
```

Current API coverage (v6.4.317, API v1.0.0):
- Authentication: `POST /api/auth/token`
- Clusters: `GET /api/clusters`
- Local ACLs: CRUD on `/api/clusters/{id}/rules`
- Global ACLs: CRUD on `/api/global/rules`

Not yet in the API: AWX endpoints, deploys, blueprints, instance management,
health checks. These are available through the web UI only. The API is
actively being developed — check `/api/swagger.yaml` for new endpoints
in future Aether versions.

### Live AWX job output
Capture output while a job is running (don't just poll status):
```bash
curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
  -H "Authorization: Bearer $AWX_TOKEN"
```

### Aether browser automation (Playwright)

Many Aether features (HAProxy management, blueprints, deploys) are **not in
the JWT API** — they use session-authenticated routes with CSRF protection
that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions.

**MCP server**: Configured in `.mcp.json` — provides `browser_navigate`,
`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
the Playwright MCP server is running. Prefer MCP tools when available.

**Helper script**: `bin/helpers/aether-browser` provides standalone
Playwright automation when MCP tools aren't loaded:
```bash
source envs/beelink/env  # loads AETHER_ADMIN_PASSWORD
NODE_PATH=~/node_modules node bin/helpers/aether-browser <action> [args]
```

Actions:
- `login` — authenticate and save session cookies to `/tmp/aether-cookies.json`
- `screenshot <path>` — navigate to path, take screenshot
- `navigate <path>` — navigate and show page title/URL
- `haproxy-status` — screenshot HAProxy management page
- `haproxy-images` — list built HAProxy images via session API
- `api-get <endpoint>` — authenticated GET using browser session
- `api-post <endpoint> <json>` — authenticated POST with CSRF handling
- `eval <js>` — evaluate JavaScript on the current page

**When to use what**:
- JWT API (`/api/*` endpoints): use `curl` with bearer token
- Session-authenticated routes (`/haproxy/*`, UI actions): use Playwright
- Never use curl for session auth — CSRF protection requires browser cookies

**Key technical detail**: Aether login is at `/` (root), NOT `/login`.
The form POSTs to `/login`. CSRF token is in a hidden `<input>` field,
not in cookies.

## Critical safety rules

- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`.
- **Never hard-stop VMs during first boot** -- corrupts TPM permanently.
- **Boot timeout is 180s** -- do not reduce (sysext download takes 30-120s).
- **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition.
- **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters).

## Interacting with external systems (Aether, AWX, Incus, etc.)

**Strict priority order** — follow this hierarchy in all guides, scripts,
and automation flows:

1. **API first**: if the system has a documented REST API, use it.
   Authenticate as a regular user (JWT, PAT), never as root / DB superuser.
2. **Web UI fallback**: if the API doesn't cover a feature yet, document
   the UI steps (navigation path, form fields, expected result).
3. **Never use root / direct DB access in guides or automation.** Poking
   around in containers, editing databases directly, or using `incus exec`
   into Aether is acceptable **only for investigation and learning** — never
   as the documented path for users to follow.

APIs improve over time. When a feature is UI-only today, note it and check
`/api/swagger.yaml` (or equivalent) in future versions before assuming it's
still UI-only. The API endpoint may have been added since the guide was written.

## Coding conventions

- **Shell**: bash with `set -euo pipefail`
- **Arithmetic**: `var=$((var + 1))` not `((var++))` (set -e safety)
- **Colors**: support `NO_COLOR=1` and `TERM=dumb`; use `setup_colors()` pattern
- **Flags**: both short (`-d`) and long (`--defaults`)
- **Defaults**: useful behavior with zero flags
- **Dry run**: all scripts support `--dry-run`
- **Config**: deploy scripts use `--config FILE` for environment-specific settings
- **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second
- **Errors**: include actionable remediation steps
- **No hardcoded package managers**: say "install the Incus client" with a link

## Technical context loading

Detailed technical context loads automatically from `.claude/rules/` based on
which files are being edited:

| Rule file | Loads when editing |
|---|---|
| `incusos-scripts.md` | incusos-iso, incusos-seed, incusos-proxmox, helpers |
| `proxmox-deployment.md` | incusos-proxmox, observe-deploy, helpers, examples |
| `clustering.md` | lab-test, incusos-proxmox, clustering/production guides |
| `operations-center.md` | OC guide, incusos-proxmox, OC example configs |
| `networking-storage.md` | networking, storage, migration, UTM guides |
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
| `observability.md` | deploy-observability, observability dashboards/guide |
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
| `aether-api-payloads.md` | incus-mitm, api-interception |
| `incusos-immutability.md` | incusos-health, incusos-break-fix |
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
| `hetzner-setup.md` | envs/hetzner/, setup scripts |

For deep reference, see the guides in `notes/`.

## Git workflow

- **Main branch**: `master`
- **Remotes**:
  - `origin`: `ssh://git@192.168.1.200:2222/maarten/incus-contrib.git` (private Gitea)
  - `aether`: `_gitea@code.sovereignprivatecloud.nl:maarten/incus-contrib.git`
- Push to **both remotes** when committing
- Development happens on feature branches