Validate AWX guide, document Aether API, enforce API-first policy

- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups,
  add missing Step 5 push commands, add migration pod to expected
  output, replace PostgreSQL workaround with Aether UI instructions,
  reference deploy-awx as automated path
- Aether guide: document JWT REST API (auth, clusters, ACLs),
  swagger UI at /api/docs, rate limits, coverage vs UI-only features
- CLAUDE.md: add Aether API capability, add interaction hierarchy
  (API first → UI fallback → never root/DB in guides)
- deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks"
- post-deploy.yml: clarify cert push path vs EE runtime mount path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.claude/rules/awx-integration.md

@@ -51,6 +51,7 @@ It does NOT pass: `vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`.
   infinite recursion. Use `ffsdn_*` vars directly.
 - **`environment` is reserved**: resolves to `[]` as extra var.
 - **AWX config API bug**: `PUT /api/clusters/{id}/awx-config` returns
-  "Invalid cluster ID". Workaround: direct PostgreSQL UPDATE.
+  "Invalid cluster ID" (v6.4.317). Use the Aether UI instead (cluster
+  Settings tab or per-blueprint AWX template IDs).
 - **Lifecycle hooks**: post-deploy failure -> auto-rollback (instance deleted).
   Decommission failure does NOT block deletion.

CLAUDE.md

@@ -35,7 +35,7 @@ incus-contrib/
 │   ├── proxmox.yaml       # Proxmox connection (gitignored)
 │   ├── TESTING.md
 │   └── examples/          # Example seed + Proxmox YAML files
-├── env                    # PROXMOX_TOKEN_SECRET + PROXMOX_ROOT_PASSWORD (gitignored)
+├── env                    # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored)
 └── notes/                 # Reference guides (clustering, networking, OC, AWX, etc.)
 ```
 
@@ -57,6 +57,37 @@ incusos/helpers/proxmox-api GET /nodes/pve/status --json
 incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
 ```
 
+### Aether API
+Aether has a documented REST API at `https://192.168.102.160:8443/api/`.
+
+- **Swagger UI**: `https://192.168.102.160:8443/api/docs`
+- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
+- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
+  Credentials in `env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
+
+```bash
+# Get JWT
+TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
+  -H "Content-Type: application/json" \
+  -d "{\"username\":\"admin\",\"password\":\"$AETHER_ADMIN_PASSWORD\"}" \
+  | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
+
+# Use it
+curl -sk https://192.168.102.160:8443/api/clusters \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+Current API coverage (v6.4.317, API v1.0.0):
+- Authentication: `POST /api/auth/token`
+- Clusters: `GET /api/clusters`
+- Local ACLs: CRUD on `/api/clusters/{id}/rules`
+- Global ACLs: CRUD on `/api/global/rules`
+
+Not yet in the API: AWX endpoints, deploys, blueprints, instance management,
+health checks. These are available through the web UI only. The API is
+actively being developed — check `/api/swagger.yaml` for new endpoints
+in future Aether versions.
+
 ### Live AWX job output
 Capture output while a job is running (don't just poll status):
 ```bash
@@ -72,6 +103,24 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
 - **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition.
 - **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters).
 
+## Interacting with external systems (Aether, AWX, Incus, etc.)
+
+**Strict priority order** — follow this hierarchy in all guides, scripts,
+and automation flows:
+
+1. **API first**: if the system has a documented REST API, use it.
+   Authenticate as a regular user (JWT, PAT), never as root / DB superuser.
+2. **Web UI fallback**: if the API doesn't cover a feature yet, document
+   the UI steps (navigation path, form fields, expected result).
+3. **Never use root / direct DB access in guides or automation.** Poking
+   around in containers, editing databases directly, or using `incus exec`
+   into Aether is acceptable **only for investigation and learning** — never
+   as the documented path for users to follow.
+
+APIs improve over time. When a feature is UI-only today, note it and check
+`/api/swagger.yaml` (or equivalent) in future versions before assuming it's
+still UI-only. The API endpoint may have been added since the guide was written.
+
 ## Coding conventions
 
 - **Shell**: bash with `set -euo pipefail`

ansible/playbooks/post-deploy.yml

@@ -17,7 +17,9 @@
 # Incus cluster API at the node level.
 #
 # Requirements:
-#   - Incus client cert/key at /var/lib/awx/projects/incus-contrib/
+#   - Incus client cert/key pushed to AWX task pod at
+#     /var/lib/awx/projects/incus-contrib/ (mounted as /runner/project/
+#     inside the EE container during job execution)
 #   - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443)
 
 - name: Post-deploy — configure instance via Incus API

incusos/deploy-awx

@@ -31,7 +31,7 @@ AWX_MEMORY="8GiB"
 AWX_DISK="40GiB"
 GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git"
 GIT_BRANCH="master"
-PLAYBOOK_DIR="ansible/playbooks"
+PLAYBOOK_DIR="playbooks"
 AETHER_URL="https://192.168.102.160:8443"
 AETHER_CLUSTER_ID="52"
 

notes/aether-guide.md

@@ -62,6 +62,70 @@ Aether. Target the node with the most available storage.
 
 Access at `https://<IP>:8443`. Default credentials: admin / (set during setup).
 
+### REST API
+
+Aether provides a documented REST API with OpenAPI 3.0 spec.
+
+| Resource | URL |
+|----------|-----|
+| Swagger UI | `https://<IP>:8443/api/docs` |
+| OpenAPI spec | `https://<IP>:8443/api/swagger.yaml` |
+| Auth endpoint | `POST /api/auth/token` |
+
+**Authentication**: JWT Bearer token. Obtain via username/password:
+
+```bash
+# Get a JWT (valid 24 hours)
+AETHER_TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
+    -H "Content-Type: application/json" \
+    -d '{"username":"admin","password":"YOUR_PASSWORD"}' \
+    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
+
+# Use the token
+curl -sk https://192.168.102.160:8443/api/clusters \
+    -H "Authorization: Bearer $AETHER_TOKEN"
+```
+
+**JWT payload** includes: `sub` (username), `id` (user ID),
+`is_ffsnd_full_admin` (bool), `is_ffsnd_acl_admin` (bool), `exp` (expiry).
+
+**Rate limits**: 3 req/s on auth endpoint, 10 req/s on all others.
+Exceeding returns HTTP 429.
+
+**API coverage** (v6.4.317, API v1.0.0):
+
+| Tag | Endpoints | Description |
+|-----|-----------|-------------|
+| Authentication | `POST /api/auth/token` | JWT token issuance |
+| Clusters | `GET /api/clusters` | List clusters (admin only) |
+| Local ACLs | `GET/POST /api/clusters/{id}/rules` | Per-cluster firewall rules |
+| Local ACLs | `PUT/DELETE /api/clusters/{id}/rules/{rule_id}` | Update/delete rules |
+| Global ACLs | `GET/POST /api/global/rules` | Global firewall rules |
+| Global ACLs | `PUT/DELETE /api/global/rules/{rule_id}` | Update/delete global rules |
+
+Features **not yet in the API** (UI-only as of v6.4.317):
+- AWX endpoint registration and cluster AWX binding
+- Instance deployment and management
+- Blueprint design and deployment
+- Inventory, networking, OVN management
+- HAProxy load balancers
+- RBAC user management
+- Health checks
+
+The API is actively developed. Check `/api/swagger.yaml` in newer Aether
+versions for additional endpoints — features may move from UI-only to API.
+
+**ACL rule schema** (source/destination support):
+- IP addresses: `192.168.1.1`
+- CIDR: `10.0.0.0/8`
+- Metadata expressions: `user.key=value`, `user.key=^prefix`, `user.key=*substring`
+- Boolean logic: `(user.env=prod AND user.tier=web) OR user.region=^eu`
+- Comma-separated: `192.168.1.1,user.app=web`
+- Protocols: `tcp`, `udp`, `icmp4`
+- Actions: `allow`, `drop`, `reject`
+- Direction: `ingress`, `egress`, `in_out`
+- Apply to: `instances`, `dfw_ovn`
+
 ### Post-deploy script
 
 `/home/ffsdn/post_deploy.sh` performs 7 steps:

notes/awx-guide.md

@@ -84,6 +84,12 @@ finished with status: failed. Instance has been deleted."
 
 ## Lab deployment
 
+**Automated path**: the `incusos/deploy-awx` script automates the entire
+deployment and configuration process. Run `deploy-awx --deploy` for a full
+install, `deploy-awx --configure` to set up project/templates, and
+`deploy-awx --status` to check health. The manual steps below are for
+reference, troubleshooting, and understanding what the script does.
+
 ### Resource requirements
 
 | Resource | Recommended | Minimum |
@@ -211,11 +217,14 @@ Key settings discovered during testing:
 - **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access)
 
 ```bash
-# Push manifests to VM and apply
+# Push manifests to VM
 incus exec oc-node-02:awx -- mkdir -p /opt/awx/base
+incus file push incusos/awx-manifests/base/awx.yaml \
+    oc-node-02:awx/opt/awx/base/awx.yaml
+incus file push incusos/awx-manifests/base/kustomization.yaml \
+    oc-node-02:awx/opt/awx/base/kustomization.yaml
 
-# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/)
+# Apply
-# Then apply:
 incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/
 
 # Wait 5-10 min for all pods to start
@@ -224,12 +233,16 @@ incus exec oc-node-02:awx -- kubectl -n awx get pods -w
 
 Expected final pod state:
 ```
+awx-migration-24.6.1-xxx              0/1     Completed
 awx-operator-controller-manager-xxx   2/2     Running
 awx-postgres-15-0                     1/1     Running
 awx-task-xxx                          4/4     Running
 awx-web-xxx                           3/3     Running
 ```
 
+The migration pod runs once during initial deployment and shows `Completed`
+status — this is expected.
+
 #### Step 6: Verify
 
 ```bash
@@ -328,35 +341,56 @@ curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
 
 ### Job templates
 
-| Template | Playbook | Template ID |
+Both must have `ask_variables_on_launch: true` so Aether can pass `ffsdn_*`
-|----------|----------|-------------|
+variables at launch time.
-| `post-deploy` | `playbooks/post-deploy.yml` | 10 |
-| `decommission` | `playbooks/decommission.yml` | 11 |
-
-Both must have `ask_variables_on_launch: true`:
 
+**Look up resource IDs first** — they vary by installation:
 ```bash
+# Find your project, inventory, and credential IDs
+curl -sk http://192.168.102.161:30080/api/v2/projects/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; [print(f'{p[\"id\"]:3d} {p[\"name\"]}') \
+        for p in json.load(sys.stdin)['results']]"
+
+curl -sk http://192.168.102.161:30080/api/v2/inventories/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; [print(f'{i[\"id\"]:3d} {i[\"name\"]}') \
+        for i in json.load(sys.stdin)['results']]"
+
+curl -sk http://192.168.102.161:30080/api/v2/credentials/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; [print(f'{c[\"id\"]:3d} {c[\"name\"]}') \
+        for c in json.load(sys.stdin)['results']]"
+```
+
+Create templates using the IDs from above (example uses lab values):
+```bash
+# Replace PROJECT_ID, INVENTORY_ID with values from the lookups above
 curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
     -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
         "name": "post-deploy",
         "organization": 1,
-        "project": 6,
+        "project": PROJECT_ID,
         "playbook": "playbooks/post-deploy.yml",
-        "inventory": 2,
+        "inventory": INVENTORY_ID,
         "ask_variables_on_launch": true
     }'
 ```
 
 **Attach credentials** to templates (required even if playbook doesn't use SSH):
 ```bash
-curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \
+# Replace TEMPLATE_ID and CREDENTIAL_ID with actual values
+curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/TEMPLATE_ID/credentials/ \
     -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
-    -d '{"id": 4}'
+    -d '{"id": CREDENTIAL_ID}'
 ```
 
+In the lab, the current IDs are: project=9, inventory=2, credential=4,
+post-deploy template=10, decommission template=11.
+
 ### Personal Access Token
 
 ```bash
@@ -369,10 +403,28 @@ AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/persona
 
 ## Aether integration
 
-### Register AWX endpoint
+**Automated path**: `deploy-awx --join-aether` creates a PAT and prints
+the Aether registration details (endpoint URL, token, template IDs).
+Then complete registration via the Aether UI below.
 
-Via Aether API (requires session cookies + CSRF token):
+### Step 1: Register AWX endpoint
 
+1. Log into Aether at `https://192.168.102.160:8443`
+2. Navigate to **Ansible Automation** (`/awx-endpoints`)
+3. Add a new endpoint:
+   - **Name**: `lab-awx`
+   - **URL**: `http://192.168.102.161:30080`
+   - **Token**: the AWX PAT (from the PAT creation step above, or
+     from `deploy-awx --join-aether` output)
+   - **Verify SSL**: unchecked (AWX is HTTP on NodePort)
+
+After saving, Aether checks AWX health automatically. Verify at
+`/api/health/awx`:
+```json
+{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
+```
+
+The API equivalent (requires session cookies + CSRF token):
 ```bash
 curl -sSk -b /tmp/aether-cookies.txt \
     -H "X-CSRF-Token: $CSRF" \
@@ -387,41 +439,35 @@ curl -sSk -b /tmp/aether-cookies.txt \
     }'
 ```
 
-**Tested result**: endpoint ID 2 created, AWX health shows "healthy"
+### Step 2: Configure AWX template IDs
-at `/api/health/awx`:
-```json
-{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
-```
 
-### Bind cluster to AWX
+AWX template IDs can be configured at two levels:
 
-**Known bug**: `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid
+**Per-blueprint** (recommended): In **Blueprint Design** (`/blueprintdesign`),
-cluster ID"}` for all valid cluster IDs. This Aether API endpoint appears
+each blueprint has optional fields for **AWX Post-Deploy Template ID** and
-non-functional in v6.4.317.
+**AWX Decommission Template ID**. Set these to the template IDs from the
+"Job templates" section above.
 
-**Workaround**: direct PostgreSQL update on the Aether database:
+**Per-cluster** (global default): In **Manage INCUS Clusters** (`/incus-infra`),
-```bash
+select the cluster → **Settings** tab. Configure the AWX endpoint and
-incus exec oc-node-01:aether -- bash -c "
+default template IDs here. These apply to all deploys on this cluster
-    docker exec aether-postgres psql -U aether -d incusovnsdnc -c \"
+unless overridden by blueprint-level settings.
-        UPDATE incus_ovn_clusters SET
+
-            awx_endpoint_id = 2,
+**Note**: The `PUT /api/clusters/{id}/awx-config` API endpoint was
-            awx_post_deploy_template_id = 10,
+non-functional in Aether v6.4.317 (returned "Invalid cluster ID").
-            awx_decommission_template_id = 11,
+Use the UI for cluster-level configuration.
-            awx_job_timeout_seconds = 600
-        WHERE cluster_id = 52;
-    \"
-"
-```
 
 ### Verify integration
 
+Deploy a test instance via Aether. If AWX is configured, Aether
+automatically triggers the post-deploy job template after instance creation.
+
 ```bash
-# Check Aether's AWX health endpoint
+# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
+
+# Check AWX health from Aether's perspective
 curl -sSk -b /tmp/aether-cookies.txt \
     https://192.168.102.160:8443/api/health/awx
-
-# Deploy a test instance via Aether → AWX job triggers automatically
-# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
 ```
 
 ## Writing playbooks
@@ -651,7 +697,8 @@ exec endpoints that work regardless of instance network topology.
 `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}`
 for all valid cluster IDs. Tested with different CSRF tokens, session
 cookies, request formats (JSON fields, POST vs PUT). The endpoint is not
-documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE.
+documented in Aether's swagger.yaml. Use the Aether UI (cluster Settings
+tab or per-blueprint configuration) instead.
 
 ### AWX web pod OOMKill at 1 GiB
 
@@ -740,7 +787,7 @@ curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
     | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
         for j in json.load(sys.stdin)['results']]"
 
-# Manually trigger post-deploy
+# Manually trigger post-deploy (replace 10 with your template ID)
 curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
     -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \