From 0100182dd834489d5f8e8b3ed379da60f28e9065 Mon Sep 17 00:00:00 2001 From: Maarten Date: Tue, 24 Feb 2026 10:33:46 +0100 Subject: [PATCH] Validate AWX guide, document Aether API, enforce API-first policy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - AWX guide: fix wrong project ID (6→9), add dynamic ID lookups, add missing Step 5 push commands, add migration pod to expected output, replace PostgreSQL workaround with Aether UI instructions, reference deploy-awx as automated path - Aether guide: document JWT REST API (auth, clusters, ACLs), swagger UI at /api/docs, rate limits, coverage vs UI-only features - CLAUDE.md: add Aether API capability, add interaction hierarchy (API first → UI fallback → never root/DB in guides) - deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks" - post-deploy.yml: clarify cert push path vs EE runtime mount path Co-Authored-By: Claude Opus 4.6 --- .claude/rules/awx-integration.md | 3 +- CLAUDE.md | 51 +++++++++++- ansible/playbooks/post-deploy.yml | 4 +- incusos/deploy-awx | 2 +- notes/aether-guide.md | 64 ++++++++++++++ notes/awx-guide.md | 133 ++++++++++++++++++++---------- 6 files changed, 210 insertions(+), 47 deletions(-) diff --git a/.claude/rules/awx-integration.md b/.claude/rules/awx-integration.md index dd258a9..42171b5 100644 --- a/.claude/rules/awx-integration.md +++ b/.claude/rules/awx-integration.md @@ -51,6 +51,7 @@ It does NOT pass: `vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`. infinite recursion. Use `ffsdn_*` vars directly. - **`environment` is reserved**: resolves to `[]` as extra var. - **AWX config API bug**: `PUT /api/clusters/{id}/awx-config` returns - "Invalid cluster ID". Workaround: direct PostgreSQL UPDATE. + "Invalid cluster ID" (v6.4.317). Use the Aether UI instead (cluster + Settings tab or per-blueprint AWX template IDs). - **Lifecycle hooks**: post-deploy failure -> auto-rollback (instance deleted). Decommission failure does NOT block deletion. diff --git a/CLAUDE.md b/CLAUDE.md index d657382..8b12ac0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,7 +35,7 @@ incus-contrib/ │ ├── proxmox.yaml # Proxmox connection (gitignored) │ ├── TESTING.md │ └── examples/ # Example seed + Proxmox YAML files -├── env # PROXMOX_TOKEN_SECRET + PROXMOX_ROOT_PASSWORD (gitignored) +├── env # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored) └── notes/ # Reference guides (clustering, networking, OC, AWX, etc.) ``` @@ -57,6 +57,37 @@ incusos/helpers/proxmox-api GET /nodes/pve/status --json incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop ``` +### Aether API +Aether has a documented REST API at `https://192.168.102.160:8443/api/`. + +- **Swagger UI**: `https://192.168.102.160:8443/api/docs` +- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml` +- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}` + Credentials in `env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h. + +```bash +# Get JWT +TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \ + -H "Content-Type: application/json" \ + -d "{\"username\":\"admin\",\"password\":\"$AETHER_ADMIN_PASSWORD\"}" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") + +# Use it +curl -sk https://192.168.102.160:8443/api/clusters \ + -H "Authorization: Bearer $TOKEN" +``` + +Current API coverage (v6.4.317, API v1.0.0): +- Authentication: `POST /api/auth/token` +- Clusters: `GET /api/clusters` +- Local ACLs: CRUD on `/api/clusters/{id}/rules` +- Global ACLs: CRUD on `/api/global/rules` + +Not yet in the API: AWX endpoints, deploys, blueprints, instance management, +health checks. These are available through the web UI only. The API is +actively being developed — check `/api/swagger.yaml` for new endpoints +in future Aether versions. + ### Live AWX job output Capture output while a job is running (don't just poll status): ```bash @@ -72,6 +103,24 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \ - **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition. - **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters). +## Interacting with external systems (Aether, AWX, Incus, etc.) + +**Strict priority order** — follow this hierarchy in all guides, scripts, +and automation flows: + +1. **API first**: if the system has a documented REST API, use it. + Authenticate as a regular user (JWT, PAT), never as root / DB superuser. +2. **Web UI fallback**: if the API doesn't cover a feature yet, document + the UI steps (navigation path, form fields, expected result). +3. **Never use root / direct DB access in guides or automation.** Poking + around in containers, editing databases directly, or using `incus exec` + into Aether is acceptable **only for investigation and learning** — never + as the documented path for users to follow. + +APIs improve over time. When a feature is UI-only today, note it and check +`/api/swagger.yaml` (or equivalent) in future versions before assuming it's +still UI-only. The API endpoint may have been added since the guide was written. + ## Coding conventions - **Shell**: bash with `set -euo pipefail` diff --git a/ansible/playbooks/post-deploy.yml b/ansible/playbooks/post-deploy.yml index b42181c..a5d6fd6 100644 --- a/ansible/playbooks/post-deploy.yml +++ b/ansible/playbooks/post-deploy.yml @@ -17,7 +17,9 @@ # Incus cluster API at the node level. # # Requirements: -# - Incus client cert/key at /var/lib/awx/projects/incus-contrib/ +# - Incus client cert/key pushed to AWX task pod at +# /var/lib/awx/projects/incus-contrib/ (mounted as /runner/project/ +# inside the EE container during job execution) # - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443) - name: Post-deploy — configure instance via Incus API diff --git a/incusos/deploy-awx b/incusos/deploy-awx index 37e3613..771de01 100755 --- a/incusos/deploy-awx +++ b/incusos/deploy-awx @@ -31,7 +31,7 @@ AWX_MEMORY="8GiB" AWX_DISK="40GiB" GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git" GIT_BRANCH="master" -PLAYBOOK_DIR="ansible/playbooks" +PLAYBOOK_DIR="playbooks" AETHER_URL="https://192.168.102.160:8443" AETHER_CLUSTER_ID="52" diff --git a/notes/aether-guide.md b/notes/aether-guide.md index 7d66428..09ba8b1 100644 --- a/notes/aether-guide.md +++ b/notes/aether-guide.md @@ -62,6 +62,70 @@ Aether. Target the node with the most available storage. Access at `https://:8443`. Default credentials: admin / (set during setup). +### REST API + +Aether provides a documented REST API with OpenAPI 3.0 spec. + +| Resource | URL | +|----------|-----| +| Swagger UI | `https://:8443/api/docs` | +| OpenAPI spec | `https://:8443/api/swagger.yaml` | +| Auth endpoint | `POST /api/auth/token` | + +**Authentication**: JWT Bearer token. Obtain via username/password: + +```bash +# Get a JWT (valid 24 hours) +AETHER_TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \ + -H "Content-Type: application/json" \ + -d '{"username":"admin","password":"YOUR_PASSWORD"}' \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") + +# Use the token +curl -sk https://192.168.102.160:8443/api/clusters \ + -H "Authorization: Bearer $AETHER_TOKEN" +``` + +**JWT payload** includes: `sub` (username), `id` (user ID), +`is_ffsnd_full_admin` (bool), `is_ffsnd_acl_admin` (bool), `exp` (expiry). + +**Rate limits**: 3 req/s on auth endpoint, 10 req/s on all others. +Exceeding returns HTTP 429. + +**API coverage** (v6.4.317, API v1.0.0): + +| Tag | Endpoints | Description | +|-----|-----------|-------------| +| Authentication | `POST /api/auth/token` | JWT token issuance | +| Clusters | `GET /api/clusters` | List clusters (admin only) | +| Local ACLs | `GET/POST /api/clusters/{id}/rules` | Per-cluster firewall rules | +| Local ACLs | `PUT/DELETE /api/clusters/{id}/rules/{rule_id}` | Update/delete rules | +| Global ACLs | `GET/POST /api/global/rules` | Global firewall rules | +| Global ACLs | `PUT/DELETE /api/global/rules/{rule_id}` | Update/delete global rules | + +Features **not yet in the API** (UI-only as of v6.4.317): +- AWX endpoint registration and cluster AWX binding +- Instance deployment and management +- Blueprint design and deployment +- Inventory, networking, OVN management +- HAProxy load balancers +- RBAC user management +- Health checks + +The API is actively developed. Check `/api/swagger.yaml` in newer Aether +versions for additional endpoints — features may move from UI-only to API. + +**ACL rule schema** (source/destination support): +- IP addresses: `192.168.1.1` +- CIDR: `10.0.0.0/8` +- Metadata expressions: `user.key=value`, `user.key=^prefix`, `user.key=*substring` +- Boolean logic: `(user.env=prod AND user.tier=web) OR user.region=^eu` +- Comma-separated: `192.168.1.1,user.app=web` +- Protocols: `tcp`, `udp`, `icmp4` +- Actions: `allow`, `drop`, `reject` +- Direction: `ingress`, `egress`, `in_out` +- Apply to: `instances`, `dfw_ovn` + ### Post-deploy script `/home/ffsdn/post_deploy.sh` performs 7 steps: diff --git a/notes/awx-guide.md b/notes/awx-guide.md index d52611d..7312154 100644 --- a/notes/awx-guide.md +++ b/notes/awx-guide.md @@ -84,6 +84,12 @@ finished with status: failed. Instance has been deleted." ## Lab deployment +**Automated path**: the `incusos/deploy-awx` script automates the entire +deployment and configuration process. Run `deploy-awx --deploy` for a full +install, `deploy-awx --configure` to set up project/templates, and +`deploy-awx --status` to check health. The manual steps below are for +reference, troubleshooting, and understanding what the script does. + ### Resource requirements | Resource | Recommended | Minimum | @@ -211,11 +217,14 @@ Key settings discovered during testing: - **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access) ```bash -# Push manifests to VM and apply +# Push manifests to VM incus exec oc-node-02:awx -- mkdir -p /opt/awx/base +incus file push incusos/awx-manifests/base/awx.yaml \ + oc-node-02:awx/opt/awx/base/awx.yaml +incus file push incusos/awx-manifests/base/kustomization.yaml \ + oc-node-02:awx/opt/awx/base/kustomization.yaml -# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/) -# Then apply: +# Apply incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/ # Wait 5-10 min for all pods to start @@ -224,12 +233,16 @@ incus exec oc-node-02:awx -- kubectl -n awx get pods -w Expected final pod state: ``` +awx-migration-24.6.1-xxx 0/1 Completed awx-operator-controller-manager-xxx 2/2 Running awx-postgres-15-0 1/1 Running awx-task-xxx 4/4 Running awx-web-xxx 3/3 Running ``` +The migration pod runs once during initial deployment and shows `Completed` +status — this is expected. + #### Step 6: Verify ```bash @@ -328,35 +341,56 @@ curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \ ### Job templates -| Template | Playbook | Template ID | -|----------|----------|-------------| -| `post-deploy` | `playbooks/post-deploy.yml` | 10 | -| `decommission` | `playbooks/decommission.yml` | 11 | - -Both must have `ask_variables_on_launch: true`: +Both must have `ask_variables_on_launch: true` so Aether can pass `ffsdn_*` +variables at launch time. +**Look up resource IDs first** — they vary by installation: ```bash +# Find your project, inventory, and credential IDs +curl -sk http://192.168.102.161:30080/api/v2/projects/ \ + -H "Authorization: Bearer $AWX_TOKEN" \ + | python3 -c "import sys,json; [print(f'{p[\"id\"]:3d} {p[\"name\"]}') \ + for p in json.load(sys.stdin)['results']]" + +curl -sk http://192.168.102.161:30080/api/v2/inventories/ \ + -H "Authorization: Bearer $AWX_TOKEN" \ + | python3 -c "import sys,json; [print(f'{i[\"id\"]:3d} {i[\"name\"]}') \ + for i in json.load(sys.stdin)['results']]" + +curl -sk http://192.168.102.161:30080/api/v2/credentials/ \ + -H "Authorization: Bearer $AWX_TOKEN" \ + | python3 -c "import sys,json; [print(f'{c[\"id\"]:3d} {c[\"name\"]}') \ + for c in json.load(sys.stdin)['results']]" +``` + +Create templates using the IDs from above (example uses lab values): +```bash +# Replace PROJECT_ID, INVENTORY_ID with values from the lookups above curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \ -H "Authorization: Bearer $AWX_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "post-deploy", "organization": 1, - "project": 6, + "project": PROJECT_ID, "playbook": "playbooks/post-deploy.yml", - "inventory": 2, + "inventory": INVENTORY_ID, "ask_variables_on_launch": true }' ``` **Attach credentials** to templates (required even if playbook doesn't use SSH): ```bash -curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \ +# Replace TEMPLATE_ID and CREDENTIAL_ID with actual values +curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/TEMPLATE_ID/credentials/ \ -H "Authorization: Bearer $AWX_TOKEN" \ -H "Content-Type: application/json" \ - -d '{"id": 4}' + -d '{"id": CREDENTIAL_ID}' ``` +In the lab, the current IDs are: project=9, inventory=2, credential=4, +post-deploy template=10, decommission template=11. + ### Personal Access Token ```bash @@ -369,10 +403,28 @@ AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/persona ## Aether integration -### Register AWX endpoint +**Automated path**: `deploy-awx --join-aether` creates a PAT and prints +the Aether registration details (endpoint URL, token, template IDs). +Then complete registration via the Aether UI below. -Via Aether API (requires session cookies + CSRF token): +### Step 1: Register AWX endpoint +1. Log into Aether at `https://192.168.102.160:8443` +2. Navigate to **Ansible Automation** (`/awx-endpoints`) +3. Add a new endpoint: + - **Name**: `lab-awx` + - **URL**: `http://192.168.102.161:30080` + - **Token**: the AWX PAT (from the PAT creation step above, or + from `deploy-awx --join-aether` output) + - **Verify SSL**: unchecked (AWX is HTTP on NodePort) + +After saving, Aether checks AWX health automatically. Verify at +`/api/health/awx`: +```json +{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"} +``` + +The API equivalent (requires session cookies + CSRF token): ```bash curl -sSk -b /tmp/aether-cookies.txt \ -H "X-CSRF-Token: $CSRF" \ @@ -387,41 +439,35 @@ curl -sSk -b /tmp/aether-cookies.txt \ }' ``` -**Tested result**: endpoint ID 2 created, AWX health shows "healthy" -at `/api/health/awx`: -```json -{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"} -``` +### Step 2: Configure AWX template IDs -### Bind cluster to AWX +AWX template IDs can be configured at two levels: -**Known bug**: `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid -cluster ID"}` for all valid cluster IDs. This Aether API endpoint appears -non-functional in v6.4.317. +**Per-blueprint** (recommended): In **Blueprint Design** (`/blueprintdesign`), +each blueprint has optional fields for **AWX Post-Deploy Template ID** and +**AWX Decommission Template ID**. Set these to the template IDs from the +"Job templates" section above. -**Workaround**: direct PostgreSQL update on the Aether database: -```bash -incus exec oc-node-01:aether -- bash -c " - docker exec aether-postgres psql -U aether -d incusovnsdnc -c \" - UPDATE incus_ovn_clusters SET - awx_endpoint_id = 2, - awx_post_deploy_template_id = 10, - awx_decommission_template_id = 11, - awx_job_timeout_seconds = 600 - WHERE cluster_id = 52; - \" -" -``` +**Per-cluster** (global default): In **Manage INCUS Clusters** (`/incus-infra`), +select the cluster → **Settings** tab. Configure the AWX endpoint and +default template IDs here. These apply to all deploys on this cluster +unless overridden by blueprint-level settings. + +**Note**: The `PUT /api/clusters/{id}/awx-config` API endpoint was +non-functional in Aether v6.4.317 (returned "Invalid cluster ID"). +Use the UI for cluster-level configuration. ### Verify integration +Deploy a test instance via Aether. If AWX is configured, Aether +automatically triggers the post-deploy job template after instance creation. + ```bash -# Check Aether's AWX health endpoint +# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs" + +# Check AWX health from Aether's perspective curl -sSk -b /tmp/aether-cookies.txt \ https://192.168.102.160:8443/api/health/awx - -# Deploy a test instance via Aether → AWX job triggers automatically -# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs" ``` ## Writing playbooks @@ -651,7 +697,8 @@ exec endpoints that work regardless of instance network topology. `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}` for all valid cluster IDs. Tested with different CSRF tokens, session cookies, request formats (JSON fields, POST vs PUT). The endpoint is not -documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE. +documented in Aether's swagger.yaml. Use the Aether UI (cluster Settings +tab or per-blueprint configuration) instead. ### AWX web pod OOMKill at 1 GiB @@ -740,7 +787,7 @@ curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \ | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \ for j in json.load(sys.stdin)['results']]" -# Manually trigger post-deploy +# Manually trigger post-deploy (replace 10 with your template ID) curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \ -H "Authorization: Bearer $AWX_TOKEN" \ -H "Content-Type: application/json" \