Validate AWX guide, document Aether API, enforce API-first policy
- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups, add missing Step 5 push commands, add migration pod to expected output, replace PostgreSQL workaround with Aether UI instructions, reference deploy-awx as automated path - Aether guide: document JWT REST API (auth, clusters, ACLs), swagger UI at /api/docs, rate limits, coverage vs UI-only features - CLAUDE.md: add Aether API capability, add interaction hierarchy (API first → UI fallback → never root/DB in guides) - deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks" - post-deploy.yml: clarify cert push path vs EE runtime mount path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
c4efcbe30d
commit
0100182dd8
|
|
@ -51,6 +51,7 @@ It does NOT pass: `vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`.
|
|||
infinite recursion. Use `ffsdn_*` vars directly.
|
||||
- **`environment` is reserved**: resolves to `[]` as extra var.
|
||||
- **AWX config API bug**: `PUT /api/clusters/{id}/awx-config` returns
|
||||
"Invalid cluster ID". Workaround: direct PostgreSQL UPDATE.
|
||||
"Invalid cluster ID" (v6.4.317). Use the Aether UI instead (cluster
|
||||
Settings tab or per-blueprint AWX template IDs).
|
||||
- **Lifecycle hooks**: post-deploy failure -> auto-rollback (instance deleted).
|
||||
Decommission failure does NOT block deletion.
|
||||
|
|
|
|||
51
CLAUDE.md
51
CLAUDE.md
|
|
@ -35,7 +35,7 @@ incus-contrib/
|
|||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||
│ ├── TESTING.md
|
||||
│ └── examples/ # Example seed + Proxmox YAML files
|
||||
├── env # PROXMOX_TOKEN_SECRET + PROXMOX_ROOT_PASSWORD (gitignored)
|
||||
├── env # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored)
|
||||
└── notes/ # Reference guides (clustering, networking, OC, AWX, etc.)
|
||||
```
|
||||
|
||||
|
|
@ -57,6 +57,37 @@ incusos/helpers/proxmox-api GET /nodes/pve/status --json
|
|||
incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
|
||||
```
|
||||
|
||||
### Aether API
|
||||
Aether has a documented REST API at `https://192.168.102.160:8443/api/`.
|
||||
|
||||
- **Swagger UI**: `https://192.168.102.160:8443/api/docs`
|
||||
- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
|
||||
- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
|
||||
Credentials in `env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
|
||||
|
||||
```bash
|
||||
# Get JWT
|
||||
TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"username\":\"admin\",\"password\":\"$AETHER_ADMIN_PASSWORD\"}" \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
|
||||
|
||||
# Use it
|
||||
curl -sk https://192.168.102.160:8443/api/clusters \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
Current API coverage (v6.4.317, API v1.0.0):
|
||||
- Authentication: `POST /api/auth/token`
|
||||
- Clusters: `GET /api/clusters`
|
||||
- Local ACLs: CRUD on `/api/clusters/{id}/rules`
|
||||
- Global ACLs: CRUD on `/api/global/rules`
|
||||
|
||||
Not yet in the API: AWX endpoints, deploys, blueprints, instance management,
|
||||
health checks. These are available through the web UI only. The API is
|
||||
actively being developed — check `/api/swagger.yaml` for new endpoints
|
||||
in future Aether versions.
|
||||
|
||||
### Live AWX job output
|
||||
Capture output while a job is running (don't just poll status):
|
||||
```bash
|
||||
|
|
@ -72,6 +103,24 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
|
|||
- **No `force_reboot` in seeds for Proxmox** -- causes crontab race condition.
|
||||
- **VMID ranges**: 400-499 (OC), 800-809 (single), 900-939 (clusters).
|
||||
|
||||
## Interacting with external systems (Aether, AWX, Incus, etc.)
|
||||
|
||||
**Strict priority order** — follow this hierarchy in all guides, scripts,
|
||||
and automation flows:
|
||||
|
||||
1. **API first**: if the system has a documented REST API, use it.
|
||||
Authenticate as a regular user (JWT, PAT), never as root / DB superuser.
|
||||
2. **Web UI fallback**: if the API doesn't cover a feature yet, document
|
||||
the UI steps (navigation path, form fields, expected result).
|
||||
3. **Never use root / direct DB access in guides or automation.** Poking
|
||||
around in containers, editing databases directly, or using `incus exec`
|
||||
into Aether is acceptable **only for investigation and learning** — never
|
||||
as the documented path for users to follow.
|
||||
|
||||
APIs improve over time. When a feature is UI-only today, note it and check
|
||||
`/api/swagger.yaml` (or equivalent) in future versions before assuming it's
|
||||
still UI-only. The API endpoint may have been added since the guide was written.
|
||||
|
||||
## Coding conventions
|
||||
|
||||
- **Shell**: bash with `set -euo pipefail`
|
||||
|
|
|
|||
|
|
@ -17,7 +17,9 @@
|
|||
# Incus cluster API at the node level.
|
||||
#
|
||||
# Requirements:
|
||||
# - Incus client cert/key at /var/lib/awx/projects/incus-contrib/
|
||||
# - Incus client cert/key pushed to AWX task pod at
|
||||
# /var/lib/awx/projects/incus-contrib/ (mounted as /runner/project/
|
||||
# inside the EE container during job execution)
|
||||
# - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443)
|
||||
|
||||
- name: Post-deploy — configure instance via Incus API
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ AWX_MEMORY="8GiB"
|
|||
AWX_DISK="40GiB"
|
||||
GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git"
|
||||
GIT_BRANCH="master"
|
||||
PLAYBOOK_DIR="ansible/playbooks"
|
||||
PLAYBOOK_DIR="playbooks"
|
||||
AETHER_URL="https://192.168.102.160:8443"
|
||||
AETHER_CLUSTER_ID="52"
|
||||
|
||||
|
|
|
|||
|
|
@ -62,6 +62,70 @@ Aether. Target the node with the most available storage.
|
|||
|
||||
Access at `https://<IP>:8443`. Default credentials: admin / (set during setup).
|
||||
|
||||
### REST API
|
||||
|
||||
Aether provides a documented REST API with OpenAPI 3.0 spec.
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| Swagger UI | `https://<IP>:8443/api/docs` |
|
||||
| OpenAPI spec | `https://<IP>:8443/api/swagger.yaml` |
|
||||
| Auth endpoint | `POST /api/auth/token` |
|
||||
|
||||
**Authentication**: JWT Bearer token. Obtain via username/password:
|
||||
|
||||
```bash
|
||||
# Get a JWT (valid 24 hours)
|
||||
AETHER_TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username":"admin","password":"YOUR_PASSWORD"}' \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
|
||||
|
||||
# Use the token
|
||||
curl -sk https://192.168.102.160:8443/api/clusters \
|
||||
-H "Authorization: Bearer $AETHER_TOKEN"
|
||||
```
|
||||
|
||||
**JWT payload** includes: `sub` (username), `id` (user ID),
|
||||
`is_ffsnd_full_admin` (bool), `is_ffsnd_acl_admin` (bool), `exp` (expiry).
|
||||
|
||||
**Rate limits**: 3 req/s on auth endpoint, 10 req/s on all others.
|
||||
Exceeding returns HTTP 429.
|
||||
|
||||
**API coverage** (v6.4.317, API v1.0.0):
|
||||
|
||||
| Tag | Endpoints | Description |
|
||||
|-----|-----------|-------------|
|
||||
| Authentication | `POST /api/auth/token` | JWT token issuance |
|
||||
| Clusters | `GET /api/clusters` | List clusters (admin only) |
|
||||
| Local ACLs | `GET/POST /api/clusters/{id}/rules` | Per-cluster firewall rules |
|
||||
| Local ACLs | `PUT/DELETE /api/clusters/{id}/rules/{rule_id}` | Update/delete rules |
|
||||
| Global ACLs | `GET/POST /api/global/rules` | Global firewall rules |
|
||||
| Global ACLs | `PUT/DELETE /api/global/rules/{rule_id}` | Update/delete global rules |
|
||||
|
||||
Features **not yet in the API** (UI-only as of v6.4.317):
|
||||
- AWX endpoint registration and cluster AWX binding
|
||||
- Instance deployment and management
|
||||
- Blueprint design and deployment
|
||||
- Inventory, networking, OVN management
|
||||
- HAProxy load balancers
|
||||
- RBAC user management
|
||||
- Health checks
|
||||
|
||||
The API is actively developed. Check `/api/swagger.yaml` in newer Aether
|
||||
versions for additional endpoints — features may move from UI-only to API.
|
||||
|
||||
**ACL rule schema** (source/destination support):
|
||||
- IP addresses: `192.168.1.1`
|
||||
- CIDR: `10.0.0.0/8`
|
||||
- Metadata expressions: `user.key=value`, `user.key=^prefix`, `user.key=*substring`
|
||||
- Boolean logic: `(user.env=prod AND user.tier=web) OR user.region=^eu`
|
||||
- Comma-separated: `192.168.1.1,user.app=web`
|
||||
- Protocols: `tcp`, `udp`, `icmp4`
|
||||
- Actions: `allow`, `drop`, `reject`
|
||||
- Direction: `ingress`, `egress`, `in_out`
|
||||
- Apply to: `instances`, `dfw_ovn`
|
||||
|
||||
### Post-deploy script
|
||||
|
||||
`/home/ffsdn/post_deploy.sh` performs 7 steps:
|
||||
|
|
|
|||
|
|
@ -84,6 +84,12 @@ finished with status: failed. Instance has been deleted."
|
|||
|
||||
## Lab deployment
|
||||
|
||||
**Automated path**: the `incusos/deploy-awx` script automates the entire
|
||||
deployment and configuration process. Run `deploy-awx --deploy` for a full
|
||||
install, `deploy-awx --configure` to set up project/templates, and
|
||||
`deploy-awx --status` to check health. The manual steps below are for
|
||||
reference, troubleshooting, and understanding what the script does.
|
||||
|
||||
### Resource requirements
|
||||
|
||||
| Resource | Recommended | Minimum |
|
||||
|
|
@ -211,11 +217,14 @@ Key settings discovered during testing:
|
|||
- **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access)
|
||||
|
||||
```bash
|
||||
# Push manifests to VM and apply
|
||||
# Push manifests to VM
|
||||
incus exec oc-node-02:awx -- mkdir -p /opt/awx/base
|
||||
incus file push incusos/awx-manifests/base/awx.yaml \
|
||||
oc-node-02:awx/opt/awx/base/awx.yaml
|
||||
incus file push incusos/awx-manifests/base/kustomization.yaml \
|
||||
oc-node-02:awx/opt/awx/base/kustomization.yaml
|
||||
|
||||
# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/)
|
||||
# Then apply:
|
||||
# Apply
|
||||
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/
|
||||
|
||||
# Wait 5-10 min for all pods to start
|
||||
|
|
@ -224,12 +233,16 @@ incus exec oc-node-02:awx -- kubectl -n awx get pods -w
|
|||
|
||||
Expected final pod state:
|
||||
```
|
||||
awx-migration-24.6.1-xxx 0/1 Completed
|
||||
awx-operator-controller-manager-xxx 2/2 Running
|
||||
awx-postgres-15-0 1/1 Running
|
||||
awx-task-xxx 4/4 Running
|
||||
awx-web-xxx 3/3 Running
|
||||
```
|
||||
|
||||
The migration pod runs once during initial deployment and shows `Completed`
|
||||
status — this is expected.
|
||||
|
||||
#### Step 6: Verify
|
||||
|
||||
```bash
|
||||
|
|
@ -328,35 +341,56 @@ curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
|
|||
|
||||
### Job templates
|
||||
|
||||
| Template | Playbook | Template ID |
|
||||
|----------|----------|-------------|
|
||||
| `post-deploy` | `playbooks/post-deploy.yml` | 10 |
|
||||
| `decommission` | `playbooks/decommission.yml` | 11 |
|
||||
|
||||
Both must have `ask_variables_on_launch: true`:
|
||||
Both must have `ask_variables_on_launch: true` so Aether can pass `ffsdn_*`
|
||||
variables at launch time.
|
||||
|
||||
**Look up resource IDs first** — they vary by installation:
|
||||
```bash
|
||||
# Find your project, inventory, and credential IDs
|
||||
curl -sk http://192.168.102.161:30080/api/v2/projects/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
| python3 -c "import sys,json; [print(f'{p[\"id\"]:3d} {p[\"name\"]}') \
|
||||
for p in json.load(sys.stdin)['results']]"
|
||||
|
||||
curl -sk http://192.168.102.161:30080/api/v2/inventories/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
| python3 -c "import sys,json; [print(f'{i[\"id\"]:3d} {i[\"name\"]}') \
|
||||
for i in json.load(sys.stdin)['results']]"
|
||||
|
||||
curl -sk http://192.168.102.161:30080/api/v2/credentials/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
| python3 -c "import sys,json; [print(f'{c[\"id\"]:3d} {c[\"name\"]}') \
|
||||
for c in json.load(sys.stdin)['results']]"
|
||||
```
|
||||
|
||||
Create templates using the IDs from above (example uses lab values):
|
||||
```bash
|
||||
# Replace PROJECT_ID, INVENTORY_ID with values from the lookups above
|
||||
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "post-deploy",
|
||||
"organization": 1,
|
||||
"project": 6,
|
||||
"project": PROJECT_ID,
|
||||
"playbook": "playbooks/post-deploy.yml",
|
||||
"inventory": 2,
|
||||
"inventory": INVENTORY_ID,
|
||||
"ask_variables_on_launch": true
|
||||
}'
|
||||
```
|
||||
|
||||
**Attach credentials** to templates (required even if playbook doesn't use SSH):
|
||||
```bash
|
||||
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \
|
||||
# Replace TEMPLATE_ID and CREDENTIAL_ID with actual values
|
||||
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/TEMPLATE_ID/credentials/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"id": 4}'
|
||||
-d '{"id": CREDENTIAL_ID}'
|
||||
```
|
||||
|
||||
In the lab, the current IDs are: project=9, inventory=2, credential=4,
|
||||
post-deploy template=10, decommission template=11.
|
||||
|
||||
### Personal Access Token
|
||||
|
||||
```bash
|
||||
|
|
@ -369,10 +403,28 @@ AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/persona
|
|||
|
||||
## Aether integration
|
||||
|
||||
### Register AWX endpoint
|
||||
**Automated path**: `deploy-awx --join-aether` creates a PAT and prints
|
||||
the Aether registration details (endpoint URL, token, template IDs).
|
||||
Then complete registration via the Aether UI below.
|
||||
|
||||
Via Aether API (requires session cookies + CSRF token):
|
||||
### Step 1: Register AWX endpoint
|
||||
|
||||
1. Log into Aether at `https://192.168.102.160:8443`
|
||||
2. Navigate to **Ansible Automation** (`/awx-endpoints`)
|
||||
3. Add a new endpoint:
|
||||
- **Name**: `lab-awx`
|
||||
- **URL**: `http://192.168.102.161:30080`
|
||||
- **Token**: the AWX PAT (from the PAT creation step above, or
|
||||
from `deploy-awx --join-aether` output)
|
||||
- **Verify SSL**: unchecked (AWX is HTTP on NodePort)
|
||||
|
||||
After saving, Aether checks AWX health automatically. Verify at
|
||||
`/api/health/awx`:
|
||||
```json
|
||||
{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
|
||||
```
|
||||
|
||||
The API equivalent (requires session cookies + CSRF token):
|
||||
```bash
|
||||
curl -sSk -b /tmp/aether-cookies.txt \
|
||||
-H "X-CSRF-Token: $CSRF" \
|
||||
|
|
@ -387,41 +439,35 @@ curl -sSk -b /tmp/aether-cookies.txt \
|
|||
}'
|
||||
```
|
||||
|
||||
**Tested result**: endpoint ID 2 created, AWX health shows "healthy"
|
||||
at `/api/health/awx`:
|
||||
```json
|
||||
{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
|
||||
```
|
||||
### Step 2: Configure AWX template IDs
|
||||
|
||||
### Bind cluster to AWX
|
||||
AWX template IDs can be configured at two levels:
|
||||
|
||||
**Known bug**: `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid
|
||||
cluster ID"}` for all valid cluster IDs. This Aether API endpoint appears
|
||||
non-functional in v6.4.317.
|
||||
**Per-blueprint** (recommended): In **Blueprint Design** (`/blueprintdesign`),
|
||||
each blueprint has optional fields for **AWX Post-Deploy Template ID** and
|
||||
**AWX Decommission Template ID**. Set these to the template IDs from the
|
||||
"Job templates" section above.
|
||||
|
||||
**Workaround**: direct PostgreSQL update on the Aether database:
|
||||
```bash
|
||||
incus exec oc-node-01:aether -- bash -c "
|
||||
docker exec aether-postgres psql -U aether -d incusovnsdnc -c \"
|
||||
UPDATE incus_ovn_clusters SET
|
||||
awx_endpoint_id = 2,
|
||||
awx_post_deploy_template_id = 10,
|
||||
awx_decommission_template_id = 11,
|
||||
awx_job_timeout_seconds = 600
|
||||
WHERE cluster_id = 52;
|
||||
\"
|
||||
"
|
||||
```
|
||||
**Per-cluster** (global default): In **Manage INCUS Clusters** (`/incus-infra`),
|
||||
select the cluster → **Settings** tab. Configure the AWX endpoint and
|
||||
default template IDs here. These apply to all deploys on this cluster
|
||||
unless overridden by blueprint-level settings.
|
||||
|
||||
**Note**: The `PUT /api/clusters/{id}/awx-config` API endpoint was
|
||||
non-functional in Aether v6.4.317 (returned "Invalid cluster ID").
|
||||
Use the UI for cluster-level configuration.
|
||||
|
||||
### Verify integration
|
||||
|
||||
Deploy a test instance via Aether. If AWX is configured, Aether
|
||||
automatically triggers the post-deploy job template after instance creation.
|
||||
|
||||
```bash
|
||||
# Check Aether's AWX health endpoint
|
||||
# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
|
||||
|
||||
# Check AWX health from Aether's perspective
|
||||
curl -sSk -b /tmp/aether-cookies.txt \
|
||||
https://192.168.102.160:8443/api/health/awx
|
||||
|
||||
# Deploy a test instance via Aether → AWX job triggers automatically
|
||||
# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
|
||||
```
|
||||
|
||||
## Writing playbooks
|
||||
|
|
@ -651,7 +697,8 @@ exec endpoints that work regardless of instance network topology.
|
|||
`PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}`
|
||||
for all valid cluster IDs. Tested with different CSRF tokens, session
|
||||
cookies, request formats (JSON fields, POST vs PUT). The endpoint is not
|
||||
documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE.
|
||||
documented in Aether's swagger.yaml. Use the Aether UI (cluster Settings
|
||||
tab or per-blueprint configuration) instead.
|
||||
|
||||
### AWX web pod OOMKill at 1 GiB
|
||||
|
||||
|
|
@ -740,7 +787,7 @@ curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
|
|||
| python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
|
||||
for j in json.load(sys.stdin)['results']]"
|
||||
|
||||
# Manually trigger post-deploy
|
||||
# Manually trigger post-deploy (replace 10 with your template ID)
|
||||
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
|
||||
-H "Authorization: Bearer $AWX_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
|
|
|
|||
Loading…
Reference in New Issue