incus-contrib/notes/awx-guide.md

# AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST
API for running Ansible playbooks. Aether integrates with AWX to run
post-deploy and decommission playbooks as lifecycle hooks on every instance
create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to
Aether, writing lifecycle playbooks, and troubleshooting.

## Architecture

```
  User deploys instance via Aether UI
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: ffsdn_instance_name, ffsdn_instance_ip, ...
          |
          v
  AWX runs post-deploy.yml playbook
    1. Pushes setup script to instance (Incus file API)
    2. Executes script (Incus exec API)
    3. Verifies /etc/deploy-info was written
    4. Logs to deployment ledger
          |
          v
  Aether marks deployment complete (or rolls back on failure)
```

Decommission works in reverse — Aether triggers the decommission template
before deleting the instance. Decommission failures do NOT block deletion.

### Separation of concerns

| Layer | Tool | Responsibility |
|-------|------|----------------|
| Infrastructure | Aether + Incus | Instance creation, networking, resources, lifecycle |
| Configuration | AWX + Ansible | OS config, packages, services, integrations |
| Automation glue | Aether AWX binding | Triggers playbooks at create/delete |

### What Aether passes to AWX (validated)

Every AWX job launched by Aether receives these `ffsdn_` prefixed extra vars:

| Variable | Description | Example |
|----------|-------------|---------|
| `ffsdn_instance_name` | Instance name | `awx-e2e-test` |
| `ffsdn_instance_ip` | IP address assigned by Aether | `10.207.217.5` |
| `ffsdn_cluster_id` | Numeric cluster ID | `52` |
| `ffsdn_cluster_name` | Cluster display name | `oc-lab-cluster` |
| `ffsdn_deployed_by` | Aether username who triggered deploy | `admin` |
| `ffsdn_image_os` | Image OS | `Debian` |
| `ffsdn_image_release` | Image release | `bookworm` |
| `ffsdn_image_alias` | Image alias (may be empty) | `""` |

**Important**: Aether does NOT pass `vm_name`, `vm_ip`, `environment`,
`owner`, or `cost_center`. The original plan assumed these names, but
real testing revealed Aether uses the `ffsdn_` prefix exclusively.

### AWX API endpoints Aether uses

| Endpoint | Purpose |
|----------|---------|
| `GET /api/v2/ping/` | Health check |
| `GET /api/v2/job_templates/{id}/` | Validate template exists |
| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
| `GET /api/v2/jobs/{id}/` | Poll job status |

### Lifecycle behavior

| Hook | Trigger | On failure |
|------|---------|------------|
| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
| Decommission | Before instance deletion | Failure does NOT block deletion |

Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
state (successful, failed, error) or the timeout expires. Post-deploy
rollback means the instance is deleted — the user sees "Ansible job N
finished with status: failed. Instance has been deleted."

## Lab deployment

**Automated path**: the `incusos/deploy-awx` script automates the entire
deployment and configuration process. Run `deploy-awx --deploy` for a full
install, `deploy-awx --configure` to set up project/templates, and
`deploy-awx --status` to check health. The manual steps below are for
reference, troubleshooting, and understanding what the script does.

### Resource requirements

| Resource | Recommended | Minimum |
|----------|-------------|---------|
| vCPU | 4 | 2 |
| RAM | 8 GiB | 4 GiB |
| Disk | 40 GiB | 20 GiB |

K3s uses ~600 MiB, AWX pods ~3 GiB, PostgreSQL ~400 MiB. The web pod
was originally set to 1 GiB limit but OOMKilled; 2 GiB is the tested
minimum.

### Lab details

| Setting | Value |
|---------|-------|
| VM name | `awx` |
| Location | oc-node-02 |
| IP | 192.168.102.161/22 (VLAN 69) |
| Gateway | 192.168.100.1 |
| DNS | 192.168.100.1 |
| AWX API port | 30080 (K3s NodePort) |
| OS | Debian 12 |
| K8s | K3s (single-node) |
| AWX | 24.6.1 |
| AWX Operator | 2.19.1 |

**Note**: AWX is exposed via K3s NodePort on port 30080, NOT via Traefik
ingress. The ingress returns 404 when accessed by IP (requires hostname).
NodePort works reliably with IP-based access.

### Manual deployment (tested)

#### Step 1: Create VM

```bash
# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Replace default NIC with macvlan for direct VLAN access
# IMPORTANT: use 'config device add' not 'profile device remove'
# Profile device removal fails on cluster members
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt
```

**Bug found**: `incus profile device remove` fails on cluster members
with "Profile device eth0 not found" because profiles are cluster-wide.
Use `incus config device add` instead, which creates an instance-level
override that takes priority over the profile.

#### Step 2: Configure static IP

Debian 12 uses `systemd-networkd`, **not netplan**. The original plan
assumed netplan but that's Ubuntu-only.

```bash
incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx

    # Debian 12 uses systemd-networkd
    cat > /etc/systemd/network/10-static.network << 'NETCFG'
[Match]
Name=enp5s0

[Network]
Address=192.168.102.161/22
Gateway=192.168.100.1
DNS=192.168.100.1
NETCFG

    systemctl restart systemd-networkd
"
```

After IP change, wait a few seconds for the new IP to take effect.

#### Step 3: Install prerequisites and K3s

```bash
# Debian 12 minimal doesn't have curl or git
incus exec oc-node-02:awx -- bash -c "
    apt-get update && apt-get install -y curl git

    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"
```

#### Step 4: Deploy AWX Operator

```bash
incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"
```

#### Step 5: Deploy AWX instance

The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
Key settings discovered during testing:

- **PVC**: must use `ReadWriteOnce` (not `ReadWriteMany`) for K3s local-path
- **Web pod memory**: 2 GiB limit minimum (1 GiB causes OOMKill)
- **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access)

```bash
# Push manifests to VM
incus exec oc-node-02:awx -- mkdir -p /opt/awx/base
incus file push incusos/awx-manifests/base/awx.yaml \
    oc-node-02:awx/opt/awx/base/awx.yaml
incus file push incusos/awx-manifests/base/kustomization.yaml \
    oc-node-02:awx/opt/awx/base/kustomization.yaml

# Apply
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w
```

Expected final pod state:
```
awx-migration-24.6.1-xxx              0/1     Completed
awx-operator-controller-manager-xxx   2/2     Running
awx-postgres-15-0                     1/1     Running
awx-task-xxx                          4/4     Running
awx-web-xxx                           3/3     Running
```

The migration pod runs once during initial deployment and shows `Completed`
status — this is expected.

#### Step 6: Verify

```bash
# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API via NodePort
curl -sk http://192.168.102.161:30080/api/v2/ping/
```

AWX web UI: `http://192.168.102.161:30080/` — login with `admin` / password above.

## AWX configuration (tested)

### Project (manual, no Git dependency)

AWX Git-based projects require SSH key configuration and network access
to the Git server. For simplicity, use a **manual project** — playbook
files are pushed directly to the AWX task pod:

```bash
# Create manual project via API
AWX_TOKEN="..."  # from PAT creation below
curl -sk -X POST http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "",
        "local_path": "incus-contrib"
    }'
```

Then push playbooks to the task pod:
```bash
AWX_POD=$(incus exec oc-node-02:awx -- kubectl -n awx get pods \
    -l app.kubernetes.io/name=awx-task -o jsonpath='{.items[0].metadata.name}')

# Create project directory structure
incus exec oc-node-02:awx -- kubectl -n awx exec $AWX_POD -- \
    mkdir -p /var/lib/awx/projects/incus-contrib/playbooks

# Push playbooks (via base64 to handle stdin issues)
B64=$(base64 -w0 ansible/playbooks/post-deploy.yml)
incus exec oc-node-02:awx -- bash -c "echo '$B64' | base64 -d > /root/post-deploy.yml"
incus exec oc-node-02:awx -- kubectl -n awx cp /root/post-deploy.yml \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/playbooks/post-deploy.yml
```

**Important**: also push the Incus client certificate and key to the
project directory — the playbooks use these for Incus API calls:
```bash
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.crt \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.crt
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.key \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.key
```

### Inventory

The playbooks use `localhost` exclusively (no SSH, Incus API only).
The inventory is a placeholder required by AWX:

```bash
curl -sk -X POST http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'
```

### Credentials

A machine credential (SSH key) is created but only used as a fallback.
The primary playbook approach uses the Incus REST API with client
certificates, not SSH.

```bash
# Generate SSH key on AWX VM
incus exec oc-node-02:awx -- ssh-keygen -t ed25519 -f /root/.ssh/awx_key -N ""

# Create machine credential in AWX
PRIV_KEY=$(incus exec oc-node-02:awx -- cat /root/.ssh/awx_key)
curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
        \"name\": \"incus-instances\",
        \"organization\": 1,
        \"credential_type\": 1,
        \"inputs\": {\"username\": \"root\", \"ssh_key_data\": $(echo "$PRIV_KEY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}
    }"
```

### Job templates

Both must have `ask_variables_on_launch: true` so Aether can pass `ffsdn_*`
variables at launch time.

**Look up resource IDs first** — they vary by installation:
```bash
# Find your project, inventory, and credential IDs
curl -sk http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{p[\"id\"]:3d} {p[\"name\"]}') \
        for p in json.load(sys.stdin)['results']]"

curl -sk http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{i[\"id\"]:3d} {i[\"name\"]}') \
        for i in json.load(sys.stdin)['results']]"

curl -sk http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{c[\"id\"]:3d} {c[\"name\"]}') \
        for c in json.load(sys.stdin)['results']]"
```

Create templates using the IDs from above (example uses lab values):
```bash
# Replace PROJECT_ID, INVENTORY_ID with values from the lookups above
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": PROJECT_ID,
        "playbook": "playbooks/post-deploy.yml",
        "inventory": INVENTORY_ID,
        "ask_variables_on_launch": true
    }'
```

**Attach credentials** to templates (required even if playbook doesn't use SSH):
```bash
# Replace TEMPLATE_ID and CREDENTIAL_ID with actual values
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/TEMPLATE_ID/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"id": CREDENTIAL_ID}'
```

In the lab, the current IDs are: project=9, inventory=2, credential=4,
post-deploy template=10, decommission template=11.

### Personal Access Token

```bash
AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
```

## Aether integration

**Automated path**: `deploy-awx --join-aether` creates a PAT and prints
the Aether registration details (endpoint URL, token, template IDs).
Then complete registration via the Aether UI below.

### Step 1: Register AWX endpoint

1. Log into Aether at `https://192.168.102.160:8443`
2. Navigate to **Ansible Automation** (`/awx-endpoints`)
3. Add a new endpoint:
   - **Name**: `lab-awx`
   - **URL**: `http://192.168.102.161:30080`
   - **Token**: the AWX PAT (from the PAT creation step above, or
     from `deploy-awx --join-aether` output)
   - **Verify SSL**: unchecked (AWX is HTTP on NodePort)

After saving, Aether checks AWX health automatically. Verify at
`/api/health/awx`:
```json
{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
```

The API equivalent (requires session cookies + CSRF token):
```bash
curl -sSk -b /tmp/aether-cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "http://192.168.102.161:30080",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'
```

### Step 2: Configure AWX template IDs

AWX template IDs can be configured at two levels:

**Per-blueprint** (recommended): In **Blueprint Design** (`/blueprintdesign`),
each blueprint has optional fields for **AWX Post-Deploy Template ID** and
**AWX Decommission Template ID**. Set these to the template IDs from the
"Job templates" section above.

**Per-cluster** (global default): In **Manage INCUS Clusters** (`/incus-infra`),
select the cluster → **Settings** tab. Configure the AWX endpoint and
default template IDs here. These apply to all deploys on this cluster
unless overridden by blueprint-level settings.

**Note**: The `PUT /api/clusters/{id}/awx-config` API endpoint was
non-functional in Aether v6.4.317 (returned "Invalid cluster ID").
Use the UI for cluster-level configuration.

### Verify integration

Deploy a test instance via Aether. If AWX is configured, Aether
automatically triggers the post-deploy job template after instance creation.

```bash
# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"

# Check AWX health from Aether's perspective
curl -sSk -b /tmp/aether-cookies.txt \
    https://192.168.102.160:8443/api/health/awx
```

## Writing playbooks

### Key design decision: Incus API, not SSH

The playbooks use the **Incus REST API** (file push + exec) instead of
SSH for all instance configuration. This is necessary because:

1. **Bridge network isolation**: containers on incusbr0 (10.207.217.0/24)
   are not routable from AWX (192.168.102.161). The bridge is NAT'd and
   IncusOS nodes don't forward inbound traffic to the bridge.
2. **No SSH dependency**: fresh Debian 12 containers don't have python3
   installed. With SSH, a raw bootstrap step is needed first.
3. **Works with any network**: the Incus API is accessible at the cluster
   node level (192.168.102.140-142:8443), regardless of which overlay
   or bridge network the container is on.

The trade-off: playbooks require an Incus client certificate in the AWX
project directory, and the playbook pattern is less "Ansible-native" (no
SSH connection plugins, no gather_facts, no modules running on the target).

### File paths in AWX

AWX runs playbooks inside an **Execution Environment** (EE) container via
receptor. During job execution:

| Path | Description |
|------|-------------|
| `/runner/project/` | Project root (maps to the manual project directory) |
| `/runner/project/playbooks/` | Playbook directory |
| `/runner/project/incus-client.crt` | Incus client certificate |
| `/runner/project/incus-client.key` | Incus client key |
| `/runner/artifacts/{job_id}/` | Job artifacts (logs, SSH keys) |

**Important**: files at `/var/lib/awx/projects/` (the task pod) are NOT
accessible at the same path during job execution. The EE mounts the
project at `/runner/project/`.

### post-deploy.yml pattern (tested, validated)

```yaml
- name: Post-deploy — configure instance via Incus API
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    # 1. Validate required ffsdn_* variables from Aether
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - ffsdn_instance_name is defined
          - ffsdn_instance_name | length > 0

    # 2. Wait for instance to be Running
    - name: Wait for instance to be running
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: state
      until: state.json.metadata.status == "Running"
      retries: 12
      delay: 5

    # 3. Push setup script via Incus file API
    - name: Push setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/setup.sh"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body: "{{ setup_script }}"
        headers:
          Content-Type: "application/octet-stream"
          X-Incus-type: "file"
          X-Incus-mode: "0755"

    # 4. Execute via Incus exec API
    - name: Execute setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body_format: json
        body:
          command: ["/bin/bash", "/tmp/setup.sh"]
          record-output: true
          interactive: false
          wait-for-websocket: false
        status_code: [202]
      register: exec_result

    # 5. Wait for completion via operation API
    - name: Wait for script to complete
      ansible.builtin.uri:
        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: exec_wait

    # 6. Verify exit code
    - name: Verify exit code
      ansible.builtin.assert:
        that:
          - exec_wait.json.metadata.metadata.return | int == 0

    # 7. Verify result via file API
    - name: Read /etc/deploy-info
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        return_content: true
      register: deploy_info
```

### Incus API endpoints used by playbooks

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/1.0/instances/{name}/state` | GET | Check if instance is Running |
| `/1.0/instances/{name}/files?path=...` | POST | Push files to instance |
| `/1.0/instances/{name}/files?path=...` | GET | Read files from instance |
| `/1.0/instances/{name}/exec` | POST | Execute commands in instance |
| `/1.0/operations/{uuid}/wait` | GET | Wait for async operation |
| `/1.0/instances/{name}` | GET | Check if instance exists |

**Note**: exec output (stdout/stderr from `record-output: true`) is NOT
retrievable via the `/1.0/instances/{name}/logs/` API. The log filename
format (`exec_UUID.stdout`) is rejected as "not valid". Verify results by
reading files written by the script instead.

### decommission.yml pattern (tested, validated)

```yaml
- name: Decommission — graceful shutdown and logging
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    - name: Check if instance still exists
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        status_code: [200, 404]
      register: instance_check

    - name: Stop services (best-effort)
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        # ... service stop command
      ignore_errors: true
      when: instance_check.status == 200

    - name: Record in ledger
      ansible.builtin.lineinfile:
        path: /tmp/awx-deploy-ledger.log
        # ... log entry
```

## Bugs discovered during validation

### Ansible reserved keyword: `environment`

Ansible's `environment` keyword is reserved — it sets process environment
variables for tasks. If Aether passed `environment: "lab"` as an extra var,
Ansible would interpret it as the keyword, resolving to `[]` instead of
the string value. The fix was to use `deploy_env` instead. In practice,
Aether uses `ffsdn_*` prefixes which avoid this entirely.

### Recursive template loop with self-referencing vars

```yaml
# BROKEN — causes infinite recursion if vm_ip is not provided
vars:
  vm_ip: "{{ vm_ip | default('') }}"
```

When the variable name matches the template reference AND the variable
is not provided as an extra var, Ansible enters infinite recursion.
The fix: don't self-reference. Use the `ffsdn_*` variables directly
without redeclaring them in vars blocks.

### AWX EE runs at /runner/project/, not /var/lib/awx/projects/

Files pushed to the AWX task pod at `/var/lib/awx/projects/incus-contrib/`
are mounted in the EE container at `/runner/project/`. All file paths in
playbooks must use the `/runner/project/` prefix.

### Bridge network not routable from AWX VM

Containers on incusbr0 (10.207.217.0/24) are NOT reachable from the AWX
VM (192.168.102.161). The bridge is NAT'd outbound only; IncusOS nodes
don't forward inbound traffic from the management network to the bridge.
Adding a static route on the AWX VM (`ip route add 10.207.217.0/24 via
192.168.102.140`) doesn't work because the node's nftables rules block
non-established connections from external interfaces to the bridge.

**Solution**: use the Incus REST API instead of SSH. The cluster API
(192.168.102.140:8443) is always reachable, and provides file push +
exec endpoints that work regardless of instance network topology.

### Aether cluster AWX config API bug

`PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}`
for all valid cluster IDs. Tested with different CSRF tokens, session
cookies, request formats (JSON fields, POST vs PUT). The endpoint is not
documented in Aether's swagger.yaml. Use the Aether UI (cluster Settings
tab or per-blueprint configuration) instead.

### AWX web pod OOMKill at 1 GiB

The AWX web pod (nginx + uwsgi) needs at least 2 GiB memory limit.
With 1 GiB, it starts successfully but gets OOMKilled under load.
Set in the AWX CR: `web_resource_requirements.limits.memory: 2Gi`.

### AWX PVC needs ReadWriteOnce

K3s local-path provisioner creates `hostPath` volumes which only support
`ReadWriteOnce`. If the AWX CR specifies `ReadWriteMany`, the PVC stays
Pending and PostgreSQL can't start. Set: `postgres_storage_class: local-path`
and ensure `ReadWriteOnce` access mode.

## Troubleshooting

### Pod status

```bash
# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
```

### Job output

```bash
# Get real-time job output (text format)
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/stdout/?format=txt \
    -H "Authorization: Bearer $AWX_TOKEN"

# Check job status
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); \
        print('status:', d['status'], 'failed:', d['failed'])"
```

### Common issues

**AWX pods stuck in Pending**: insufficient resources or PVC issue. Check:
```bash
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
incus exec oc-node-02:awx -- kubectl -n awx get pvc
```

**Job fails with recursive template error**: self-referencing var definition.
Check playbook vars blocks for patterns like `var: "{{ var | default('') }}"`.

**Job fails with "No such file or directory"**: cert/key not at the correct
path. Remember: EE uses `/runner/project/`, not `/var/lib/awx/projects/`.

**Job fails with "Connection failure"**: Incus API not reachable from AWX
pod, or client cert not trusted by cluster. Test from inside the pod:
```bash
incus exec oc-node-02:awx -- kubectl -n awx exec <task-pod> -c awx-ee -- \
    curl -sk --cert /runner/project/incus-client.crt \
    --key /runner/project/incus-client.key \
    https://192.168.102.140:8443/1.0
```

**Aether deploy rolls back immediately**: AWX job failed. Check the AWX
job output for the specific error. Common causes: template misconfiguration,
missing credential attachment, playbook syntax errors.

### Useful commands

```bash
# AWX version
curl -sk http://192.168.102.161:30080/api/v2/ping/

# List job templates
curl -sk http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{t[\"id\"]:3d} {t[\"name\"]}') \
        for t in json.load(sys.stdin)['results']]"

# List recent jobs
curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
        for j in json.load(sys.stdin)['results']]"

# Manually trigger post-deploy (replace 10 with your template ID)
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {
        "ffsdn_instance_name": "test-ct",
        "ffsdn_instance_ip": "10.207.217.99",
        "ffsdn_cluster_name": "oc-lab-cluster",
        "ffsdn_deployed_by": "admin",
        "ffsdn_image_os": "Debian",
        "ffsdn_image_release": "bookworm"
    }}'
```

### Rollback

If AWX is broken beyond repair:
```bash
incus delete oc-node-02:awx --force
# Then redeploy from Step 1
```

The AWX VM is a standalone workload with no cluster dependencies. Destroying
and redeploying has zero impact on the Incus cluster or other workloads.