incus-contrib/notes/awx-guide.md

# AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST
API for running Ansible playbooks. Aether integrates with AWX to run
post-deploy and decommission playbooks as lifecycle hooks on every instance
create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to
Aether, writing lifecycle playbooks, and troubleshooting.

## Architecture

```
  User deploys instance via Aether UI
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: ffsdn_instance_name, ffsdn_instance_ip, ...
          |
          v
  AWX runs post-deploy.yml playbook
    1. Pushes setup script to instance (Incus file API)
    2. Executes script (Incus exec API)
    3. Verifies /etc/deploy-info was written
    4. Logs to deployment ledger
          |
          v
  Aether marks deployment complete (or rolls back on failure)
```

Decommission works in reverse — Aether triggers the decommission template
before deleting the instance. Decommission failures do NOT block deletion.

### Separation of concerns

| Layer | Tool | Responsibility |
|-------|------|----------------|
| Infrastructure | Aether + Incus | Instance creation, networking, resources, lifecycle |
| Configuration | AWX + Ansible | OS config, packages, services, integrations |
| Automation glue | Aether AWX binding | Triggers playbooks at create/delete |

### What Aether passes to AWX (validated)

Every AWX job launched by Aether receives these `ffsdn_` prefixed extra vars:

| Variable | Description | Example |
|----------|-------------|---------|
| `ffsdn_instance_name` | Instance name | `awx-e2e-test` |
| `ffsdn_instance_ip` | IP address assigned by Aether | `10.207.217.5` |
| `ffsdn_cluster_id` | Numeric cluster ID | `52` |
| `ffsdn_cluster_name` | Cluster display name | `oc-lab-cluster` |
| `ffsdn_deployed_by` | Aether username who triggered deploy | `admin` |
| `ffsdn_image_os` | Image OS | `Debian` |
| `ffsdn_image_release` | Image release | `bookworm` |
| `ffsdn_image_alias` | Image alias (may be empty) | `""` |

**Important**: Aether does NOT pass `vm_name`, `vm_ip`, `environment`,
`owner`, or `cost_center`. The original plan assumed these names, but
real testing revealed Aether uses the `ffsdn_` prefix exclusively.

### AWX API endpoints Aether uses

| Endpoint | Purpose |
|----------|---------|
| `GET /api/v2/ping/` | Health check |
| `GET /api/v2/job_templates/{id}/` | Validate template exists |
| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
| `GET /api/v2/jobs/{id}/` | Poll job status |

### Lifecycle behavior

| Hook | Trigger | On failure |
|------|---------|------------|
| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
| Decommission | Before instance deletion | Failure does NOT block deletion |

Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
state (successful, failed, error) or the timeout expires. Post-deploy
rollback means the instance is deleted — the user sees "Ansible job N
finished with status: failed. Instance has been deleted."

## Lab deployment

### Resource requirements

| Resource | Recommended | Minimum |
|----------|-------------|---------|
| vCPU | 4 | 2 |
| RAM | 8 GiB | 4 GiB |
| Disk | 40 GiB | 20 GiB |

K3s uses ~600 MiB, AWX pods ~3 GiB, PostgreSQL ~400 MiB. The web pod
was originally set to 1 GiB limit but OOMKilled; 2 GiB is the tested
minimum.

### Lab details

| Setting | Value |
|---------|-------|
| VM name | `awx` |
| Location | oc-node-02 |
| IP | 192.168.102.161/22 (VLAN 69) |
| Gateway | 192.168.100.1 |
| DNS | 192.168.100.1 |
| AWX API port | 30080 (K3s NodePort) |
| OS | Debian 12 |
| K8s | K3s (single-node) |
| AWX | 24.6.1 |
| AWX Operator | 2.19.1 |

**Note**: AWX is exposed via K3s NodePort on port 30080, NOT via Traefik
ingress. The ingress returns 404 when accessed by IP (requires hostname).
NodePort works reliably with IP-based access.

### Manual deployment (tested)

#### Step 1: Create VM

```bash
# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Replace default NIC with macvlan for direct VLAN access
# IMPORTANT: use 'config device add' not 'profile device remove'
# Profile device removal fails on cluster members
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt
```

**Bug found**: `incus profile device remove` fails on cluster members
with "Profile device eth0 not found" because profiles are cluster-wide.
Use `incus config device add` instead, which creates an instance-level
override that takes priority over the profile.

#### Step 2: Configure static IP

Debian 12 uses `systemd-networkd`, **not netplan**. The original plan
assumed netplan but that's Ubuntu-only.

```bash
incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx

    # Debian 12 uses systemd-networkd
    cat > /etc/systemd/network/10-static.network << 'NETCFG'
[Match]
Name=enp5s0

[Network]
Address=192.168.102.161/22
Gateway=192.168.100.1
DNS=192.168.100.1
NETCFG

    systemctl restart systemd-networkd
"
```

After IP change, wait a few seconds for the new IP to take effect.

#### Step 3: Install prerequisites and K3s

```bash
# Debian 12 minimal doesn't have curl or git
incus exec oc-node-02:awx -- bash -c "
    apt-get update && apt-get install -y curl git

    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"
```

#### Step 4: Deploy AWX Operator

```bash
incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"
```

#### Step 5: Deploy AWX instance

The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
Key settings discovered during testing:

- **PVC**: must use `ReadWriteOnce` (not `ReadWriteMany`) for K3s local-path
- **Web pod memory**: 2 GiB limit minimum (1 GiB causes OOMKill)
- **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access)

```bash
# Push manifests to VM and apply
incus exec oc-node-02:awx -- mkdir -p /opt/awx/base

# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/)
# Then apply:
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w
```

Expected final pod state:
```
awx-operator-controller-manager-xxx   2/2     Running
awx-postgres-15-0                     1/1     Running
awx-task-xxx                          4/4     Running
awx-web-xxx                           3/3     Running
```

#### Step 6: Verify

```bash
# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API via NodePort
curl -sk http://192.168.102.161:30080/api/v2/ping/
```

AWX web UI: `http://192.168.102.161:30080/` — login with `admin` / password above.

## AWX configuration (tested)

### Project (manual, no Git dependency)

AWX Git-based projects require SSH key configuration and network access
to the Git server. For simplicity, use a **manual project** — playbook
files are pushed directly to the AWX task pod:

```bash
# Create manual project via API
AWX_TOKEN="..."  # from PAT creation below
curl -sk -X POST http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "",
        "local_path": "incus-contrib"
    }'
```

Then push playbooks to the task pod:
```bash
AWX_POD=$(incus exec oc-node-02:awx -- kubectl -n awx get pods \
    -l app.kubernetes.io/name=awx-task -o jsonpath='{.items[0].metadata.name}')

# Create project directory structure
incus exec oc-node-02:awx -- kubectl -n awx exec $AWX_POD -- \
    mkdir -p /var/lib/awx/projects/incus-contrib/playbooks

# Push playbooks (via base64 to handle stdin issues)
B64=$(base64 -w0 ansible/playbooks/post-deploy.yml)
incus exec oc-node-02:awx -- bash -c "echo '$B64' | base64 -d > /root/post-deploy.yml"
incus exec oc-node-02:awx -- kubectl -n awx cp /root/post-deploy.yml \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/playbooks/post-deploy.yml
```

**Important**: also push the Incus client certificate and key to the
project directory — the playbooks use these for Incus API calls:
```bash
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.crt \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.crt
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.key \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.key
```

### Inventory

The playbooks use `localhost` exclusively (no SSH, Incus API only).
The inventory is a placeholder required by AWX:

```bash
curl -sk -X POST http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'
```

### Credentials

A machine credential (SSH key) is created but only used as a fallback.
The primary playbook approach uses the Incus REST API with client
certificates, not SSH.

```bash
# Generate SSH key on AWX VM
incus exec oc-node-02:awx -- ssh-keygen -t ed25519 -f /root/.ssh/awx_key -N ""

# Create machine credential in AWX
PRIV_KEY=$(incus exec oc-node-02:awx -- cat /root/.ssh/awx_key)
curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
        \"name\": \"incus-instances\",
        \"organization\": 1,
        \"credential_type\": 1,
        \"inputs\": {\"username\": \"root\", \"ssh_key_data\": $(echo "$PRIV_KEY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}
    }"
```

### Job templates

| Template | Playbook | Template ID |
|----------|----------|-------------|
| `post-deploy` | `playbooks/post-deploy.yml` | 10 |
| `decommission` | `playbooks/decommission.yml` | 11 |

Both must have `ask_variables_on_launch: true`:

```bash
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": 6,
        "playbook": "playbooks/post-deploy.yml",
        "inventory": 2,
        "ask_variables_on_launch": true
    }'
```

**Attach credentials** to templates (required even if playbook doesn't use SSH):
```bash
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"id": 4}'
```

### Personal Access Token

```bash
AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
```

## Aether integration

### Register AWX endpoint

Via Aether API (requires session cookies + CSRF token):

```bash
curl -sSk -b /tmp/aether-cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "http://192.168.102.161:30080",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'
```

**Tested result**: endpoint ID 2 created, AWX health shows "healthy"
at `/api/health/awx`:
```json
{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
```

### Bind cluster to AWX

**Known bug**: `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid
cluster ID"}` for all valid cluster IDs. This Aether API endpoint appears
non-functional in v6.4.317.

**Workaround**: direct PostgreSQL update on the Aether database:
```bash
incus exec oc-node-01:aether -- bash -c "
    docker exec aether-postgres psql -U aether -d incusovnsdnc -c \"
        UPDATE incus_ovn_clusters SET
            awx_endpoint_id = 2,
            awx_post_deploy_template_id = 10,
            awx_decommission_template_id = 11,
            awx_job_timeout_seconds = 600
        WHERE cluster_id = 52;
    \"
"
```

### Verify integration

```bash
# Check Aether's AWX health endpoint
curl -sSk -b /tmp/aether-cookies.txt \
    https://192.168.102.160:8443/api/health/awx

# Deploy a test instance via Aether → AWX job triggers automatically
# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
```

## Writing playbooks

### Key design decision: Incus API, not SSH

The playbooks use the **Incus REST API** (file push + exec) instead of
SSH for all instance configuration. This is necessary because:

1. **Bridge network isolation**: containers on incusbr0 (10.207.217.0/24)
   are not routable from AWX (192.168.102.161). The bridge is NAT'd and
   IncusOS nodes don't forward inbound traffic to the bridge.
2. **No SSH dependency**: fresh Debian 12 containers don't have python3
   installed. With SSH, a raw bootstrap step is needed first.
3. **Works with any network**: the Incus API is accessible at the cluster
   node level (192.168.102.140-142:8443), regardless of which overlay
   or bridge network the container is on.

The trade-off: playbooks require an Incus client certificate in the AWX
project directory, and the playbook pattern is less "Ansible-native" (no
SSH connection plugins, no gather_facts, no modules running on the target).

### File paths in AWX

AWX runs playbooks inside an **Execution Environment** (EE) container via
receptor. During job execution:

| Path | Description |
|------|-------------|
| `/runner/project/` | Project root (maps to the manual project directory) |
| `/runner/project/playbooks/` | Playbook directory |
| `/runner/project/incus-client.crt` | Incus client certificate |
| `/runner/project/incus-client.key` | Incus client key |
| `/runner/artifacts/{job_id}/` | Job artifacts (logs, SSH keys) |

**Important**: files at `/var/lib/awx/projects/` (the task pod) are NOT
accessible at the same path during job execution. The EE mounts the
project at `/runner/project/`.

### post-deploy.yml pattern (tested, validated)

```yaml
- name: Post-deploy — configure instance via Incus API
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    # 1. Validate required ffsdn_* variables from Aether
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - ffsdn_instance_name is defined
          - ffsdn_instance_name | length > 0

    # 2. Wait for instance to be Running
    - name: Wait for instance to be running
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: state
      until: state.json.metadata.status == "Running"
      retries: 12
      delay: 5

    # 3. Push setup script via Incus file API
    - name: Push setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/setup.sh"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body: "{{ setup_script }}"
        headers:
          Content-Type: "application/octet-stream"
          X-Incus-type: "file"
          X-Incus-mode: "0755"

    # 4. Execute via Incus exec API
    - name: Execute setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body_format: json
        body:
          command: ["/bin/bash", "/tmp/setup.sh"]
          record-output: true
          interactive: false
          wait-for-websocket: false
        status_code: [202]
      register: exec_result

    # 5. Wait for completion via operation API
    - name: Wait for script to complete
      ansible.builtin.uri:
        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: exec_wait

    # 6. Verify exit code
    - name: Verify exit code
      ansible.builtin.assert:
        that:
          - exec_wait.json.metadata.metadata.return | int == 0

    # 7. Verify result via file API
    - name: Read /etc/deploy-info
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        return_content: true
      register: deploy_info
```

### Incus API endpoints used by playbooks

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/1.0/instances/{name}/state` | GET | Check if instance is Running |
| `/1.0/instances/{name}/files?path=...` | POST | Push files to instance |
| `/1.0/instances/{name}/files?path=...` | GET | Read files from instance |
| `/1.0/instances/{name}/exec` | POST | Execute commands in instance |
| `/1.0/operations/{uuid}/wait` | GET | Wait for async operation |
| `/1.0/instances/{name}` | GET | Check if instance exists |

**Note**: exec output (stdout/stderr from `record-output: true`) is NOT
retrievable via the `/1.0/instances/{name}/logs/` API. The log filename
format (`exec_UUID.stdout`) is rejected as "not valid". Verify results by
reading files written by the script instead.

### decommission.yml pattern (tested, validated)

```yaml
- name: Decommission — graceful shutdown and logging
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    - name: Check if instance still exists
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        status_code: [200, 404]
      register: instance_check

    - name: Stop services (best-effort)
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        # ... service stop command
      ignore_errors: true
      when: instance_check.status == 200

    - name: Record in ledger
      ansible.builtin.lineinfile:
        path: /tmp/awx-deploy-ledger.log
        # ... log entry
```

## Bugs discovered during validation

### Ansible reserved keyword: `environment`

Ansible's `environment` keyword is reserved — it sets process environment
variables for tasks. If Aether passed `environment: "lab"` as an extra var,
Ansible would interpret it as the keyword, resolving to `[]` instead of
the string value. The fix was to use `deploy_env` instead. In practice,
Aether uses `ffsdn_*` prefixes which avoid this entirely.

### Recursive template loop with self-referencing vars

```yaml
# BROKEN — causes infinite recursion if vm_ip is not provided
vars:
  vm_ip: "{{ vm_ip | default('') }}"
```

When the variable name matches the template reference AND the variable
is not provided as an extra var, Ansible enters infinite recursion.
The fix: don't self-reference. Use the `ffsdn_*` variables directly
without redeclaring them in vars blocks.

### AWX EE runs at /runner/project/, not /var/lib/awx/projects/

Files pushed to the AWX task pod at `/var/lib/awx/projects/incus-contrib/`
are mounted in the EE container at `/runner/project/`. All file paths in
playbooks must use the `/runner/project/` prefix.

### Bridge network not routable from AWX VM

Containers on incusbr0 (10.207.217.0/24) are NOT reachable from the AWX
VM (192.168.102.161). The bridge is NAT'd outbound only; IncusOS nodes
don't forward inbound traffic from the management network to the bridge.
Adding a static route on the AWX VM (`ip route add 10.207.217.0/24 via
192.168.102.140`) doesn't work because the node's nftables rules block
non-established connections from external interfaces to the bridge.

**Solution**: use the Incus REST API instead of SSH. The cluster API
(192.168.102.140:8443) is always reachable, and provides file push +
exec endpoints that work regardless of instance network topology.

### Aether cluster AWX config API bug

`PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}`
for all valid cluster IDs. Tested with different CSRF tokens, session
cookies, request formats (JSON fields, POST vs PUT). The endpoint is not
documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE.

### AWX web pod OOMKill at 1 GiB

The AWX web pod (nginx + uwsgi) needs at least 2 GiB memory limit.
With 1 GiB, it starts successfully but gets OOMKilled under load.
Set in the AWX CR: `web_resource_requirements.limits.memory: 2Gi`.

### AWX PVC needs ReadWriteOnce

K3s local-path provisioner creates `hostPath` volumes which only support
`ReadWriteOnce`. If the AWX CR specifies `ReadWriteMany`, the PVC stays
Pending and PostgreSQL can't start. Set: `postgres_storage_class: local-path`
and ensure `ReadWriteOnce` access mode.

## Troubleshooting

### Pod status

```bash
# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
```

### Job output

```bash
# Get real-time job output (text format)
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/stdout/?format=txt \
    -H "Authorization: Bearer $AWX_TOKEN"

# Check job status
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); \
        print('status:', d['status'], 'failed:', d['failed'])"
```

### Common issues

**AWX pods stuck in Pending**: insufficient resources or PVC issue. Check:
```bash
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
incus exec oc-node-02:awx -- kubectl -n awx get pvc
```

**Job fails with recursive template error**: self-referencing var definition.
Check playbook vars blocks for patterns like `var: "{{ var | default('') }}"`.

**Job fails with "No such file or directory"**: cert/key not at the correct
path. Remember: EE uses `/runner/project/`, not `/var/lib/awx/projects/`.

**Job fails with "Connection failure"**: Incus API not reachable from AWX
pod, or client cert not trusted by cluster. Test from inside the pod:
```bash
incus exec oc-node-02:awx -- kubectl -n awx exec <task-pod> -c awx-ee -- \
    curl -sk --cert /runner/project/incus-client.crt \
    --key /runner/project/incus-client.key \
    https://192.168.102.140:8443/1.0
```

**Aether deploy rolls back immediately**: AWX job failed. Check the AWX
job output for the specific error. Common causes: template misconfiguration,
missing credential attachment, playbook syntax errors.

### Useful commands

```bash
# AWX version
curl -sk http://192.168.102.161:30080/api/v2/ping/

# List job templates
curl -sk http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{t[\"id\"]:3d} {t[\"name\"]}') \
        for t in json.load(sys.stdin)['results']]"

# List recent jobs
curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
        for j in json.load(sys.stdin)['results']]"

# Manually trigger post-deploy
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {
        "ffsdn_instance_name": "test-ct",
        "ffsdn_instance_ip": "10.207.217.99",
        "ffsdn_cluster_name": "oc-lab-cluster",
        "ffsdn_deployed_by": "admin",
        "ffsdn_image_os": "Debian",
        "ffsdn_image_release": "bookworm"
    }}'
```

### Rollback

If AWX is broken beyond repair:
```bash
incus delete oc-node-02:awx --force
# Then redeploy from Step 1
```

The AWX VM is a standalone workload with no cluster dependencies. Destroying
and redeploying has zero impact on the Incus cluster or other workloads.