28 KiB

Raw Blame History

AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST API for running Ansible playbooks. Aether integrates with AWX to run post-deploy and decommission playbooks as lifecycle hooks on every instance create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to Aether, writing lifecycle playbooks, and troubleshooting.

Architecture

  User deploys instance via Aether UI
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: ffsdn_instance_name, ffsdn_instance_ip, ...
          |
          v
  AWX runs post-deploy.yml playbook
    1. Pushes setup script to instance (Incus file API)
    2. Executes script (Incus exec API)
    3. Verifies /etc/deploy-info was written
    4. Logs to deployment ledger
          |
          v
  Aether marks deployment complete (or rolls back on failure)

Decommission works in reverse — Aether triggers the decommission template before deleting the instance. Decommission failures do NOT block deletion.

Separation of concerns

Layer	Tool	Responsibility
Infrastructure	Aether + Incus	Instance creation, networking, resources, lifecycle
Configuration	AWX + Ansible	OS config, packages, services, integrations
Automation glue	Aether AWX binding	Triggers playbooks at create/delete

What Aether passes to AWX (validated)

Every AWX job launched by Aether receives these ffsdn_ prefixed extra vars:

Variable	Description	Example
`ffsdn_instance_name`	Instance name	`awx-e2e-test`
`ffsdn_instance_ip`	IP address assigned by Aether	`10.207.217.5`
`ffsdn_cluster_id`	Numeric cluster ID	`52`
`ffsdn_cluster_name`	Cluster display name	`oc-lab-cluster`
`ffsdn_deployed_by`	Aether username who triggered deploy	`admin`
`ffsdn_image_os`	Image OS	`Debian`
`ffsdn_image_release`	Image release	`bookworm`
`ffsdn_image_alias`	Image alias (may be empty)	`""`

Important: Aether does NOT pass vm_name, vm_ip, environment, owner, or cost_center. The original plan assumed these names, but real testing revealed Aether uses the ffsdn_ prefix exclusively.

AWX API endpoints Aether uses

Endpoint	Purpose
`GET /api/v2/ping/`	Health check
`GET /api/v2/job_templates/{id}/`	Validate template exists
`POST /api/v2/job_templates/{id}/launch/`	Trigger job with extra_vars
`GET /api/v2/jobs/{id}/`	Poll job status

Lifecycle behavior

Hook	Trigger	On failure
Post-deploy	After instance creation	Aether auto-rollbacks (deletes instance)
Decommission	Before instance deletion	Failure does NOT block deletion

Aether polls GET /api/v2/jobs/{id}/ until the job reaches a terminal state (successful, failed, error) or the timeout expires. Post-deploy rollback means the instance is deleted — the user sees "Ansible job N finished with status: failed. Instance has been deleted."

Lab deployment

Automated path: the incusos/deploy-awx script automates the entire deployment and configuration process. Run deploy-awx --deploy for a full install, deploy-awx --configure to set up project/templates, and deploy-awx --status to check health. The manual steps below are for reference, troubleshooting, and understanding what the script does.

Resource requirements

Resource	Recommended	Minimum
vCPU	4	2
RAM	8 GiB	4 GiB
Disk	40 GiB	20 GiB

K3s uses ~600 MiB, AWX pods ~3 GiB, PostgreSQL ~400 MiB. The web pod was originally set to 1 GiB limit but OOMKilled; 2 GiB is the tested minimum.

Lab details

Setting	Value
VM name	`awx`
Location	oc-node-02
IP	192.168.102.161/22 (VLAN 69)
Gateway	192.168.100.1
DNS	192.168.100.1
AWX API port	30080 (K3s NodePort)
OS	Debian 12
K8s	K3s (single-node)
AWX	24.6.1
AWX Operator	2.19.1

Note: AWX is exposed via K3s NodePort on port 30080, NOT via Traefik ingress. The ingress returns 404 when accessed by IP (requires hostname). NodePort works reliably with IP-based access.

Manual deployment (tested)

Step 1: Create VM

# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Replace default NIC with macvlan for direct VLAN access
# IMPORTANT: use 'config device add' not 'profile device remove'
# Profile device removal fails on cluster members
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt

Bug found: incus profile device remove fails on cluster members with "Profile device eth0 not found" because profiles are cluster-wide. Use incus config device add instead, which creates an instance-level override that takes priority over the profile.

Step 2: Configure static IP

Debian 12 uses systemd-networkd, not netplan. The original plan assumed netplan but that's Ubuntu-only.

incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx

    # Debian 12 uses systemd-networkd
    cat > /etc/systemd/network/10-static.network << 'NETCFG'
[Match]
Name=enp5s0

[Network]
Address=192.168.102.161/22
Gateway=192.168.100.1
DNS=192.168.100.1
NETCFG

    systemctl restart systemd-networkd
"

After IP change, wait a few seconds for the new IP to take effect.

Step 3: Install prerequisites and K3s

# Debian 12 minimal doesn't have curl or git
incus exec oc-node-02:awx -- bash -c "
    apt-get update && apt-get install -y curl git

    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"

Step 4: Deploy AWX Operator

incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"

Step 5: Deploy AWX instance

The AWX custom resource is defined in incusos/awx-manifests/base/awx.yaml. Key settings discovered during testing:

PVC: must use ReadWriteOnce (not ReadWriteMany) for K3s local-path
Web pod memory: 2 GiB limit minimum (1 GiB causes OOMKill)
Service type: NodePort on port 30080 (ingress returns 404 for IP access)

# Push manifests to VM
incus exec oc-node-02:awx -- mkdir -p /opt/awx/base
incus file push incusos/awx-manifests/base/awx.yaml \
    oc-node-02:awx/opt/awx/base/awx.yaml
incus file push incusos/awx-manifests/base/kustomization.yaml \
    oc-node-02:awx/opt/awx/base/kustomization.yaml

# Apply
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w

Expected final pod state:

awx-migration-24.6.1-xxx              0/1     Completed
awx-operator-controller-manager-xxx   2/2     Running
awx-postgres-15-0                     1/1     Running
awx-task-xxx                          4/4     Running
awx-web-xxx                           3/3     Running

The migration pod runs once during initial deployment and shows Completed status — this is expected.

Step 6: Verify

# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API via NodePort
curl -sk http://192.168.102.161:30080/api/v2/ping/

AWX web UI: http://192.168.102.161:30080/ — login with admin / password above.

AWX configuration (tested)

Project (manual, no Git dependency)

AWX Git-based projects require SSH key configuration and network access to the Git server. For simplicity, use a manual project — playbook files are pushed directly to the AWX task pod:

# Create manual project via API
AWX_TOKEN="..."  # from PAT creation below
curl -sk -X POST http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "",
        "local_path": "incus-contrib"
    }'

Then push playbooks to the task pod:

AWX_POD=$(incus exec oc-node-02:awx -- kubectl -n awx get pods \
    -l app.kubernetes.io/name=awx-task -o jsonpath='{.items[0].metadata.name}')

# Create project directory structure
incus exec oc-node-02:awx -- kubectl -n awx exec $AWX_POD -- \
    mkdir -p /var/lib/awx/projects/incus-contrib/playbooks

# Push playbooks (via base64 to handle stdin issues)
B64=$(base64 -w0 ansible/playbooks/post-deploy.yml)
incus exec oc-node-02:awx -- bash -c "echo '$B64' | base64 -d > /root/post-deploy.yml"
incus exec oc-node-02:awx -- kubectl -n awx cp /root/post-deploy.yml \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/playbooks/post-deploy.yml

Important: also push the Incus client certificate and key to the project directory — the playbooks use these for Incus API calls:

incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.crt \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.crt
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.key \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.key

Inventory

The playbooks use localhost exclusively (no SSH, Incus API only). The inventory is a placeholder required by AWX:

curl -sk -X POST http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'

Credentials

A machine credential (SSH key) is created but only used as a fallback. The primary playbook approach uses the Incus REST API with client certificates, not SSH.

# Generate SSH key on AWX VM
incus exec oc-node-02:awx -- ssh-keygen -t ed25519 -f /root/.ssh/awx_key -N ""

# Create machine credential in AWX
PRIV_KEY=$(incus exec oc-node-02:awx -- cat /root/.ssh/awx_key)
curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
        \"name\": \"incus-instances\",
        \"organization\": 1,
        \"credential_type\": 1,
        \"inputs\": {\"username\": \"root\", \"ssh_key_data\": $(echo "$PRIV_KEY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}
    }"

Job templates

Both must have ask_variables_on_launch: true so Aether can pass ffsdn_* variables at launch time.

Look up resource IDs first — they vary by installation:

# Find your project, inventory, and credential IDs
curl -sk http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{p[\"id\"]:3d} {p[\"name\"]}') \
        for p in json.load(sys.stdin)['results']]"

curl -sk http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{i[\"id\"]:3d} {i[\"name\"]}') \
        for i in json.load(sys.stdin)['results']]"

curl -sk http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{c[\"id\"]:3d} {c[\"name\"]}') \
        for c in json.load(sys.stdin)['results']]"

Create templates using the IDs from above (example uses lab values):

# Replace PROJECT_ID, INVENTORY_ID with values from the lookups above
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": PROJECT_ID,
        "playbook": "playbooks/post-deploy.yml",
        "inventory": INVENTORY_ID,
        "ask_variables_on_launch": true
    }'

Attach credentials to templates (required even if playbook doesn't use SSH):

# Replace TEMPLATE_ID and CREDENTIAL_ID with actual values
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/TEMPLATE_ID/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"id": CREDENTIAL_ID}'

In the lab, the current IDs are: project=9, inventory=2, credential=4, post-deploy template=10, decommission template=11.

Personal Access Token

AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

Aether integration

Automated path: deploy-awx --join-aether creates a PAT and prints the Aether registration details (endpoint URL, token, template IDs). Then complete registration via the Aether UI below.

Step 1: Register AWX endpoint

Log into Aether at https://192.168.102.160:8443
Navigate to Ansible Automation (/awx-endpoints)
Add a new endpoint:
- Name: lab-awx
- URL: http://192.168.102.161:30080
- Token: the AWX PAT (from the PAT creation step above, or from deploy-awx --join-aether output)
- Verify SSL: unchecked (AWX is HTTP on NodePort)

After saving, Aether checks AWX health automatically. Verify at /api/health/awx:

{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}

The API equivalent (requires session cookies + CSRF token):

curl -sSk -b /tmp/aether-cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "http://192.168.102.161:30080",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'

Step 2: Configure AWX template IDs

AWX template IDs can be configured at two levels:

Per-blueprint (recommended): In Blueprint Design (/blueprintdesign), each blueprint has optional fields for AWX Post-Deploy Template ID and AWX Decommission Template ID. Set these to the template IDs from the "Job templates" section above.

Per-cluster (global default): In Manage INCUS Clusters (/incus-infra), select the cluster → Settings tab. Configure the AWX endpoint and default template IDs here. These apply to all deploys on this cluster unless overridden by blueprint-level settings.

Note: The PUT /api/clusters/{id}/awx-config API endpoint was non-functional in Aether v6.4.317 (returned "Invalid cluster ID"). Use the UI for cluster-level configuration.

Verify integration

Deploy a test instance via Aether. If AWX is configured, Aether automatically triggers the post-deploy job template after instance creation.

# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"

# Check AWX health from Aether's perspective
curl -sSk -b /tmp/aether-cookies.txt \
    https://192.168.102.160:8443/api/health/awx

Writing playbooks

Key design decision: Incus API, not SSH

The playbooks use the Incus REST API (file push + exec) instead of SSH for all instance configuration. This is necessary because:

Bridge network isolation: containers on incusbr0 (10.207.217.0/24) are not routable from AWX (192.168.102.161). The bridge is NAT'd and IncusOS nodes don't forward inbound traffic to the bridge.
No SSH dependency: fresh Debian 12 containers don't have python3 installed. With SSH, a raw bootstrap step is needed first.
Works with any network: the Incus API is accessible at the cluster node level (192.168.102.140-142:8443), regardless of which overlay or bridge network the container is on.

The trade-off: playbooks require an Incus client certificate in the AWX project directory, and the playbook pattern is less "Ansible-native" (no SSH connection plugins, no gather_facts, no modules running on the target).

File paths in AWX

AWX runs playbooks inside an Execution Environment (EE) container via receptor. During job execution:

Path	Description
`/runner/project/`	Project root (maps to the manual project directory)
`/runner/project/playbooks/`	Playbook directory
`/runner/project/incus-client.crt`	Incus client certificate
`/runner/project/incus-client.key`	Incus client key
`/runner/artifacts/{job_id}/`	Job artifacts (logs, SSH keys)

Important: files at /var/lib/awx/projects/ (the task pod) are NOT accessible at the same path during job execution. The EE mounts the project at /runner/project/.

post-deploy.yml pattern (tested, validated)

- name: Post-deploy — configure instance via Incus API
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    # 1. Validate required ffsdn_* variables from Aether
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - ffsdn_instance_name is defined
          - ffsdn_instance_name | length > 0

    # 2. Wait for instance to be Running
    - name: Wait for instance to be running
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: state
      until: state.json.metadata.status == "Running"
      retries: 12
      delay: 5

    # 3. Push setup script via Incus file API
    - name: Push setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/setup.sh"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body: "{{ setup_script }}"
        headers:
          Content-Type: "application/octet-stream"
          X-Incus-type: "file"
          X-Incus-mode: "0755"

    # 4. Execute via Incus exec API
    - name: Execute setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body_format: json
        body:
          command: ["/bin/bash", "/tmp/setup.sh"]
          record-output: true
          interactive: false
          wait-for-websocket: false
        status_code: [202]
      register: exec_result

    # 5. Wait for completion via operation API
    - name: Wait for script to complete
      ansible.builtin.uri:
        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: exec_wait

    # 6. Verify exit code
    - name: Verify exit code
      ansible.builtin.assert:
        that:
          - exec_wait.json.metadata.metadata.return | int == 0

    # 7. Verify result via file API
    - name: Read /etc/deploy-info
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        return_content: true
      register: deploy_info

Incus API endpoints used by playbooks

Endpoint	Method	Purpose
`/1.0/instances/{name}/state`	GET	Check if instance is Running
`/1.0/instances/{name}/files?path=...`	POST	Push files to instance
`/1.0/instances/{name}/files?path=...`	GET	Read files from instance
`/1.0/instances/{name}/exec`	POST	Execute commands in instance
`/1.0/operations/{uuid}/wait`	GET	Wait for async operation
`/1.0/instances/{name}`	GET	Check if instance exists

Note: exec output (stdout/stderr from record-output: true) is NOT retrievable via the /1.0/instances/{name}/logs/ API. The log filename format (exec_UUID.stdout) is rejected as "not valid". Verify results by reading files written by the script instead.

decommission.yml pattern (tested, validated)

- name: Decommission — graceful shutdown and logging
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    - name: Check if instance still exists
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        status_code: [200, 404]
      register: instance_check

    - name: Stop services (best-effort)
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        # ... service stop command
      ignore_errors: true
      when: instance_check.status == 200

    - name: Record in ledger
      ansible.builtin.lineinfile:
        path: /tmp/awx-deploy-ledger.log
        # ... log entry

Bugs discovered during validation

Ansible reserved keyword: `environment`

Ansible's environment keyword is reserved — it sets process environment variables for tasks. If Aether passed environment: "lab" as an extra var, Ansible would interpret it as the keyword, resolving to [] instead of the string value. The fix was to use deploy_env instead. In practice, Aether uses ffsdn_* prefixes which avoid this entirely.

Recursive template loop with self-referencing vars

# BROKEN — causes infinite recursion if vm_ip is not provided
vars:
  vm_ip: "{{ vm_ip | default('') }}"

When the variable name matches the template reference AND the variable is not provided as an extra var, Ansible enters infinite recursion. The fix: don't self-reference. Use the ffsdn_* variables directly without redeclaring them in vars blocks.

AWX EE runs at /runner/project/, not /var/lib/awx/projects/

Files pushed to the AWX task pod at /var/lib/awx/projects/incus-contrib/ are mounted in the EE container at /runner/project/. All file paths in playbooks must use the /runner/project/ prefix.

Bridge network not routable from AWX VM

Containers on incusbr0 (10.207.217.0/24) are NOT reachable from the AWX VM (192.168.102.161). The bridge is NAT'd outbound only; IncusOS nodes don't forward inbound traffic from the management network to the bridge. Adding a static route on the AWX VM (ip route add 10.207.217.0/24 via 192.168.102.140) doesn't work because the node's nftables rules block non-established connections from external interfaces to the bridge.

Solution: use the Incus REST API instead of SSH. The cluster API (192.168.102.140:8443) is always reachable, and provides file push + exec endpoints that work regardless of instance network topology.

Aether cluster AWX config API bug

PUT /api/clusters/{id}/awx-config returns {"error":"Invalid cluster ID"} for all valid cluster IDs. Tested with different CSRF tokens, session cookies, request formats (JSON fields, POST vs PUT). The endpoint is not documented in Aether's swagger.yaml. Use the Aether UI (cluster Settings tab or per-blueprint configuration) instead.

AWX web pod OOMKill at 1 GiB

The AWX web pod (nginx + uwsgi) needs at least 2 GiB memory limit. With 1 GiB, it starts successfully but gets OOMKilled under load. Set in the AWX CR: web_resource_requirements.limits.memory: 2Gi.

AWX PVC needs ReadWriteOnce

K3s local-path provisioner creates hostPath volumes which only support ReadWriteOnce. If the AWX CR specifies ReadWriteMany, the PVC stays Pending and PostgreSQL can't start. Set: postgres_storage_class: local-path and ensure ReadWriteOnce access mode.

Troubleshooting

Pod status

# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>

Job output

# Get real-time job output (text format)
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/stdout/?format=txt \
    -H "Authorization: Bearer $AWX_TOKEN"

# Check job status
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); \
        print('status:', d['status'], 'failed:', d['failed'])"

Common issues

AWX pods stuck in Pending: insufficient resources or PVC issue. Check:

incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
incus exec oc-node-02:awx -- kubectl -n awx get pvc

Job fails with recursive template error: self-referencing var definition. Check playbook vars blocks for patterns like var: "{{ var | default('') }}".

Job fails with "No such file or directory": cert/key not at the correct path. Remember: EE uses /runner/project/, not /var/lib/awx/projects/.

Job fails with "Connection failure": Incus API not reachable from AWX pod, or client cert not trusted by cluster. Test from inside the pod:

incus exec oc-node-02:awx -- kubectl -n awx exec <task-pod> -c awx-ee -- \
    curl -sk --cert /runner/project/incus-client.crt \
    --key /runner/project/incus-client.key \
    https://192.168.102.140:8443/1.0

Aether deploy rolls back immediately: AWX job failed. Check the AWX job output for the specific error. Common causes: template misconfiguration, missing credential attachment, playbook syntax errors.

Useful commands

# AWX version
curl -sk http://192.168.102.161:30080/api/v2/ping/

# List job templates
curl -sk http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{t[\"id\"]:3d} {t[\"name\"]}') \
        for t in json.load(sys.stdin)['results']]"

# List recent jobs
curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
        for j in json.load(sys.stdin)['results']]"

# Manually trigger post-deploy (replace 10 with your template ID)
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {
        "ffsdn_instance_name": "test-ct",
        "ffsdn_instance_ip": "10.207.217.99",
        "ffsdn_cluster_name": "oc-lab-cluster",
        "ffsdn_deployed_by": "admin",
        "ffsdn_image_os": "Debian",
        "ffsdn_image_release": "bookworm"
    }}'

Rollback

If AWX is broken beyond repair:

incus delete oc-node-02:awx --force
# Then redeploy from Step 1

The AWX VM is a standalone workload with no cluster dependencies. Destroying and redeploying has zero impact on the Incus cluster or other workloads.

28 KiB Raw Blame History