17 KiB

Raw Blame History

AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST API for running Ansible playbooks. Aether integrates with AWX to run post-deploy and decommission playbooks as lifecycle hooks on every instance create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to Aether, writing lifecycle playbooks, and using the deploy-awx management script.

Architecture

  User deploys VM via Aether
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: vm_name, vm_ip, environment, owner, cost_center
          |
          v
  AWX runs post-deploy.yml playbook
    1. SSH to new instance (vm_ip)
    2. Set hostname, install packages, configure timezone
    3. Log to deployment ledger
          |
          v
  Instance ready, user notified

Decommission works in reverse — Aether triggers the decommission template before deleting the instance.

Separation of concerns

Layer	Tool	Responsibility
Infrastructure	Aether + Incus	VM creation, networking, resources, lifecycle
Configuration	AWX + Ansible	OS config, packages, services, integrations
Automation glue	Aether AWX binding	Triggers playbooks at create/delete

What Aether passes to AWX

Every AWX job launched by Aether receives these extra vars:

Variable	Description
`vm_name`	Instance hostname
`vm_ip`	IP address assigned by Aether
`environment`	Deployment environment (dev, staging, prod)
`owner`	VM owner or team
`cost_center`	Billing/cost allocation
(custom tags)	All blueprint tags and instance metadata

AWX API endpoints Aether uses

Endpoint	Purpose
`GET /api/v2/ping/`	Health check
`GET /api/v2/job_templates/{id}/`	Validate template exists
`POST /api/v2/job_templates/{id}/launch/`	Trigger job with extra_vars
`GET /api/v2/jobs/{id}/`	Poll job status

Lab deployment

Resource requirements

Resource	Recommended	Minimum
vCPU	4	2
RAM	8 GiB	4 GiB
Disk	40 GiB	20 GiB

K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM can run AWX but leaves no headroom for execution environments.

Lab details

Setting	Value
VM name	`awx`
Location	oc-node-02
IP	192.168.102.161/22 (VLAN 69)
Gateway	192.168.100.1
DNS	192.168.100.1
Port	443 (HTTPS via K3s Traefik)
OS	Debian 12
K8s	K3s (single-node)
AWX Operator	2.19.1

IP is adjacent to Aether at .160 on the same VLAN.

Quick deploy

# Check prerequisites
./incusos/deploy-awx --doctor

# Full deployment (~10-15 minutes)
./incusos/deploy-awx --deploy

# Configure AWX (project, inventory, templates)
./incusos/deploy-awx --configure

# Register with Aether
./incusos/deploy-awx --join-aether

Manual deployment

Step 1: Create VM

# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Switch to macvlan for direct VLAN access
incus config device remove oc-node-02:awx eth0
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt

Step 2: Configure static IP

incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx
    cat > /etc/netplan/50-static.yaml << 'NETPLAN'
network:
  version: 2
  ethernets:
    enp5s0:
      addresses: [192.168.102.161/22]
      routes:
        - to: default
          via: 192.168.100.1
      nameservers:
        addresses: [192.168.100.1]
NETPLAN
    chmod 600 /etc/netplan/50-static.yaml
    netplan apply
"

Step 3: Install K3s

incus exec oc-node-02:awx -- bash -c "
    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"

Step 4: Deploy AWX Operator

incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"

Step 5: Deploy AWX instance

The AWX custom resource is defined in incusos/awx-manifests/base/awx.yaml. Push and apply:

incus file push incusos/awx-manifests/base/awx.yaml \
    oc-node-02:awx/opt/awx/base/awx.yaml
incus file push incusos/awx-manifests/base/kustomization.yaml \
    oc-node-02:awx/opt/awx/base/kustomization.yaml

incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w

Step 6: Verify

# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API
curl -sk https://192.168.102.161/api/v2/ping/

Web UI: https://192.168.102.161/ — login with admin / password above.

AWX configuration

All configuration is API-driven so it can be reproduced. The deploy-awx --configure command automates these steps.

Project (Git repository)

Create a project pointing to the ansible/ directory in this repo:

curl -sk -X POST https://192.168.102.161/api/v2/projects/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "git",
        "scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git",
        "scm_branch": "master",
        "scm_update_on_launch": true
    }'

AWX clones the repo and discovers playbooks under ansible/playbooks/.

Inventory

Aether passes vm_ip as an extra var. Playbooks use add_host to dynamically create an in-memory inventory entry for the target instance. The AWX inventory is a placeholder:

curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'

Credentials

Two credentials:

SCM credential (type 2) — SSH key for cloning the private Git repo
Machine credential (type 1) — SSH key for connecting to managed instances

# Machine credential
curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-instances",
        "organization": 1,
        "credential_type": 1,
        "inputs": {"username": "root", "ssh_key_data": "..."}
    }'

Job templates

Two templates matching Aether's lifecycle hooks:

Template	Playbook	Purpose
`post-deploy`	`ansible/playbooks/post-deploy.yml`	Runs after instance creation
`decommission`	`ansible/playbooks/decommission.yml`	Runs before instance deletion

Both must have ask_variables_on_launch: true so Aether can inject extra vars.

curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": <PROJECT_ID>,
        "playbook": "ansible/playbooks/post-deploy.yml",
        "inventory": <INVENTORY_ID>,
        "ask_variables_on_launch": true
    }'

Personal Access Token (for Aether)

AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

Aether integration

Register AWX endpoint

In the Aether web UI: Settings → Ansible Automation (/awx-endpoints) → Add Endpoint:

Field	Value
Name	`lab-awx`
URL	`https://192.168.102.161`
Token	(PAT from above)
Verify SSL	unchecked (self-signed)

Or via API:

curl -sSk -b cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "https://192.168.102.161",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'

Bind cluster to AWX

In the Aether UI: AWX endpoint → cluster config, or via API:

curl -sSk -b cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \
    -H "Content-Type: application/json" \
    -d '{
        "cluster_id": 52,
        "awx_endpoint_id": <ENDPOINT_ID>,
        "awx_post_deploy_template_id": <POST_DEPLOY_TEMPLATE_ID>,
        "awx_decommission_template_id": <DECOMMISSION_TEMPLATE_ID>,
        "awx_job_timeout_seconds": 600
    }'

Verify integration

Aether Health page (/health) shows AWX endpoint as "Healthy"
Deploy a test instance via Aether — AWX post-deploy job should trigger
Check AWX job history for the completed run
Delete the test instance — decommission job should trigger

Lifecycle behavior

Hook	Trigger	On failure
Post-deploy	After instance creation	Aether auto-rollbacks (deletes instance)
Decommission	Before instance deletion	Failure does NOT block deletion

Aether polls GET /api/v2/jobs/{id}/ until the job reaches a terminal state (successful, failed, error) or the timeout expires.

Writing playbooks

Directory structure

ansible/
├── ansible.cfg                    # Project-level config
├── playbooks/
│   ├── post-deploy.yml            # Aether post-deploy hook
│   └── decommission.yml           # Aether decommission hook
└── roles/
    └── base-config/
        ├── tasks/main.yml         # Base OS configuration
        └── templates/             # Jinja2 templates (future)

Pattern: dynamic host from extra vars

Aether passes vm_ip — playbooks use add_host to target it:

- name: Add target host
  hosts: localhost
  gather_facts: false
  tasks:
    - ansible.builtin.add_host:
        name: "{{ vm_ip }}"
        groups: target
        ansible_user: root

- name: Configure target
  hosts: target
  tasks:
    - ansible.builtin.wait_for_connection:
        delay: 5
        timeout: 300
    # ... configuration tasks

Pattern: best-effort decommission

Decommission playbooks should never block instance deletion:

- name: Attempt cleanup on target
  hosts: target
  ignore_unreachable: true
  tasks:
    - name: Check if host is reachable
      ansible.builtin.wait_for_connection:
        timeout: 15
      ignore_errors: true
      register: host_check

    - name: Cleanup tasks
      # ...
      when: host_check is succeeded

The base-config role

The included base-config role performs minimal post-deploy setup:

Set hostname
Update apt cache
Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades)
Set timezone to UTC
Enable unattended security upgrades
Write deployment metadata to /etc/deploy-info

Extending playbooks

Common additions for production environments:

Task	Method	Module
DNS registration	API from AWX controller	`community.general.nsupdate` or REST
Monitoring agent	SSH to target	`ansible.builtin.package` + config
SSH key distribution	SSH to target	`ansible.posix.authorized_key`
Certificate provisioning	API from controller	`community.crypto.acme_certificate`
Backup enrollment	API from controller	REST calls to backup system

External system tasks use delegate_to: localhost to run from the AWX controller rather than on the target instance.

The deploy-awx script

Usage

deploy-awx [OPTIONS]

Actions:
    --deploy          Full deploy: create VM → K3s → AWX → configure
    --status          Check AWX health, K3s pods, Aether connection
    --heal            Restart failed pods, re-sync project, re-validate
    --configure       (Re-)configure AWX: project, inventory, templates
    --join-aether     Register AWX with Aether + print binding instructions
    --cleanup         Destroy the AWX VM
    --doctor          Check prerequisites

Options:
    -c, --config FILE     Configuration file (YAML with awx: section)
    -n, --dry-run         Preview actions without executing
    -v, --verbose         Show detailed output
    -q, --quiet           Suppress informational output

Configuration

Defaults are built into the script for the lab environment. Override with a YAML config file:

awx:
  vm_name: awx
  target_node: oc-node-02
  ip: 192.168.102.161/22
  gateway: 192.168.100.1
  dns: 192.168.100.1
  cpu: 4
  memory: 8GiB
  disk: 40GiB
  git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
  git_branch: master
  aether_url: https://192.168.102.160:8443
  aether_cluster_id: 52

Deployment phases

Create VM — incus launch images:debian/12 with macvlan NIC
Configure network — static IP via netplan, hostname
Install K3s — single-node K3s with kubeconfig mode 644
Deploy AWX Operator — kustomize from awx-manifests/operator/
Deploy AWX instance — kustomize from awx-manifests/base/
Verify — API ping, admin password retrieval

Heal logic

The --heal command checks and fixes:

K3s service running → restart if not
All AWX pods healthy → delete unhealthy (operator recreates)
AWX web responding → restart deployments if not
Project sync current → trigger re-sync

Manifests

K8s manifests stored in incusos/awx-manifests/:

awx-manifests/
├── operator/
│   └── kustomization.yaml    # AWX Operator 2.19.1 via kustomize
└── base/
    ├── kustomization.yaml    # AWX instance resources
    └── awx.yaml              # AWX custom resource (pods, storage, ingress)

The AWX CR in awx.yaml defines resource limits tuned for the 8 GiB VM:

Web pod: 512 MiB → 1 GiB
Task pod: 512 MiB → 2 GiB
EE pod: 256 MiB → 1 GiB
PostgreSQL: 256 MiB → 512 MiB

Troubleshooting

Pod status

# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>

Common issues

AWX pods stuck in Pending: insufficient resources. Check node capacity:

incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources"

AWX web returns 502: task or web pod crashed. Restart:

incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task

Project sync fails: SCM credential or network issue. Check:

incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm

Job fails with "Host unreachable": target instance not booted yet or SSH not available. Increase the wait_for_connection timeout in the playbook or add a delay in Aether's job timeout setting.

K3s not starting: VM resources exhausted. Check:

incus exec oc-node-02:awx -- systemctl status k3s
incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50

Cannot reach AWX from outside: macvlan NIC misconfigured. Verify:

incus exec oc-node-02:awx -- ip addr show enp5s0
incus exec oc-node-02:awx -- ip route

Useful commands

# AWX version
curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool

# List job templates
curl -sk https://192.168.102.161/api/v2/job_templates/ \
    --user "admin:$PW" | python3 -m json.tool

# List recent jobs
curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \
    --user "admin:$PW" | python3 -c "
import sys,json
for j in json.load(sys.stdin)['results']:
    print(f'{j[\"id\"]:4d}  {j[\"status\"]:12s}  {j[\"name\"]}')"

# Manually trigger post-deploy
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \
    --user "admin:$PW" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}'

Rollback

If AWX is broken beyond repair:

./incusos/deploy-awx --cleanup
./incusos/deploy-awx --deploy
./incusos/deploy-awx --configure
./incusos/deploy-awx --join-aether

The AWX VM is a standalone workload with no cluster dependencies. Destroying and redeploying has zero impact on the Incus cluster or other workloads.

17 KiB Raw Blame History