incus-contrib/notes/awx-guide.md

# AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST
API for running Ansible playbooks. Aether integrates with AWX to run
post-deploy and decommission playbooks as lifecycle hooks on every instance
create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to
Aether, writing lifecycle playbooks, and using the `deploy-awx` management
script.

## Architecture

```
  User deploys VM via Aether
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: vm_name, vm_ip, environment, owner, cost_center
          |
          v
  AWX runs post-deploy.yml playbook
    1. SSH to new instance (vm_ip)
    2. Set hostname, install packages, configure timezone
    3. Log to deployment ledger
          |
          v
  Instance ready, user notified
```

Decommission works in reverse — Aether triggers the decommission template
before deleting the instance.

### Separation of concerns

| Layer | Tool | Responsibility |
|-------|------|----------------|
| Infrastructure | Aether + Incus | VM creation, networking, resources, lifecycle |
| Configuration | AWX + Ansible | OS config, packages, services, integrations |
| Automation glue | Aether AWX binding | Triggers playbooks at create/delete |

### What Aether passes to AWX

Every AWX job launched by Aether receives these extra vars:

| Variable | Description |
|----------|-------------|
| `vm_name` | Instance hostname |
| `vm_ip` | IP address assigned by Aether |
| `environment` | Deployment environment (dev, staging, prod) |
| `owner` | VM owner or team |
| `cost_center` | Billing/cost allocation |
| (custom tags) | All blueprint tags and instance metadata |

### AWX API endpoints Aether uses

| Endpoint | Purpose |
|----------|---------|
| `GET /api/v2/ping/` | Health check |
| `GET /api/v2/job_templates/{id}/` | Validate template exists |
| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
| `GET /api/v2/jobs/{id}/` | Poll job status |

## Lab deployment

### Resource requirements

| Resource | Recommended | Minimum |
|----------|-------------|---------|
| vCPU | 4 | 2 |
| RAM | 8 GiB | 4 GiB |
| Disk | 40 GiB | 20 GiB |

K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM
can run AWX but leaves no headroom for execution environments.

### Lab details

| Setting | Value |
|---------|-------|
| VM name | `awx` |
| Location | oc-node-02 |
| IP | 192.168.102.161/22 (VLAN 69) |
| Gateway | 192.168.100.1 |
| DNS | 192.168.100.1 |
| Port | 443 (HTTPS via K3s Traefik) |
| OS | Debian 12 |
| K8s | K3s (single-node) |
| AWX Operator | 2.19.1 |

IP is adjacent to Aether at .160 on the same VLAN.

### Quick deploy

```bash
# Check prerequisites
./incusos/deploy-awx --doctor

# Full deployment (~10-15 minutes)
./incusos/deploy-awx --deploy

# Configure AWX (project, inventory, templates)
./incusos/deploy-awx --configure

# Register with Aether
./incusos/deploy-awx --join-aether
```

### Manual deployment

#### Step 1: Create VM

```bash
# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Switch to macvlan for direct VLAN access
incus config device remove oc-node-02:awx eth0
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt
```

#### Step 2: Configure static IP

```bash
incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx
    cat > /etc/netplan/50-static.yaml << 'NETPLAN'
network:
  version: 2
  ethernets:
    enp5s0:
      addresses: [192.168.102.161/22]
      routes:
        - to: default
          via: 192.168.100.1
      nameservers:
        addresses: [192.168.100.1]
NETPLAN
    chmod 600 /etc/netplan/50-static.yaml
    netplan apply
"
```

#### Step 3: Install K3s

```bash
incus exec oc-node-02:awx -- bash -c "
    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"
```

#### Step 4: Deploy AWX Operator

```bash
incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"
```

#### Step 5: Deploy AWX instance

The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
Push and apply:

```bash
incus file push incusos/awx-manifests/base/awx.yaml \
    oc-node-02:awx/opt/awx/base/awx.yaml
incus file push incusos/awx-manifests/base/kustomization.yaml \
    oc-node-02:awx/opt/awx/base/kustomization.yaml

incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w
```

#### Step 6: Verify

```bash
# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API
curl -sk https://192.168.102.161/api/v2/ping/
```

Web UI: `https://192.168.102.161/` — login with `admin` / password above.

## AWX configuration

All configuration is API-driven so it can be reproduced. The `deploy-awx
--configure` command automates these steps.

### Project (Git repository)

Create a project pointing to the `ansible/` directory in this repo:

```bash
curl -sk -X POST https://192.168.102.161/api/v2/projects/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "git",
        "scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git",
        "scm_branch": "master",
        "scm_update_on_launch": true
    }'
```

AWX clones the repo and discovers playbooks under `ansible/playbooks/`.

### Inventory

Aether passes `vm_ip` as an extra var. Playbooks use `add_host` to
dynamically create an in-memory inventory entry for the target instance.
The AWX inventory is a placeholder:

```bash
curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'
```

### Credentials

Two credentials:

1. **SCM credential** (type 2) — SSH key for cloning the private Git repo
2. **Machine credential** (type 1) — SSH key for connecting to managed instances

```bash
# Machine credential
curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-instances",
        "organization": 1,
        "credential_type": 1,
        "inputs": {"username": "root", "ssh_key_data": "..."}
    }'
```

### Job templates

Two templates matching Aether's lifecycle hooks:

| Template | Playbook | Purpose |
|----------|----------|---------|
| `post-deploy` | `ansible/playbooks/post-deploy.yml` | Runs after instance creation |
| `decommission` | `ansible/playbooks/decommission.yml` | Runs before instance deletion |

Both must have `ask_variables_on_launch: true` so Aether can inject extra vars.

```bash
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": <PROJECT_ID>,
        "playbook": "ansible/playbooks/post-deploy.yml",
        "inventory": <INVENTORY_ID>,
        "ask_variables_on_launch": true
    }'
```

### Personal Access Token (for Aether)

```bash
AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
```

## Aether integration

### Register AWX endpoint

In the Aether web UI: Settings → Ansible Automation (`/awx-endpoints`) →
Add Endpoint:

| Field | Value |
|-------|-------|
| Name | `lab-awx` |
| URL | `https://192.168.102.161` |
| Token | (PAT from above) |
| Verify SSL | unchecked (self-signed) |

Or via API:
```bash
curl -sSk -b cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "https://192.168.102.161",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'
```

### Bind cluster to AWX

In the Aether UI: AWX endpoint → cluster config, or via API:

```bash
curl -sSk -b cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \
    -H "Content-Type: application/json" \
    -d '{
        "cluster_id": 52,
        "awx_endpoint_id": <ENDPOINT_ID>,
        "awx_post_deploy_template_id": <POST_DEPLOY_TEMPLATE_ID>,
        "awx_decommission_template_id": <DECOMMISSION_TEMPLATE_ID>,
        "awx_job_timeout_seconds": 600
    }'
```

### Verify integration

- Aether Health page (`/health`) shows AWX endpoint as "Healthy"
- Deploy a test instance via Aether — AWX post-deploy job should trigger
- Check AWX job history for the completed run
- Delete the test instance — decommission job should trigger

### Lifecycle behavior

| Hook | Trigger | On failure |
|------|---------|------------|
| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
| Decommission | Before instance deletion | Failure does NOT block deletion |

Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
state (successful, failed, error) or the timeout expires.

## Writing playbooks

### Directory structure

```
ansible/
├── ansible.cfg                    # Project-level config
├── playbooks/
│   ├── post-deploy.yml            # Aether post-deploy hook
│   └── decommission.yml           # Aether decommission hook
└── roles/
    └── base-config/
        ├── tasks/main.yml         # Base OS configuration
        └── templates/             # Jinja2 templates (future)
```

### Pattern: dynamic host from extra vars

Aether passes `vm_ip` — playbooks use `add_host` to target it:

```yaml
- name: Add target host
  hosts: localhost
  gather_facts: false
  tasks:
    - ansible.builtin.add_host:
        name: "{{ vm_ip }}"
        groups: target
        ansible_user: root

- name: Configure target
  hosts: target
  tasks:
    - ansible.builtin.wait_for_connection:
        delay: 5
        timeout: 300
    # ... configuration tasks
```

### Pattern: best-effort decommission

Decommission playbooks should never block instance deletion:

```yaml
- name: Attempt cleanup on target
  hosts: target
  ignore_unreachable: true
  tasks:
    - name: Check if host is reachable
      ansible.builtin.wait_for_connection:
        timeout: 15
      ignore_errors: true
      register: host_check

    - name: Cleanup tasks
      # ...
      when: host_check is succeeded
```

### The base-config role

The included `base-config` role performs minimal post-deploy setup:

1. Set hostname
2. Update apt cache
3. Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades)
4. Set timezone to UTC
5. Enable unattended security upgrades
6. Write deployment metadata to `/etc/deploy-info`

### Extending playbooks

Common additions for production environments:

| Task | Method | Module |
|------|--------|--------|
| DNS registration | API from AWX controller | `community.general.nsupdate` or REST |
| Monitoring agent | SSH to target | `ansible.builtin.package` + config |
| SSH key distribution | SSH to target | `ansible.posix.authorized_key` |
| Certificate provisioning | API from controller | `community.crypto.acme_certificate` |
| Backup enrollment | API from controller | REST calls to backup system |

External system tasks use `delegate_to: localhost` to run from the AWX
controller rather than on the target instance.

## The deploy-awx script

### Usage

```
deploy-awx [OPTIONS]

Actions:
    --deploy          Full deploy: create VM → K3s → AWX → configure
    --status          Check AWX health, K3s pods, Aether connection
    --heal            Restart failed pods, re-sync project, re-validate
    --configure       (Re-)configure AWX: project, inventory, templates
    --join-aether     Register AWX with Aether + print binding instructions
    --cleanup         Destroy the AWX VM
    --doctor          Check prerequisites

Options:
    -c, --config FILE     Configuration file (YAML with awx: section)
    -n, --dry-run         Preview actions without executing
    -v, --verbose         Show detailed output
    -q, --quiet           Suppress informational output
```

### Configuration

Defaults are built into the script for the lab environment. Override with
a YAML config file:

```yaml
awx:
  vm_name: awx
  target_node: oc-node-02
  ip: 192.168.102.161/22
  gateway: 192.168.100.1
  dns: 192.168.100.1
  cpu: 4
  memory: 8GiB
  disk: 40GiB
  git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
  git_branch: master
  aether_url: https://192.168.102.160:8443
  aether_cluster_id: 52
```

### Deployment phases

1. **Create VM** — `incus launch images:debian/12` with macvlan NIC
2. **Configure network** — static IP via netplan, hostname
3. **Install K3s** — single-node K3s with kubeconfig mode 644
4. **Deploy AWX Operator** — kustomize from `awx-manifests/operator/`
5. **Deploy AWX instance** — kustomize from `awx-manifests/base/`
6. **Verify** — API ping, admin password retrieval

### Heal logic

The `--heal` command checks and fixes:

1. K3s service running → restart if not
2. All AWX pods healthy → delete unhealthy (operator recreates)
3. AWX web responding → restart deployments if not
4. Project sync current → trigger re-sync

### Manifests

K8s manifests stored in `incusos/awx-manifests/`:

```
awx-manifests/
├── operator/
│   └── kustomization.yaml    # AWX Operator 2.19.1 via kustomize
└── base/
    ├── kustomization.yaml    # AWX instance resources
    └── awx.yaml              # AWX custom resource (pods, storage, ingress)
```

The AWX CR in `awx.yaml` defines resource limits tuned for the 8 GiB VM:
- Web pod: 512 MiB → 1 GiB
- Task pod: 512 MiB → 2 GiB
- EE pod: 256 MiB → 1 GiB
- PostgreSQL: 256 MiB → 512 MiB

## Troubleshooting

### Pod status

```bash
# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
```

### Common issues

**AWX pods stuck in Pending**: insufficient resources. Check node capacity:
```bash
incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources"
```

**AWX web returns 502**: task or web pod crashed. Restart:
```bash
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task
```

**Project sync fails**: SCM credential or network issue. Check:
```bash
incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm
```

**Job fails with "Host unreachable"**: target instance not booted yet or
SSH not available. Increase the `wait_for_connection` timeout in the playbook
or add a delay in Aether's job timeout setting.

**K3s not starting**: VM resources exhausted. Check:
```bash
incus exec oc-node-02:awx -- systemctl status k3s
incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50
```

**Cannot reach AWX from outside**: macvlan NIC misconfigured. Verify:
```bash
incus exec oc-node-02:awx -- ip addr show enp5s0
incus exec oc-node-02:awx -- ip route
```

### Useful commands

```bash
# AWX version
curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool

# List job templates
curl -sk https://192.168.102.161/api/v2/job_templates/ \
    --user "admin:$PW" | python3 -m json.tool

# List recent jobs
curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \
    --user "admin:$PW" | python3 -c "
import sys,json
for j in json.load(sys.stdin)['results']:
    print(f'{j[\"id\"]:4d}  {j[\"status\"]:12s}  {j[\"name\"]}')"

# Manually trigger post-deploy
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \
    --user "admin:$PW" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}'
```

### Rollback

If AWX is broken beyond repair:
```bash
./incusos/deploy-awx --cleanup
./incusos/deploy-awx --deploy
./incusos/deploy-awx --configure
./incusos/deploy-awx --join-aether
```

The AWX VM is a standalone workload with no cluster dependencies. Destroying
and redeploying has zero impact on the Incus cluster or other workloads.