maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

WIP: Add AWX deployment, Ansible playbooks, and Aether integration 

Work-in-progress commit to enable AWX project sync from git.
Files will be updated with validated output from real deployment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-24 00:04:28 +01:00
parent fd30ef27d4
commit 6a74925fc4
11 changed files with 2139 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

35
CLAUDE.md
View File

@ -13,11 +13,21 @@ incu-contrib/
├── CLAUDE.md # This file -- project context ├── CLAUDE.md # This file -- project context
├── README.md # Main overview ├── README.md # Main overview
├── .gitignore ├── .gitignore
├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks
│ ├── ansible.cfg # Project-level Ansible config
│ ├── playbooks/
│ │ ├── post-deploy.yml # Runs after Aether creates an instance
│ │ └── decommission.yml # Runs before Aether deletes an instance
│ └── roles/
│ └── base-config/ # Base OS configuration role
│ └── tasks/main.yml
├── incusos/ # IncusOS installation tooling ├── incusos/ # IncusOS installation tooling
│ ├── README.md # Detailed usage docs │ ├── README.md # Detailed usage docs
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
│ ├── deploy-awx # AWX deployment + management on Incus cluster
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
│ ├── lab-test # Guided lab validation (12 test phases) │ ├── lab-test # Guided lab validation (12 test phases)
│ ├── observe-deploy # Single-VM deploy with rapid console screenshots │ ├── observe-deploy # Single-VM deploy with rapid console screenshots
│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) │ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored)
@ -33,6 +43,7 @@ incu-contrib/
├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end) ├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end)
├── migration-guide.md # Migration paths into Incus from other hypervisors ├── migration-guide.md # Migration paths into Incus from other hypervisors
├── aether-guide.md # Aether management platform (deploy, blueprints, API) ├── aether-guide.md # Aether management platform (deploy, blueprints, API)
├── awx-guide.md # AWX + Aether Ansible automation
├── incus-version-compatibility.md # Incus versions across platforms ├── incus-version-compatibility.md # Incus versions across platforms
├── iso-download-methods.md # ISO download/customization research ├── iso-download-methods.md # ISO download/customization research
└── utm-support.md # UTM support design document (future) └── utm-support.md # UTM support design document (future)
@ -809,6 +820,30 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
- No `blockstat` equivalent -- install detection must use timeout + port polling. - No `blockstat` equivalent -- install detection must use timeout + port polling.
- Seed generation already works cross-platform (Phase 3 macOS compatibility). - Seed generation already works cross-platform (Phase 3 macOS compatibility).
### AWX integration (Ansible automation for Aether)
- **AWX** is the open-source Ansible automation platform (upstream of Ansible
Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
`--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
- **Ansible playbooks** in `ansible/` directory:
- `playbooks/post-deploy.yml` -- runs after Aether creates an instance
- `playbooks/decommission.yml` -- runs before Aether deletes an instance
- `roles/base-config/` -- hostname, packages, timezone, deploy metadata
- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with
a Personal Access Token. Per-cluster config binds post-deploy and
decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/`
with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`).
- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
deleted). Decommission failure does NOT block deletion.
- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to
create dynamic inventory → SSH to target for configuration. External system
tasks use `delegate_to: localhost`.
- See `notes/awx-guide.md` for full deployment guide and troubleshooting.
## Git workflow ## Git workflow
- Main branch: `main` - Main branch: `main`

17
README.md
View File

@ -23,8 +23,22 @@ Scripts for building IncusOS installation media and seed configurations:
YAML, and the script handles seed generation, ISO upload, VM creation with correct YAML, and the script handles seed generation, ISO upload, VM creation with correct
settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation. settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation.
- **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus
cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template
configuration, and Aether integration.
See [`incusos/README.md`](incusos/README.md) for full documentation. See [`incusos/README.md`](incusos/README.md) for full documentation.
### [`ansible/`](ansible/)
Ansible playbooks and roles for AWX/Aether lifecycle hooks:
- **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance.
Configures hostname, installs base packages, sets timezone, logs deployment.
- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
Graceful service shutdown and decommission logging (best-effort).
- **`roles/base-config/`** -- Base OS configuration role used by post-deploy.
### [`notes/`](notes/) ### [`notes/`](notes/)
Guides and reference material (roughly in learning order): Guides and reference material (roughly in learning order):
@ -44,6 +58,9 @@ Guides and reference material (roughly in learning order):
- [`aether-guide.md`](notes/aether-guide.md) -- - [`aether-guide.md`](notes/aether-guide.md) --
Aether management platform: deployment, instance/blueprint lifecycle, API Aether management platform: deployment, instance/blueprint lifecycle, API
reference, OVN recovery, and node resizing. reference, OVN recovery, and node resizing.
- [`awx-guide.md`](notes/awx-guide.md) --
AWX deployment, Aether integration, Ansible playbook patterns, and
the deploy-awx management script.
- [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) -- - [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) --
Incus package versions across platforms and install instructions. Incus package versions across platforms and install instructions.
- [`utm-support.md`](notes/utm-support.md) -- - [`utm-support.md`](notes/utm-support.md) --

28
ansible/ansible.cfg Normal file
View File

@ -0,0 +1,28 @@
[defaults]
# Inventory — AWX manages inventory; this is for local testing only
inventory = localhost,
# Roles path
roles_path = roles
# Reduce SSH overhead
forks = 10
timeout = 30
# Retry files clutter the repo
retry_files_enabled = False
# Human-readable output
stdout_callback = yaml
# Host key checking — disabled for dynamic lab instances
host_key_checking = False
[privilege_escalation]
become = True
become_method = sudo
[ssh_connection]
# Speed up SSH connections
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no

78
ansible/playbooks/decommission.yml Normal file
View File

@ -0,0 +1,78 @@
---
# decommission.yml — Runs before Aether deletes an instance
#
# Aether passes the same extra vars as post-deploy:
# vm_name, vm_ip, environment, owner, cost_center
#
# This playbook runs best-effort: failures do NOT block instance
# deletion in Aether. The playbook should be defensive and handle
# unreachable hosts gracefully.
- name: Decommission — prepare instance for deletion
hosts: localhost
gather_facts: false
connection: local
vars:
vm_name: "{{ vm_name | default('unknown') }}"
vm_ip: "{{ vm_ip | default('') }}"
environment_name: "{{ environment | default('lab') }}"
owner: "{{ owner | default('unassigned') }}"
cost_center: "{{ cost_center | default('') }}"
ssh_user: root
tasks:
- name: Add target host to in-memory inventory
ansible.builtin.add_host:
name: "{{ vm_ip }}"
groups: target
ansible_user: "{{ ssh_user }}"
ansible_host: "{{ vm_ip }}"
when: vm_ip | length > 0
- name: Attempt graceful shutdown on target
hosts: target
gather_facts: false
ignore_unreachable: true
tasks:
- name: Check if host is reachable
ansible.builtin.wait_for_connection:
delay: 0
timeout: 15
register: host_check
ignore_errors: true
- name: Stop application services (best-effort)
ansible.builtin.shell: |
# Stop user-facing services before deletion
systemctl list-units --type=service --state=running --no-legend \
| awk '{print $1}' \
| grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
| head -20 \
| xargs -r -n1 systemctl stop
register: service_stop
ignore_errors: true
when: host_check is succeeded
- name: Flush logs to persistent storage
ansible.builtin.command: journalctl --flush
ignore_errors: true
when: host_check is succeeded
- name: Log decommission
hosts: localhost
gather_facts: false
connection: local
tasks:
- name: Record decommission in ledger
ansible.builtin.lineinfile:
path: /var/log/awx-deploy-ledger.log
line: >-
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }}
environment={{ environment_name }} owner={{ owner }}
cost_center={{ cost_center }} status=completed
create: true
mode: "0644"

88
ansible/playbooks/post-deploy.yml Normal file
View File

@ -0,0 +1,88 @@
---
# post-deploy.yml — Runs after Aether creates an instance
#
# Aether passes extra vars:
# vm_name — instance hostname
# vm_ip — IP address assigned by Aether
# environment — deployment environment (dev, staging, prod)
# owner — VM owner or team
# cost_center — billing/cost allocation
# (plus any custom tags from the blueprint)
#
# AWX job template must have "ask_variables_on_launch: true" so Aether
# can inject these variables at runtime.
- name: Post-deploy — configure newly provisioned instance
hosts: localhost
gather_facts: false
connection: local
vars:
# Defaults (overridden by Aether extra vars)
vm_name: "{{ vm_name | default('unknown') }}"
vm_ip: "{{ vm_ip | default('') }}"
environment_name: "{{ environment | default('lab') }}"
owner: "{{ owner | default('unassigned') }}"
cost_center: "{{ cost_center | default('') }}"
ssh_user: root
base_packages:
- curl
- vim
- htop
- jq
- tmux
- rsync
- unattended-upgrades
pre_tasks:
- name: Validate required variables
ansible.builtin.assert:
that:
- vm_ip | length > 0
- vm_name | length > 0
fail_msg: "vm_ip and vm_name must be provided by Aether"
- name: Add target host to in-memory inventory
ansible.builtin.add_host:
name: "{{ vm_ip }}"
groups: target
ansible_user: "{{ ssh_user }}"
ansible_host: "{{ vm_ip }}"
- name: Wait for instance and apply base configuration
hosts: target
gather_facts: false
pre_tasks:
- name: Wait for SSH connectivity (up to 5 minutes)
ansible.builtin.wait_for_connection:
delay: 5
timeout: 300
- name: Gather facts after connection
ansible.builtin.setup:
roles:
- role: base-config
vars:
hostname: "{{ hostvars['localhost']['vm_name'] }}"
deploy_environment: "{{ hostvars['localhost']['environment_name'] }}"
deploy_owner: "{{ hostvars['localhost']['owner'] }}"
packages: "{{ hostvars['localhost']['base_packages'] }}"
- name: Log deployment
hosts: localhost
gather_facts: false
connection: local
tasks:
- name: Record deployment in ledger
ansible.builtin.lineinfile:
path: /var/log/awx-deploy-ledger.log
line: >-
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }}
environment={{ environment_name }} owner={{ owner }}
cost_center={{ cost_center }} status=success
create: true
mode: "0644"

53
ansible/roles/base-config/tasks/main.yml Normal file
View File

@ -0,0 +1,53 @@
---
# base-config — Minimal post-deploy configuration for new instances
#
# Variables (passed from post-deploy.yml):
# hostname — desired hostname
# deploy_environment — dev/staging/prod
# deploy_owner — owner tag
# packages — list of packages to install
- name: Set hostname
ansible.builtin.hostname:
name: "{{ hostname }}"
- name: Add hostname to /etc/hosts
ansible.builtin.lineinfile:
path: /etc/hosts
regexp: '^127\.0\.1\.1'
line: "127.0.1.1 {{ hostname }}"
- name: Update apt cache
ansible.builtin.apt:
update_cache: true
cache_valid_time: 3600
when: ansible_os_family == "Debian"
- name: Install base packages
ansible.builtin.apt:
name: "{{ packages }}"
state: present
when: ansible_os_family == "Debian"
- name: Set timezone to UTC
community.general.timezone:
name: UTC
- name: Enable unattended security upgrades
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
content: |
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
mode: "0644"
when: ansible_os_family == "Debian"
- name: Add deployment metadata to /etc/deploy-info
ansible.builtin.copy:
dest: /etc/deploy-info
content: |
hostname={{ hostname }}
environment={{ deploy_environment }}
owner={{ deploy_owner }}
deployed={{ ansible_date_time.iso8601 }}
mode: "0644"

59
incusos/awx-manifests/base/awx.yaml Normal file
View File

@ -0,0 +1,59 @@
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
namespace: awx
spec:
# NodePort — expose on port 30080 (HTTP) and 30443 (HTTPS)
# Simpler than ingress for direct IP access without DNS
service_type: nodeport
nodeport_port: 30080
# Resource limits — tuned for 8 GiB VM
# Web needs ~1.5 GiB (5 uWSGI workers + daphne + nginx + redis sidecar)
# Task needs ~2 GiB (dispatcher + callback receiver + workers)
web_resource_requirements:
requests:
cpu: 200m
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
task_resource_requirements:
requests:
cpu: 200m
memory: 1Gi
limits:
cpu: "2"
memory: 3Gi
ee_resource_requirements:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
# PostgreSQL — bundled (operator-managed)
postgres_storage_class: local-path
postgres_storage_requirements:
requests:
storage: 8Gi
postgres_resource_requirements:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
# Projects — persistent storage for SCM checkouts
# access_mode must be ReadWriteOnce for local-path (K3s default provisioner)
projects_persistence: true
projects_storage_class: local-path
projects_storage_size: 4Gi
projects_storage_access_mode: ReadWriteOnce
# Admin account
admin_user: admin

8
incusos/awx-manifests/base/kustomization.yaml Normal file
View File

@ -0,0 +1,8 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: awx
resources:
- awx.yaml

12
incusos/awx-manifests/operator/kustomization.yaml Normal file
View File

@ -0,0 +1,12 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
- name: quay.io/ansible/awx-operator
newTag: 2.19.1
namespace: awx

1137
incusos/deploy-awx Executable file
View File

File diff suppressed because it is too large Load Diff

624
notes/awx-guide.md Normal file
View File

@ -0,0 +1,624 @@
# AWX — Ansible Automation for Aether Lifecycle Hooks
AWX (the open-source upstream of Ansible Tower) provides a web UI and REST
API for running Ansible playbooks. Aether integrates with AWX to run
post-deploy and decommission playbooks as lifecycle hooks on every instance
create/delete.
This guide covers deploying AWX on the Incus cluster, connecting it to
Aether, writing lifecycle playbooks, and using the `deploy-awx` management
script.
## Architecture
```
User deploys VM via Aether
|
v
Aether creates instance on Incus cluster
|
v
Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
with extra_vars: vm_name, vm_ip, environment, owner, cost_center
|
v
AWX runs post-deploy.yml playbook
1. SSH to new instance (vm_ip)
2. Set hostname, install packages, configure timezone
3. Log to deployment ledger
|
v
Instance ready, user notified
```
Decommission works in reverse — Aether triggers the decommission template
before deleting the instance.
### Separation of concerns
| Layer | Tool | Responsibility |
|-------|------|----------------|
| Infrastructure | Aether + Incus | VM creation, networking, resources, lifecycle |
| Configuration | AWX + Ansible | OS config, packages, services, integrations |
| Automation glue | Aether AWX binding | Triggers playbooks at create/delete |
### What Aether passes to AWX
Every AWX job launched by Aether receives these extra vars:
| Variable | Description |
|----------|-------------|
| `vm_name` | Instance hostname |
| `vm_ip` | IP address assigned by Aether |
| `environment` | Deployment environment (dev, staging, prod) |
| `owner` | VM owner or team |
| `cost_center` | Billing/cost allocation |
| (custom tags) | All blueprint tags and instance metadata |
### AWX API endpoints Aether uses
| Endpoint | Purpose |
|----------|---------|
| `GET /api/v2/ping/` | Health check |
| `GET /api/v2/job_templates/{id}/` | Validate template exists |
| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
| `GET /api/v2/jobs/{id}/` | Poll job status |
## Lab deployment
### Resource requirements
| Resource | Recommended | Minimum |
|----------|-------------|---------|
| vCPU | 4 | 2 |
| RAM | 8 GiB | 4 GiB |
| Disk | 40 GiB | 20 GiB |
K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM
can run AWX but leaves no headroom for execution environments.
### Lab details
| Setting | Value |
|---------|-------|
| VM name | `awx` |
| Location | oc-node-02 |
| IP | 192.168.102.161/22 (VLAN 69) |
| Gateway | 192.168.100.1 |
| DNS | 192.168.100.1 |
| Port | 443 (HTTPS via K3s Traefik) |
| OS | Debian 12 |
| K8s | K3s (single-node) |
| AWX Operator | 2.19.1 |
IP is adjacent to Aether at .160 on the same VLAN.
### Quick deploy
```bash
# Check prerequisites
./incusos/deploy-awx --doctor
# Full deployment (~10-15 minutes)
./incusos/deploy-awx --deploy
# Configure AWX (project, inventory, templates)
./incusos/deploy-awx --configure
# Register with Aether
./incusos/deploy-awx --join-aether
```
### Manual deployment
#### Step 1: Create VM
```bash
# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
--target oc-node-02 \
-c limits.cpu=4 -c limits.memory=8GiB \
-d root,size=40GiB
# Switch to macvlan for direct VLAN access
incus config device remove oc-node-02:awx eth0
incus config device add oc-node-02:awx eth0 nic \
nictype=macvlan parent=mgmt
```
#### Step 2: Configure static IP
```bash
incus exec oc-node-02:awx -- bash -c "
hostnamectl set-hostname awx
cat > /etc/netplan/50-static.yaml << 'NETPLAN'
network:
version: 2
ethernets:
enp5s0:
addresses: [192.168.102.161/22]
routes:
- to: default
via: 192.168.100.1
nameservers:
addresses: [192.168.100.1]
NETPLAN
chmod 600 /etc/netplan/50-static.yaml
netplan apply
"
```
#### Step 3: Install K3s
```bash
incus exec oc-node-02:awx -- bash -c "
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
kubectl wait --for=condition=Ready node --all --timeout=120s
"
```
#### Step 4: Deploy AWX Operator
```bash
incus exec oc-node-02:awx -- bash -c "
kubectl create namespace awx
mkdir -p /opt/awx/operator
cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
- name: quay.io/ansible/awx-operator
newTag: 2.19.1
namespace: awx
EOF
kubectl apply -k /opt/awx/operator/
# Wait for operator (~2-3 min)
kubectl -n awx wait --for=condition=Available \
deployment/awx-operator-controller-manager --timeout=300s
"
```
#### Step 5: Deploy AWX instance
The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
Push and apply:
```bash
incus file push incusos/awx-manifests/base/awx.yaml \
oc-node-02:awx/opt/awx/base/awx.yaml
incus file push incusos/awx-manifests/base/kustomization.yaml \
oc-node-02:awx/opt/awx/base/kustomization.yaml
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/
# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w
```
#### Step 6: Verify
```bash
# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"
# Test API
curl -sk https://192.168.102.161/api/v2/ping/
```
Web UI: `https://192.168.102.161/` — login with `admin` / password above.
## AWX configuration
All configuration is API-driven so it can be reproduced. The `deploy-awx
--configure` command automates these steps.
### Project (Git repository)
Create a project pointing to the `ansible/` directory in this repo:
```bash
curl -sk -X POST https://192.168.102.161/api/v2/projects/ \
--user "admin:${ADMIN_PW}" \
-H "Content-Type: application/json" \
-d '{
"name": "incus-contrib",
"organization": 1,
"scm_type": "git",
"scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git",
"scm_branch": "master",
"scm_update_on_launch": true
}'
```
AWX clones the repo and discovers playbooks under `ansible/playbooks/`.
### Inventory
Aether passes `vm_ip` as an extra var. Playbooks use `add_host` to
dynamically create an in-memory inventory entry for the target instance.
The AWX inventory is a placeholder:
```bash
curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \
--user "admin:${ADMIN_PW}" \
-H "Content-Type: application/json" \
-d '{"name": "incus-instances", "organization": 1}'
```
### Credentials
Two credentials:
1. **SCM credential** (type 2) — SSH key for cloning the private Git repo
2. **Machine credential** (type 1) — SSH key for connecting to managed instances
```bash
# Machine credential
curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \
--user "admin:${ADMIN_PW}" \
-H "Content-Type: application/json" \
-d '{
"name": "incus-instances",
"organization": 1,
"credential_type": 1,
"inputs": {"username": "root", "ssh_key_data": "..."}
}'
```
### Job templates
Two templates matching Aether's lifecycle hooks:
| Template | Playbook | Purpose |
|----------|----------|---------|
| `post-deploy` | `ansible/playbooks/post-deploy.yml` | Runs after instance creation |
| `decommission` | `ansible/playbooks/decommission.yml` | Runs before instance deletion |
Both must have `ask_variables_on_launch: true` so Aether can inject extra vars.
```bash
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \
--user "admin:${ADMIN_PW}" \
-H "Content-Type: application/json" \
-d '{
"name": "post-deploy",
"organization": 1,
"project": <PROJECT_ID>,
"playbook": "ansible/playbooks/post-deploy.yml",
"inventory": <INVENTORY_ID>,
"ask_variables_on_launch": true
}'
```
### Personal Access Token (for Aether)
```bash
AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \
--user "admin:${ADMIN_PW}" \
-H "Content-Type: application/json" \
-d '{"description": "Aether integration", "scope": "write"}' \
| python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
```
## Aether integration
### Register AWX endpoint
In the Aether web UI: Settings → Ansible Automation (`/awx-endpoints`) →
Add Endpoint:
| Field | Value |
|-------|-------|
| Name | `lab-awx` |
| URL | `https://192.168.102.161` |
| Token | (PAT from above) |
| Verify SSL | unchecked (self-signed) |
Or via API:
```bash
curl -sSk -b cookies.txt \
-H "X-CSRF-Token: $CSRF" \
-H "Referer: https://192.168.102.160:8443/" \
-X POST https://192.168.102.160:8443/api/awx/endpoints \
-H "Content-Type: application/json" \
-d '{
"name": "lab-awx",
"url": "https://192.168.102.161",
"token": "'$AWX_TOKEN'",
"verify_ssl": false
}'
```
### Bind cluster to AWX
In the Aether UI: AWX endpoint → cluster config, or via API:
```bash
curl -sSk -b cookies.txt \
-H "X-CSRF-Token: $CSRF" \
-H "Referer: https://192.168.102.160:8443/" \
-X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \
-H "Content-Type: application/json" \
-d '{
"cluster_id": 52,
"awx_endpoint_id": <ENDPOINT_ID>,
"awx_post_deploy_template_id": <POST_DEPLOY_TEMPLATE_ID>,
"awx_decommission_template_id": <DECOMMISSION_TEMPLATE_ID>,
"awx_job_timeout_seconds": 600
}'
```
### Verify integration
- Aether Health page (`/health`) shows AWX endpoint as "Healthy"
- Deploy a test instance via Aether — AWX post-deploy job should trigger
- Check AWX job history for the completed run
- Delete the test instance — decommission job should trigger
### Lifecycle behavior
| Hook | Trigger | On failure |
|------|---------|------------|
| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
| Decommission | Before instance deletion | Failure does NOT block deletion |
Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
state (successful, failed, error) or the timeout expires.
## Writing playbooks
### Directory structure
```
ansible/
├── ansible.cfg # Project-level config
├── playbooks/
│ ├── post-deploy.yml # Aether post-deploy hook
│ └── decommission.yml # Aether decommission hook
└── roles/
└── base-config/
├── tasks/main.yml # Base OS configuration
└── templates/ # Jinja2 templates (future)
```
### Pattern: dynamic host from extra vars
Aether passes `vm_ip` — playbooks use `add_host` to target it:
```yaml
- name: Add target host
hosts: localhost
gather_facts: false
tasks:
- ansible.builtin.add_host:
name: "{{ vm_ip }}"
groups: target
ansible_user: root
- name: Configure target
hosts: target
tasks:
- ansible.builtin.wait_for_connection:
delay: 5
timeout: 300
# ... configuration tasks
```
### Pattern: best-effort decommission
Decommission playbooks should never block instance deletion:
```yaml
- name: Attempt cleanup on target
hosts: target
ignore_unreachable: true
tasks:
- name: Check if host is reachable
ansible.builtin.wait_for_connection:
timeout: 15
ignore_errors: true
register: host_check
- name: Cleanup tasks
# ...
when: host_check is succeeded
```
### The base-config role
The included `base-config` role performs minimal post-deploy setup:
1. Set hostname
2. Update apt cache
3. Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades)
4. Set timezone to UTC
5. Enable unattended security upgrades
6. Write deployment metadata to `/etc/deploy-info`
### Extending playbooks
Common additions for production environments:
| Task | Method | Module |
|------|--------|--------|
| DNS registration | API from AWX controller | `community.general.nsupdate` or REST |
| Monitoring agent | SSH to target | `ansible.builtin.package` + config |
| SSH key distribution | SSH to target | `ansible.posix.authorized_key` |
| Certificate provisioning | API from controller | `community.crypto.acme_certificate` |
| Backup enrollment | API from controller | REST calls to backup system |
External system tasks use `delegate_to: localhost` to run from the AWX
controller rather than on the target instance.
## The deploy-awx script
### Usage
```
deploy-awx [OPTIONS]
Actions:
--deploy Full deploy: create VM → K3s → AWX → configure
--status Check AWX health, K3s pods, Aether connection
--heal Restart failed pods, re-sync project, re-validate
--configure (Re-)configure AWX: project, inventory, templates
--join-aether Register AWX with Aether + print binding instructions
--cleanup Destroy the AWX VM
--doctor Check prerequisites
Options:
-c, --config FILE Configuration file (YAML with awx: section)
-n, --dry-run Preview actions without executing
-v, --verbose Show detailed output
-q, --quiet Suppress informational output
```
### Configuration
Defaults are built into the script for the lab environment. Override with
a YAML config file:
```yaml
awx:
vm_name: awx
target_node: oc-node-02
ip: 192.168.102.161/22
gateway: 192.168.100.1
dns: 192.168.100.1
cpu: 4
memory: 8GiB
disk: 40GiB
git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
git_branch: master
aether_url: https://192.168.102.160:8443
aether_cluster_id: 52
```
### Deployment phases
1. **Create VM** — `incus launch images:debian/12` with macvlan NIC
2. **Configure network** — static IP via netplan, hostname
3. **Install K3s** — single-node K3s with kubeconfig mode 644
4. **Deploy AWX Operator** — kustomize from `awx-manifests/operator/`
5. **Deploy AWX instance** — kustomize from `awx-manifests/base/`
6. **Verify** — API ping, admin password retrieval
### Heal logic
The `--heal` command checks and fixes:
1. K3s service running → restart if not
2. All AWX pods healthy → delete unhealthy (operator recreates)
3. AWX web responding → restart deployments if not
4. Project sync current → trigger re-sync
### Manifests
K8s manifests stored in `incusos/awx-manifests/`:
```
awx-manifests/
├── operator/
│ └── kustomization.yaml # AWX Operator 2.19.1 via kustomize
└── base/
├── kustomization.yaml # AWX instance resources
└── awx.yaml # AWX custom resource (pods, storage, ingress)
```
The AWX CR in `awx.yaml` defines resource limits tuned for the 8 GiB VM:
- Web pod: 512 MiB → 1 GiB
- Task pod: 512 MiB → 2 GiB
- EE pod: 256 MiB → 1 GiB
- PostgreSQL: 256 MiB → 512 MiB
## Troubleshooting
### Pod status
```bash
# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide
# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
```
### Common issues
**AWX pods stuck in Pending**: insufficient resources. Check node capacity:
```bash
incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources"
```
**AWX web returns 502**: task or web pod crashed. Restart:
```bash
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task
```
**Project sync fails**: SCM credential or network issue. Check:
```bash
incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm
```
**Job fails with "Host unreachable"**: target instance not booted yet or
SSH not available. Increase the `wait_for_connection` timeout in the playbook
or add a delay in Aether's job timeout setting.
**K3s not starting**: VM resources exhausted. Check:
```bash
incus exec oc-node-02:awx -- systemctl status k3s
incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50
```
**Cannot reach AWX from outside**: macvlan NIC misconfigured. Verify:
```bash
incus exec oc-node-02:awx -- ip addr show enp5s0
incus exec oc-node-02:awx -- ip route
```
### Useful commands
```bash
# AWX version
curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool
# List job templates
curl -sk https://192.168.102.161/api/v2/job_templates/ \
--user "admin:$PW" | python3 -m json.tool
# List recent jobs
curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \
--user "admin:$PW" | python3 -c "
import sys,json
for j in json.load(sys.stdin)['results']:
print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}')"
# Manually trigger post-deploy
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \
--user "admin:$PW" \
-H "Content-Type: application/json" \
-d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}'
```
### Rollback
If AWX is broken beyond repair:
```bash
./incusos/deploy-awx --cleanup
./incusos/deploy-awx --deploy
./incusos/deploy-awx --configure
./incusos/deploy-awx --join-aether
```
The AWX VM is a standalone workload with no cluster dependencies. Destroying
and redeploying has zero impact on the Incus cluster or other workloads.