WIP: Add AWX deployment, Ansible playbooks, and Aether integration
Work-in-progress commit to enable AWX project sync from git. Files will be updated with validated output from real deployment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
fd30ef27d4
commit
6a74925fc4
35
CLAUDE.md
35
CLAUDE.md
|
|
@ -13,11 +13,21 @@ incu-contrib/
|
|||
├── CLAUDE.md # This file -- project context
|
||||
├── README.md # Main overview
|
||||
├── .gitignore
|
||||
├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks
|
||||
│ ├── ansible.cfg # Project-level Ansible config
|
||||
│ ├── playbooks/
|
||||
│ │ ├── post-deploy.yml # Runs after Aether creates an instance
|
||||
│ │ └── decommission.yml # Runs before Aether deletes an instance
|
||||
│ └── roles/
|
||||
│ └── base-config/ # Base OS configuration role
|
||||
│ └── tasks/main.yml
|
||||
├── incusos/ # IncusOS installation tooling
|
||||
│ ├── README.md # Detailed usage docs
|
||||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||
│ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||
│ ├── observe-deploy # Single-VM deploy with rapid console screenshots
|
||||
│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored)
|
||||
|
|
@ -33,6 +43,7 @@ incu-contrib/
|
|||
├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end)
|
||||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
||||
├── aether-guide.md # Aether management platform (deploy, blueprints, API)
|
||||
├── awx-guide.md # AWX + Aether Ansible automation
|
||||
├── incus-version-compatibility.md # Incus versions across platforms
|
||||
├── iso-download-methods.md # ISO download/customization research
|
||||
└── utm-support.md # UTM support design document (future)
|
||||
|
|
@ -809,6 +820,30 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
|||
- No `blockstat` equivalent -- install detection must use timeout + port polling.
|
||||
- Seed generation already works cross-platform (Phase 3 macOS compatibility).
|
||||
|
||||
### AWX integration (Ansible automation for Aether)
|
||||
|
||||
- **AWX** is the open-source Ansible automation platform (upstream of Ansible
|
||||
Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
|
||||
- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
|
||||
to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
|
||||
- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
|
||||
`--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
|
||||
- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
|
||||
- **Ansible playbooks** in `ansible/` directory:
|
||||
- `playbooks/post-deploy.yml` -- runs after Aether creates an instance
|
||||
- `playbooks/decommission.yml` -- runs before Aether deletes an instance
|
||||
- `roles/base-config/` -- hostname, packages, timezone, deploy metadata
|
||||
- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with
|
||||
a Personal Access Token. Per-cluster config binds post-deploy and
|
||||
decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/`
|
||||
with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`).
|
||||
- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
|
||||
deleted). Decommission failure does NOT block deletion.
|
||||
- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to
|
||||
create dynamic inventory → SSH to target for configuration. External system
|
||||
tasks use `delegate_to: localhost`.
|
||||
- See `notes/awx-guide.md` for full deployment guide and troubleshooting.
|
||||
|
||||
## Git workflow
|
||||
|
||||
- Main branch: `main`
|
||||
|
|
|
|||
17
README.md
17
README.md
|
|
@ -23,8 +23,22 @@ Scripts for building IncusOS installation media and seed configurations:
|
|||
YAML, and the script handles seed generation, ISO upload, VM creation with correct
|
||||
settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation.
|
||||
|
||||
- **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus
|
||||
cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template
|
||||
configuration, and Aether integration.
|
||||
|
||||
See [`incusos/README.md`](incusos/README.md) for full documentation.
|
||||
|
||||
### [`ansible/`](ansible/)
|
||||
|
||||
Ansible playbooks and roles for AWX/Aether lifecycle hooks:
|
||||
|
||||
- **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance.
|
||||
Configures hostname, installs base packages, sets timezone, logs deployment.
|
||||
- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
|
||||
Graceful service shutdown and decommission logging (best-effort).
|
||||
- **`roles/base-config/`** -- Base OS configuration role used by post-deploy.
|
||||
|
||||
### [`notes/`](notes/)
|
||||
|
||||
Guides and reference material (roughly in learning order):
|
||||
|
|
@ -44,6 +58,9 @@ Guides and reference material (roughly in learning order):
|
|||
- [`aether-guide.md`](notes/aether-guide.md) --
|
||||
Aether management platform: deployment, instance/blueprint lifecycle, API
|
||||
reference, OVN recovery, and node resizing.
|
||||
- [`awx-guide.md`](notes/awx-guide.md) --
|
||||
AWX deployment, Aether integration, Ansible playbook patterns, and
|
||||
the deploy-awx management script.
|
||||
- [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) --
|
||||
Incus package versions across platforms and install instructions.
|
||||
- [`utm-support.md`](notes/utm-support.md) --
|
||||
|
|
|
|||
|
|
@ -0,0 +1,28 @@
|
|||
[defaults]
|
||||
# Inventory — AWX manages inventory; this is for local testing only
|
||||
inventory = localhost,
|
||||
|
||||
# Roles path
|
||||
roles_path = roles
|
||||
|
||||
# Reduce SSH overhead
|
||||
forks = 10
|
||||
timeout = 30
|
||||
|
||||
# Retry files clutter the repo
|
||||
retry_files_enabled = False
|
||||
|
||||
# Human-readable output
|
||||
stdout_callback = yaml
|
||||
|
||||
# Host key checking — disabled for dynamic lab instances
|
||||
host_key_checking = False
|
||||
|
||||
[privilege_escalation]
|
||||
become = True
|
||||
become_method = sudo
|
||||
|
||||
[ssh_connection]
|
||||
# Speed up SSH connections
|
||||
pipelining = True
|
||||
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
# decommission.yml — Runs before Aether deletes an instance
|
||||
#
|
||||
# Aether passes the same extra vars as post-deploy:
|
||||
# vm_name, vm_ip, environment, owner, cost_center
|
||||
#
|
||||
# This playbook runs best-effort: failures do NOT block instance
|
||||
# deletion in Aether. The playbook should be defensive and handle
|
||||
# unreachable hosts gracefully.
|
||||
|
||||
- name: Decommission — prepare instance for deletion
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
connection: local
|
||||
|
||||
vars:
|
||||
vm_name: "{{ vm_name | default('unknown') }}"
|
||||
vm_ip: "{{ vm_ip | default('') }}"
|
||||
environment_name: "{{ environment | default('lab') }}"
|
||||
owner: "{{ owner | default('unassigned') }}"
|
||||
cost_center: "{{ cost_center | default('') }}"
|
||||
ssh_user: root
|
||||
|
||||
tasks:
|
||||
- name: Add target host to in-memory inventory
|
||||
ansible.builtin.add_host:
|
||||
name: "{{ vm_ip }}"
|
||||
groups: target
|
||||
ansible_user: "{{ ssh_user }}"
|
||||
ansible_host: "{{ vm_ip }}"
|
||||
when: vm_ip | length > 0
|
||||
|
||||
- name: Attempt graceful shutdown on target
|
||||
hosts: target
|
||||
gather_facts: false
|
||||
ignore_unreachable: true
|
||||
|
||||
tasks:
|
||||
- name: Check if host is reachable
|
||||
ansible.builtin.wait_for_connection:
|
||||
delay: 0
|
||||
timeout: 15
|
||||
register: host_check
|
||||
ignore_errors: true
|
||||
|
||||
- name: Stop application services (best-effort)
|
||||
ansible.builtin.shell: |
|
||||
# Stop user-facing services before deletion
|
||||
systemctl list-units --type=service --state=running --no-legend \
|
||||
| awk '{print $1}' \
|
||||
| grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
|
||||
| head -20 \
|
||||
| xargs -r -n1 systemctl stop
|
||||
register: service_stop
|
||||
ignore_errors: true
|
||||
when: host_check is succeeded
|
||||
|
||||
- name: Flush logs to persistent storage
|
||||
ansible.builtin.command: journalctl --flush
|
||||
ignore_errors: true
|
||||
when: host_check is succeeded
|
||||
|
||||
- name: Log decommission
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
connection: local
|
||||
|
||||
tasks:
|
||||
- name: Record decommission in ledger
|
||||
ansible.builtin.lineinfile:
|
||||
path: /var/log/awx-deploy-ledger.log
|
||||
line: >-
|
||||
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
|
||||
DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }}
|
||||
environment={{ environment_name }} owner={{ owner }}
|
||||
cost_center={{ cost_center }} status=completed
|
||||
create: true
|
||||
mode: "0644"
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
---
|
||||
# post-deploy.yml — Runs after Aether creates an instance
|
||||
#
|
||||
# Aether passes extra vars:
|
||||
# vm_name — instance hostname
|
||||
# vm_ip — IP address assigned by Aether
|
||||
# environment — deployment environment (dev, staging, prod)
|
||||
# owner — VM owner or team
|
||||
# cost_center — billing/cost allocation
|
||||
# (plus any custom tags from the blueprint)
|
||||
#
|
||||
# AWX job template must have "ask_variables_on_launch: true" so Aether
|
||||
# can inject these variables at runtime.
|
||||
|
||||
- name: Post-deploy — configure newly provisioned instance
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
connection: local
|
||||
|
||||
vars:
|
||||
# Defaults (overridden by Aether extra vars)
|
||||
vm_name: "{{ vm_name | default('unknown') }}"
|
||||
vm_ip: "{{ vm_ip | default('') }}"
|
||||
environment_name: "{{ environment | default('lab') }}"
|
||||
owner: "{{ owner | default('unassigned') }}"
|
||||
cost_center: "{{ cost_center | default('') }}"
|
||||
ssh_user: root
|
||||
base_packages:
|
||||
- curl
|
||||
- vim
|
||||
- htop
|
||||
- jq
|
||||
- tmux
|
||||
- rsync
|
||||
- unattended-upgrades
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate required variables
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- vm_ip | length > 0
|
||||
- vm_name | length > 0
|
||||
fail_msg: "vm_ip and vm_name must be provided by Aether"
|
||||
|
||||
- name: Add target host to in-memory inventory
|
||||
ansible.builtin.add_host:
|
||||
name: "{{ vm_ip }}"
|
||||
groups: target
|
||||
ansible_user: "{{ ssh_user }}"
|
||||
ansible_host: "{{ vm_ip }}"
|
||||
|
||||
- name: Wait for instance and apply base configuration
|
||||
hosts: target
|
||||
gather_facts: false
|
||||
|
||||
pre_tasks:
|
||||
- name: Wait for SSH connectivity (up to 5 minutes)
|
||||
ansible.builtin.wait_for_connection:
|
||||
delay: 5
|
||||
timeout: 300
|
||||
|
||||
- name: Gather facts after connection
|
||||
ansible.builtin.setup:
|
||||
|
||||
roles:
|
||||
- role: base-config
|
||||
vars:
|
||||
hostname: "{{ hostvars['localhost']['vm_name'] }}"
|
||||
deploy_environment: "{{ hostvars['localhost']['environment_name'] }}"
|
||||
deploy_owner: "{{ hostvars['localhost']['owner'] }}"
|
||||
packages: "{{ hostvars['localhost']['base_packages'] }}"
|
||||
|
||||
- name: Log deployment
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
connection: local
|
||||
|
||||
tasks:
|
||||
- name: Record deployment in ledger
|
||||
ansible.builtin.lineinfile:
|
||||
path: /var/log/awx-deploy-ledger.log
|
||||
line: >-
|
||||
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
|
||||
DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }}
|
||||
environment={{ environment_name }} owner={{ owner }}
|
||||
cost_center={{ cost_center }} status=success
|
||||
create: true
|
||||
mode: "0644"
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
---
|
||||
# base-config — Minimal post-deploy configuration for new instances
|
||||
#
|
||||
# Variables (passed from post-deploy.yml):
|
||||
# hostname — desired hostname
|
||||
# deploy_environment — dev/staging/prod
|
||||
# deploy_owner — owner tag
|
||||
# packages — list of packages to install
|
||||
|
||||
- name: Set hostname
|
||||
ansible.builtin.hostname:
|
||||
name: "{{ hostname }}"
|
||||
|
||||
- name: Add hostname to /etc/hosts
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/hosts
|
||||
regexp: '^127\.0\.1\.1'
|
||||
line: "127.0.1.1 {{ hostname }}"
|
||||
|
||||
- name: Update apt cache
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Install base packages
|
||||
ansible.builtin.apt:
|
||||
name: "{{ packages }}"
|
||||
state: present
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Set timezone to UTC
|
||||
community.general.timezone:
|
||||
name: UTC
|
||||
|
||||
- name: Enable unattended security upgrades
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/apt/apt.conf.d/20auto-upgrades
|
||||
content: |
|
||||
APT::Periodic::Update-Package-Lists "1";
|
||||
APT::Periodic::Unattended-Upgrade "1";
|
||||
mode: "0644"
|
||||
when: ansible_os_family == "Debian"
|
||||
|
||||
- name: Add deployment metadata to /etc/deploy-info
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/deploy-info
|
||||
content: |
|
||||
hostname={{ hostname }}
|
||||
environment={{ deploy_environment }}
|
||||
owner={{ deploy_owner }}
|
||||
deployed={{ ansible_date_time.iso8601 }}
|
||||
mode: "0644"
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
apiVersion: awx.ansible.com/v1beta1
|
||||
kind: AWX
|
||||
metadata:
|
||||
name: awx
|
||||
namespace: awx
|
||||
spec:
|
||||
# NodePort — expose on port 30080 (HTTP) and 30443 (HTTPS)
|
||||
# Simpler than ingress for direct IP access without DNS
|
||||
service_type: nodeport
|
||||
nodeport_port: 30080
|
||||
|
||||
# Resource limits — tuned for 8 GiB VM
|
||||
# Web needs ~1.5 GiB (5 uWSGI workers + daphne + nginx + redis sidecar)
|
||||
# Task needs ~2 GiB (dispatcher + callback receiver + workers)
|
||||
web_resource_requirements:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2Gi
|
||||
task_resource_requirements:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 3Gi
|
||||
ee_resource_requirements:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
|
||||
# PostgreSQL — bundled (operator-managed)
|
||||
postgres_storage_class: local-path
|
||||
postgres_storage_requirements:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
postgres_resource_requirements:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
# Projects — persistent storage for SCM checkouts
|
||||
# access_mode must be ReadWriteOnce for local-path (K3s default provisioner)
|
||||
projects_persistence: true
|
||||
projects_storage_class: local-path
|
||||
projects_storage_size: 4Gi
|
||||
projects_storage_access_mode: ReadWriteOnce
|
||||
|
||||
# Admin account
|
||||
admin_user: admin
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: awx
|
||||
|
||||
resources:
|
||||
- awx.yaml
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- github.com/ansible/awx-operator/config/default?ref=2.19.1
|
||||
|
||||
images:
|
||||
- name: quay.io/ansible/awx-operator
|
||||
newTag: 2.19.1
|
||||
|
||||
namespace: awx
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,624 @@
|
|||
# AWX — Ansible Automation for Aether Lifecycle Hooks
|
||||
|
||||
AWX (the open-source upstream of Ansible Tower) provides a web UI and REST
|
||||
API for running Ansible playbooks. Aether integrates with AWX to run
|
||||
post-deploy and decommission playbooks as lifecycle hooks on every instance
|
||||
create/delete.
|
||||
|
||||
This guide covers deploying AWX on the Incus cluster, connecting it to
|
||||
Aether, writing lifecycle playbooks, and using the `deploy-awx` management
|
||||
script.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User deploys VM via Aether
|
||||
|
|
||||
v
|
||||
Aether creates instance on Incus cluster
|
||||
|
|
||||
v
|
||||
Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
|
||||
with extra_vars: vm_name, vm_ip, environment, owner, cost_center
|
||||
|
|
||||
v
|
||||
AWX runs post-deploy.yml playbook
|
||||
1. SSH to new instance (vm_ip)
|
||||
2. Set hostname, install packages, configure timezone
|
||||
3. Log to deployment ledger
|
||||
|
|
||||
v
|
||||
Instance ready, user notified
|
||||
```
|
||||
|
||||
Decommission works in reverse — Aether triggers the decommission template
|
||||
before deleting the instance.
|
||||
|
||||
### Separation of concerns
|
||||
|
||||
| Layer | Tool | Responsibility |
|
||||
|-------|------|----------------|
|
||||
| Infrastructure | Aether + Incus | VM creation, networking, resources, lifecycle |
|
||||
| Configuration | AWX + Ansible | OS config, packages, services, integrations |
|
||||
| Automation glue | Aether AWX binding | Triggers playbooks at create/delete |
|
||||
|
||||
### What Aether passes to AWX
|
||||
|
||||
Every AWX job launched by Aether receives these extra vars:
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `vm_name` | Instance hostname |
|
||||
| `vm_ip` | IP address assigned by Aether |
|
||||
| `environment` | Deployment environment (dev, staging, prod) |
|
||||
| `owner` | VM owner or team |
|
||||
| `cost_center` | Billing/cost allocation |
|
||||
| (custom tags) | All blueprint tags and instance metadata |
|
||||
|
||||
### AWX API endpoints Aether uses
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|----------|---------|
|
||||
| `GET /api/v2/ping/` | Health check |
|
||||
| `GET /api/v2/job_templates/{id}/` | Validate template exists |
|
||||
| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
|
||||
| `GET /api/v2/jobs/{id}/` | Poll job status |
|
||||
|
||||
## Lab deployment
|
||||
|
||||
### Resource requirements
|
||||
|
||||
| Resource | Recommended | Minimum |
|
||||
|----------|-------------|---------|
|
||||
| vCPU | 4 | 2 |
|
||||
| RAM | 8 GiB | 4 GiB |
|
||||
| Disk | 40 GiB | 20 GiB |
|
||||
|
||||
K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM
|
||||
can run AWX but leaves no headroom for execution environments.
|
||||
|
||||
### Lab details
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| VM name | `awx` |
|
||||
| Location | oc-node-02 |
|
||||
| IP | 192.168.102.161/22 (VLAN 69) |
|
||||
| Gateway | 192.168.100.1 |
|
||||
| DNS | 192.168.100.1 |
|
||||
| Port | 443 (HTTPS via K3s Traefik) |
|
||||
| OS | Debian 12 |
|
||||
| K8s | K3s (single-node) |
|
||||
| AWX Operator | 2.19.1 |
|
||||
|
||||
IP is adjacent to Aether at .160 on the same VLAN.
|
||||
|
||||
### Quick deploy
|
||||
|
||||
```bash
|
||||
# Check prerequisites
|
||||
./incusos/deploy-awx --doctor
|
||||
|
||||
# Full deployment (~10-15 minutes)
|
||||
./incusos/deploy-awx --deploy
|
||||
|
||||
# Configure AWX (project, inventory, templates)
|
||||
./incusos/deploy-awx --configure
|
||||
|
||||
# Register with Aether
|
||||
./incusos/deploy-awx --join-aether
|
||||
```
|
||||
|
||||
### Manual deployment
|
||||
|
||||
#### Step 1: Create VM
|
||||
|
||||
```bash
|
||||
# Launch Debian 12 VM on the cluster
|
||||
incus launch images:debian/12 oc-node-02:awx --vm \
|
||||
--target oc-node-02 \
|
||||
-c limits.cpu=4 -c limits.memory=8GiB \
|
||||
-d root,size=40GiB
|
||||
|
||||
# Switch to macvlan for direct VLAN access
|
||||
incus config device remove oc-node-02:awx eth0
|
||||
incus config device add oc-node-02:awx eth0 nic \
|
||||
nictype=macvlan parent=mgmt
|
||||
```
|
||||
|
||||
#### Step 2: Configure static IP
|
||||
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- bash -c "
|
||||
hostnamectl set-hostname awx
|
||||
cat > /etc/netplan/50-static.yaml << 'NETPLAN'
|
||||
network:
|
||||
version: 2
|
||||
ethernets:
|
||||
enp5s0:
|
||||
addresses: [192.168.102.161/22]
|
||||
routes:
|
||||
- to: default
|
||||
via: 192.168.100.1
|
||||
nameservers:
|
||||
addresses: [192.168.100.1]
|
||||
NETPLAN
|
||||
chmod 600 /etc/netplan/50-static.yaml
|
||||
netplan apply
|
||||
"
|
||||
```
|
||||
|
||||
#### Step 3: Install K3s
|
||||
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- bash -c "
|
||||
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
|
||||
kubectl wait --for=condition=Ready node --all --timeout=120s
|
||||
"
|
||||
```
|
||||
|
||||
#### Step 4: Deploy AWX Operator
|
||||
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- bash -c "
|
||||
kubectl create namespace awx
|
||||
|
||||
mkdir -p /opt/awx/operator
|
||||
cat > /opt/awx/operator/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- github.com/ansible/awx-operator/config/default?ref=2.19.1
|
||||
images:
|
||||
- name: quay.io/ansible/awx-operator
|
||||
newTag: 2.19.1
|
||||
namespace: awx
|
||||
EOF
|
||||
kubectl apply -k /opt/awx/operator/
|
||||
|
||||
# Wait for operator (~2-3 min)
|
||||
kubectl -n awx wait --for=condition=Available \
|
||||
deployment/awx-operator-controller-manager --timeout=300s
|
||||
"
|
||||
```
|
||||
|
||||
#### Step 5: Deploy AWX instance
|
||||
|
||||
The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
|
||||
Push and apply:
|
||||
|
||||
```bash
|
||||
incus file push incusos/awx-manifests/base/awx.yaml \
|
||||
oc-node-02:awx/opt/awx/base/awx.yaml
|
||||
incus file push incusos/awx-manifests/base/kustomization.yaml \
|
||||
oc-node-02:awx/opt/awx/base/kustomization.yaml
|
||||
|
||||
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/
|
||||
|
||||
# Wait 5-10 min for all pods to start
|
||||
incus exec oc-node-02:awx -- kubectl -n awx get pods -w
|
||||
```
|
||||
|
||||
#### Step 6: Verify
|
||||
|
||||
```bash
|
||||
# Get admin password
|
||||
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
|
||||
awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
|
||||
echo "Admin password: $ADMIN_PW"
|
||||
|
||||
# Test API
|
||||
curl -sk https://192.168.102.161/api/v2/ping/
|
||||
```
|
||||
|
||||
Web UI: `https://192.168.102.161/` — login with `admin` / password above.
|
||||
|
||||
## AWX configuration
|
||||
|
||||
All configuration is API-driven so it can be reproduced. The `deploy-awx
|
||||
--configure` command automates these steps.
|
||||
|
||||
### Project (Git repository)
|
||||
|
||||
Create a project pointing to the `ansible/` directory in this repo:
|
||||
|
||||
```bash
|
||||
curl -sk -X POST https://192.168.102.161/api/v2/projects/ \
|
||||
--user "admin:${ADMIN_PW}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "incus-contrib",
|
||||
"organization": 1,
|
||||
"scm_type": "git",
|
||||
"scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git",
|
||||
"scm_branch": "master",
|
||||
"scm_update_on_launch": true
|
||||
}'
|
||||
```
|
||||
|
||||
AWX clones the repo and discovers playbooks under `ansible/playbooks/`.
|
||||
|
||||
### Inventory
|
||||
|
||||
Aether passes `vm_ip` as an extra var. Playbooks use `add_host` to
|
||||
dynamically create an in-memory inventory entry for the target instance.
|
||||
The AWX inventory is a placeholder:
|
||||
|
||||
```bash
|
||||
curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \
|
||||
--user "admin:${ADMIN_PW}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "incus-instances", "organization": 1}'
|
||||
```
|
||||
|
||||
### Credentials
|
||||
|
||||
Two credentials:
|
||||
|
||||
1. **SCM credential** (type 2) — SSH key for cloning the private Git repo
|
||||
2. **Machine credential** (type 1) — SSH key for connecting to managed instances
|
||||
|
||||
```bash
|
||||
# Machine credential
|
||||
curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \
|
||||
--user "admin:${ADMIN_PW}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "incus-instances",
|
||||
"organization": 1,
|
||||
"credential_type": 1,
|
||||
"inputs": {"username": "root", "ssh_key_data": "..."}
|
||||
}'
|
||||
```
|
||||
|
||||
### Job templates
|
||||
|
||||
Two templates matching Aether's lifecycle hooks:
|
||||
|
||||
| Template | Playbook | Purpose |
|
||||
|----------|----------|---------|
|
||||
| `post-deploy` | `ansible/playbooks/post-deploy.yml` | Runs after instance creation |
|
||||
| `decommission` | `ansible/playbooks/decommission.yml` | Runs before instance deletion |
|
||||
|
||||
Both must have `ask_variables_on_launch: true` so Aether can inject extra vars.
|
||||
|
||||
```bash
|
||||
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \
|
||||
--user "admin:${ADMIN_PW}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "post-deploy",
|
||||
"organization": 1,
|
||||
"project": <PROJECT_ID>,
|
||||
"playbook": "ansible/playbooks/post-deploy.yml",
|
||||
"inventory": <INVENTORY_ID>,
|
||||
"ask_variables_on_launch": true
|
||||
}'
|
||||
```
|
||||
|
||||
### Personal Access Token (for Aether)
|
||||
|
||||
```bash
|
||||
AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \
|
||||
--user "admin:${ADMIN_PW}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"description": "Aether integration", "scope": "write"}' \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
|
||||
```
|
||||
|
||||
## Aether integration
|
||||
|
||||
### Register AWX endpoint
|
||||
|
||||
In the Aether web UI: Settings → Ansible Automation (`/awx-endpoints`) →
|
||||
Add Endpoint:
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Name | `lab-awx` |
|
||||
| URL | `https://192.168.102.161` |
|
||||
| Token | (PAT from above) |
|
||||
| Verify SSL | unchecked (self-signed) |
|
||||
|
||||
Or via API:
|
||||
```bash
|
||||
curl -sSk -b cookies.txt \
|
||||
-H "X-CSRF-Token: $CSRF" \
|
||||
-H "Referer: https://192.168.102.160:8443/" \
|
||||
-X POST https://192.168.102.160:8443/api/awx/endpoints \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "lab-awx",
|
||||
"url": "https://192.168.102.161",
|
||||
"token": "'$AWX_TOKEN'",
|
||||
"verify_ssl": false
|
||||
}'
|
||||
```
|
||||
|
||||
### Bind cluster to AWX
|
||||
|
||||
In the Aether UI: AWX endpoint → cluster config, or via API:
|
||||
|
||||
```bash
|
||||
curl -sSk -b cookies.txt \
|
||||
-H "X-CSRF-Token: $CSRF" \
|
||||
-H "Referer: https://192.168.102.160:8443/" \
|
||||
-X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"cluster_id": 52,
|
||||
"awx_endpoint_id": <ENDPOINT_ID>,
|
||||
"awx_post_deploy_template_id": <POST_DEPLOY_TEMPLATE_ID>,
|
||||
"awx_decommission_template_id": <DECOMMISSION_TEMPLATE_ID>,
|
||||
"awx_job_timeout_seconds": 600
|
||||
}'
|
||||
```
|
||||
|
||||
### Verify integration
|
||||
|
||||
- Aether Health page (`/health`) shows AWX endpoint as "Healthy"
|
||||
- Deploy a test instance via Aether — AWX post-deploy job should trigger
|
||||
- Check AWX job history for the completed run
|
||||
- Delete the test instance — decommission job should trigger
|
||||
|
||||
### Lifecycle behavior
|
||||
|
||||
| Hook | Trigger | On failure |
|
||||
|------|---------|------------|
|
||||
| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
|
||||
| Decommission | Before instance deletion | Failure does NOT block deletion |
|
||||
|
||||
Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
|
||||
state (successful, failed, error) or the timeout expires.
|
||||
|
||||
## Writing playbooks
|
||||
|
||||
### Directory structure
|
||||
|
||||
```
|
||||
ansible/
|
||||
├── ansible.cfg # Project-level config
|
||||
├── playbooks/
|
||||
│ ├── post-deploy.yml # Aether post-deploy hook
|
||||
│ └── decommission.yml # Aether decommission hook
|
||||
└── roles/
|
||||
└── base-config/
|
||||
├── tasks/main.yml # Base OS configuration
|
||||
└── templates/ # Jinja2 templates (future)
|
||||
```
|
||||
|
||||
### Pattern: dynamic host from extra vars
|
||||
|
||||
Aether passes `vm_ip` — playbooks use `add_host` to target it:
|
||||
|
||||
```yaml
|
||||
- name: Add target host
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- ansible.builtin.add_host:
|
||||
name: "{{ vm_ip }}"
|
||||
groups: target
|
||||
ansible_user: root
|
||||
|
||||
- name: Configure target
|
||||
hosts: target
|
||||
tasks:
|
||||
- ansible.builtin.wait_for_connection:
|
||||
delay: 5
|
||||
timeout: 300
|
||||
# ... configuration tasks
|
||||
```
|
||||
|
||||
### Pattern: best-effort decommission
|
||||
|
||||
Decommission playbooks should never block instance deletion:
|
||||
|
||||
```yaml
|
||||
- name: Attempt cleanup on target
|
||||
hosts: target
|
||||
ignore_unreachable: true
|
||||
tasks:
|
||||
- name: Check if host is reachable
|
||||
ansible.builtin.wait_for_connection:
|
||||
timeout: 15
|
||||
ignore_errors: true
|
||||
register: host_check
|
||||
|
||||
- name: Cleanup tasks
|
||||
# ...
|
||||
when: host_check is succeeded
|
||||
```
|
||||
|
||||
### The base-config role
|
||||
|
||||
The included `base-config` role performs minimal post-deploy setup:
|
||||
|
||||
1. Set hostname
|
||||
2. Update apt cache
|
||||
3. Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades)
|
||||
4. Set timezone to UTC
|
||||
5. Enable unattended security upgrades
|
||||
6. Write deployment metadata to `/etc/deploy-info`
|
||||
|
||||
### Extending playbooks
|
||||
|
||||
Common additions for production environments:
|
||||
|
||||
| Task | Method | Module |
|
||||
|------|--------|--------|
|
||||
| DNS registration | API from AWX controller | `community.general.nsupdate` or REST |
|
||||
| Monitoring agent | SSH to target | `ansible.builtin.package` + config |
|
||||
| SSH key distribution | SSH to target | `ansible.posix.authorized_key` |
|
||||
| Certificate provisioning | API from controller | `community.crypto.acme_certificate` |
|
||||
| Backup enrollment | API from controller | REST calls to backup system |
|
||||
|
||||
External system tasks use `delegate_to: localhost` to run from the AWX
|
||||
controller rather than on the target instance.
|
||||
|
||||
## The deploy-awx script
|
||||
|
||||
### Usage
|
||||
|
||||
```
|
||||
deploy-awx [OPTIONS]
|
||||
|
||||
Actions:
|
||||
--deploy Full deploy: create VM → K3s → AWX → configure
|
||||
--status Check AWX health, K3s pods, Aether connection
|
||||
--heal Restart failed pods, re-sync project, re-validate
|
||||
--configure (Re-)configure AWX: project, inventory, templates
|
||||
--join-aether Register AWX with Aether + print binding instructions
|
||||
--cleanup Destroy the AWX VM
|
||||
--doctor Check prerequisites
|
||||
|
||||
Options:
|
||||
-c, --config FILE Configuration file (YAML with awx: section)
|
||||
-n, --dry-run Preview actions without executing
|
||||
-v, --verbose Show detailed output
|
||||
-q, --quiet Suppress informational output
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Defaults are built into the script for the lab environment. Override with
|
||||
a YAML config file:
|
||||
|
||||
```yaml
|
||||
awx:
|
||||
vm_name: awx
|
||||
target_node: oc-node-02
|
||||
ip: 192.168.102.161/22
|
||||
gateway: 192.168.100.1
|
||||
dns: 192.168.100.1
|
||||
cpu: 4
|
||||
memory: 8GiB
|
||||
disk: 40GiB
|
||||
git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
|
||||
git_branch: master
|
||||
aether_url: https://192.168.102.160:8443
|
||||
aether_cluster_id: 52
|
||||
```
|
||||
|
||||
### Deployment phases
|
||||
|
||||
1. **Create VM** — `incus launch images:debian/12` with macvlan NIC
|
||||
2. **Configure network** — static IP via netplan, hostname
|
||||
3. **Install K3s** — single-node K3s with kubeconfig mode 644
|
||||
4. **Deploy AWX Operator** — kustomize from `awx-manifests/operator/`
|
||||
5. **Deploy AWX instance** — kustomize from `awx-manifests/base/`
|
||||
6. **Verify** — API ping, admin password retrieval
|
||||
|
||||
### Heal logic
|
||||
|
||||
The `--heal` command checks and fixes:
|
||||
|
||||
1. K3s service running → restart if not
|
||||
2. All AWX pods healthy → delete unhealthy (operator recreates)
|
||||
3. AWX web responding → restart deployments if not
|
||||
4. Project sync current → trigger re-sync
|
||||
|
||||
### Manifests
|
||||
|
||||
K8s manifests stored in `incusos/awx-manifests/`:
|
||||
|
||||
```
|
||||
awx-manifests/
|
||||
├── operator/
|
||||
│ └── kustomization.yaml # AWX Operator 2.19.1 via kustomize
|
||||
└── base/
|
||||
├── kustomization.yaml # AWX instance resources
|
||||
└── awx.yaml # AWX custom resource (pods, storage, ingress)
|
||||
```
|
||||
|
||||
The AWX CR in `awx.yaml` defines resource limits tuned for the 8 GiB VM:
|
||||
- Web pod: 512 MiB → 1 GiB
|
||||
- Task pod: 512 MiB → 2 GiB
|
||||
- EE pod: 256 MiB → 1 GiB
|
||||
- PostgreSQL: 256 MiB → 512 MiB
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pod status
|
||||
|
||||
```bash
|
||||
# List all AWX pods
|
||||
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide
|
||||
|
||||
# Describe a failing pod
|
||||
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
|
||||
|
||||
# View pod logs
|
||||
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
|
||||
```
|
||||
|
||||
### Common issues
|
||||
|
||||
**AWX pods stuck in Pending**: insufficient resources. Check node capacity:
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources"
|
||||
```
|
||||
|
||||
**AWX web returns 502**: task or web pod crashed. Restart:
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web
|
||||
incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task
|
||||
```
|
||||
|
||||
**Project sync fails**: SCM credential or network issue. Check:
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm
|
||||
```
|
||||
|
||||
**Job fails with "Host unreachable"**: target instance not booted yet or
|
||||
SSH not available. Increase the `wait_for_connection` timeout in the playbook
|
||||
or add a delay in Aether's job timeout setting.
|
||||
|
||||
**K3s not starting**: VM resources exhausted. Check:
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- systemctl status k3s
|
||||
incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50
|
||||
```
|
||||
|
||||
**Cannot reach AWX from outside**: macvlan NIC misconfigured. Verify:
|
||||
```bash
|
||||
incus exec oc-node-02:awx -- ip addr show enp5s0
|
||||
incus exec oc-node-02:awx -- ip route
|
||||
```
|
||||
|
||||
### Useful commands
|
||||
|
||||
```bash
|
||||
# AWX version
|
||||
curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool
|
||||
|
||||
# List job templates
|
||||
curl -sk https://192.168.102.161/api/v2/job_templates/ \
|
||||
--user "admin:$PW" | python3 -m json.tool
|
||||
|
||||
# List recent jobs
|
||||
curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \
|
||||
--user "admin:$PW" | python3 -c "
|
||||
import sys,json
|
||||
for j in json.load(sys.stdin)['results']:
|
||||
print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}')"
|
||||
|
||||
# Manually trigger post-deploy
|
||||
curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \
|
||||
--user "admin:$PW" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}'
|
||||
```
|
||||
|
||||
### Rollback
|
||||
|
||||
If AWX is broken beyond repair:
|
||||
```bash
|
||||
./incusos/deploy-awx --cleanup
|
||||
./incusos/deploy-awx --deploy
|
||||
./incusos/deploy-awx --configure
|
||||
./incusos/deploy-awx --join-aether
|
||||
```
|
||||
|
||||
The AWX VM is a standalone workload with no cluster dependencies. Destroying
|
||||
and redeploying has zero impact on the Incus cluster or other workloads.
|
||||
Loading…
Reference in New Issue