From 6a74925fc46166408070c036d0e560dad8983faa Mon Sep 17 00:00:00 2001 From: Maarten Date: Tue, 24 Feb 2026 00:04:28 +0100 Subject: [PATCH] WIP: Add AWX deployment, Ansible playbooks, and Aether integration Work-in-progress commit to enable AWX project sync from git. Files will be updated with validated output from real deployment. Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 35 + README.md | 17 + ansible/ansible.cfg | 28 + ansible/playbooks/decommission.yml | 78 ++ ansible/playbooks/post-deploy.yml | 88 ++ ansible/roles/base-config/tasks/main.yml | 53 + incusos/awx-manifests/base/awx.yaml | 59 + incusos/awx-manifests/base/kustomization.yaml | 8 + .../awx-manifests/operator/kustomization.yaml | 12 + incusos/deploy-awx | 1137 +++++++++++++++++ notes/awx-guide.md | 624 +++++++++ 11 files changed, 2139 insertions(+) create mode 100644 ansible/ansible.cfg create mode 100644 ansible/playbooks/decommission.yml create mode 100644 ansible/playbooks/post-deploy.yml create mode 100644 ansible/roles/base-config/tasks/main.yml create mode 100644 incusos/awx-manifests/base/awx.yaml create mode 100644 incusos/awx-manifests/base/kustomization.yaml create mode 100644 incusos/awx-manifests/operator/kustomization.yaml create mode 100755 incusos/deploy-awx create mode 100644 notes/awx-guide.md diff --git a/CLAUDE.md b/CLAUDE.md index 6ee7b85..67035b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,11 +13,21 @@ incu-contrib/ ├── CLAUDE.md # This file -- project context ├── README.md # Main overview ├── .gitignore +├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks +│ ├── ansible.cfg # Project-level Ansible config +│ ├── playbooks/ +│ │ ├── post-deploy.yml # Runs after Aether creates an instance +│ │ └── decommission.yml # Runs before Aether deletes an instance +│ └── roles/ +│ └── base-config/ # Base OS configuration role +│ └── tasks/main.yml ├── incusos/ # IncusOS installation tooling │ ├── README.md # Detailed usage docs │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle +│ ├── deploy-awx # AWX deployment + management on Incus cluster +│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with rapid console screenshots │ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) @@ -33,6 +43,7 @@ incu-contrib/ ├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end) ├── migration-guide.md # Migration paths into Incus from other hypervisors ├── aether-guide.md # Aether management platform (deploy, blueprints, API) + ├── awx-guide.md # AWX + Aether Ansible automation ├── incus-version-compatibility.md # Incus versions across platforms ├── iso-download-methods.md # ISO download/customization research └── utm-support.md # UTM support design document (future) @@ -809,6 +820,30 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ - No `blockstat` equivalent -- install detection must use timeout + port polling. - Seed generation already works cross-platform (Phase 3 macOS compatibility). +### AWX integration (Ansible automation for Aether) + +- **AWX** is the open-source Ansible automation platform (upstream of Ansible + Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster. +- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent + to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk. +- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`, + `--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`. +- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize). +- **Ansible playbooks** in `ansible/` directory: + - `playbooks/post-deploy.yml` -- runs after Aether creates an instance + - `playbooks/decommission.yml` -- runs before Aether deletes an instance + - `roles/base-config/` -- hostname, packages, timezone, deploy metadata +- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with + a Personal Access Token. Per-cluster config binds post-deploy and + decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/` + with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`). +- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance + deleted). Decommission failure does NOT block deletion. +- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to + create dynamic inventory → SSH to target for configuration. External system + tasks use `delegate_to: localhost`. +- See `notes/awx-guide.md` for full deployment guide and troubleshooting. + ## Git workflow - Main branch: `main` diff --git a/README.md b/README.md index 22100e7..389a22f 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,22 @@ Scripts for building IncusOS installation media and seed configurations: YAML, and the script handles seed generation, ISO upload, VM creation with correct settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation. +- **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus + cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template + configuration, and Aether integration. + See [`incusos/README.md`](incusos/README.md) for full documentation. +### [`ansible/`](ansible/) + +Ansible playbooks and roles for AWX/Aether lifecycle hooks: + +- **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance. + Configures hostname, installs base packages, sets timezone, logs deployment. +- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance. + Graceful service shutdown and decommission logging (best-effort). +- **`roles/base-config/`** -- Base OS configuration role used by post-deploy. + ### [`notes/`](notes/) Guides and reference material (roughly in learning order): @@ -44,6 +58,9 @@ Guides and reference material (roughly in learning order): - [`aether-guide.md`](notes/aether-guide.md) -- Aether management platform: deployment, instance/blueprint lifecycle, API reference, OVN recovery, and node resizing. +- [`awx-guide.md`](notes/awx-guide.md) -- + AWX deployment, Aether integration, Ansible playbook patterns, and + the deploy-awx management script. - [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) -- Incus package versions across platforms and install instructions. - [`utm-support.md`](notes/utm-support.md) -- diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg new file mode 100644 index 0000000..1a62528 --- /dev/null +++ b/ansible/ansible.cfg @@ -0,0 +1,28 @@ +[defaults] +# Inventory — AWX manages inventory; this is for local testing only +inventory = localhost, + +# Roles path +roles_path = roles + +# Reduce SSH overhead +forks = 10 +timeout = 30 + +# Retry files clutter the repo +retry_files_enabled = False + +# Human-readable output +stdout_callback = yaml + +# Host key checking — disabled for dynamic lab instances +host_key_checking = False + +[privilege_escalation] +become = True +become_method = sudo + +[ssh_connection] +# Speed up SSH connections +pipelining = True +ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no diff --git a/ansible/playbooks/decommission.yml b/ansible/playbooks/decommission.yml new file mode 100644 index 0000000..91f0813 --- /dev/null +++ b/ansible/playbooks/decommission.yml @@ -0,0 +1,78 @@ +--- +# decommission.yml — Runs before Aether deletes an instance +# +# Aether passes the same extra vars as post-deploy: +# vm_name, vm_ip, environment, owner, cost_center +# +# This playbook runs best-effort: failures do NOT block instance +# deletion in Aether. The playbook should be defensive and handle +# unreachable hosts gracefully. + +- name: Decommission — prepare instance for deletion + hosts: localhost + gather_facts: false + connection: local + + vars: + vm_name: "{{ vm_name | default('unknown') }}" + vm_ip: "{{ vm_ip | default('') }}" + environment_name: "{{ environment | default('lab') }}" + owner: "{{ owner | default('unassigned') }}" + cost_center: "{{ cost_center | default('') }}" + ssh_user: root + + tasks: + - name: Add target host to in-memory inventory + ansible.builtin.add_host: + name: "{{ vm_ip }}" + groups: target + ansible_user: "{{ ssh_user }}" + ansible_host: "{{ vm_ip }}" + when: vm_ip | length > 0 + +- name: Attempt graceful shutdown on target + hosts: target + gather_facts: false + ignore_unreachable: true + + tasks: + - name: Check if host is reachable + ansible.builtin.wait_for_connection: + delay: 0 + timeout: 15 + register: host_check + ignore_errors: true + + - name: Stop application services (best-effort) + ansible.builtin.shell: | + # Stop user-facing services before deletion + systemctl list-units --type=service --state=running --no-legend \ + | awk '{print $1}' \ + | grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \ + | head -20 \ + | xargs -r -n1 systemctl stop + register: service_stop + ignore_errors: true + when: host_check is succeeded + + - name: Flush logs to persistent storage + ansible.builtin.command: journalctl --flush + ignore_errors: true + when: host_check is succeeded + +- name: Log decommission + hosts: localhost + gather_facts: false + connection: local + + tasks: + - name: Record decommission in ledger + ansible.builtin.lineinfile: + path: /var/log/awx-deploy-ledger.log + line: >- + {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }} + DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }} + environment={{ environment_name }} owner={{ owner }} + cost_center={{ cost_center }} status=completed + create: true + mode: "0644" diff --git a/ansible/playbooks/post-deploy.yml b/ansible/playbooks/post-deploy.yml new file mode 100644 index 0000000..bcbe505 --- /dev/null +++ b/ansible/playbooks/post-deploy.yml @@ -0,0 +1,88 @@ +--- +# post-deploy.yml — Runs after Aether creates an instance +# +# Aether passes extra vars: +# vm_name — instance hostname +# vm_ip — IP address assigned by Aether +# environment — deployment environment (dev, staging, prod) +# owner — VM owner or team +# cost_center — billing/cost allocation +# (plus any custom tags from the blueprint) +# +# AWX job template must have "ask_variables_on_launch: true" so Aether +# can inject these variables at runtime. + +- name: Post-deploy — configure newly provisioned instance + hosts: localhost + gather_facts: false + connection: local + + vars: + # Defaults (overridden by Aether extra vars) + vm_name: "{{ vm_name | default('unknown') }}" + vm_ip: "{{ vm_ip | default('') }}" + environment_name: "{{ environment | default('lab') }}" + owner: "{{ owner | default('unassigned') }}" + cost_center: "{{ cost_center | default('') }}" + ssh_user: root + base_packages: + - curl + - vim + - htop + - jq + - tmux + - rsync + - unattended-upgrades + + pre_tasks: + - name: Validate required variables + ansible.builtin.assert: + that: + - vm_ip | length > 0 + - vm_name | length > 0 + fail_msg: "vm_ip and vm_name must be provided by Aether" + + - name: Add target host to in-memory inventory + ansible.builtin.add_host: + name: "{{ vm_ip }}" + groups: target + ansible_user: "{{ ssh_user }}" + ansible_host: "{{ vm_ip }}" + +- name: Wait for instance and apply base configuration + hosts: target + gather_facts: false + + pre_tasks: + - name: Wait for SSH connectivity (up to 5 minutes) + ansible.builtin.wait_for_connection: + delay: 5 + timeout: 300 + + - name: Gather facts after connection + ansible.builtin.setup: + + roles: + - role: base-config + vars: + hostname: "{{ hostvars['localhost']['vm_name'] }}" + deploy_environment: "{{ hostvars['localhost']['environment_name'] }}" + deploy_owner: "{{ hostvars['localhost']['owner'] }}" + packages: "{{ hostvars['localhost']['base_packages'] }}" + +- name: Log deployment + hosts: localhost + gather_facts: false + connection: local + + tasks: + - name: Record deployment in ledger + ansible.builtin.lineinfile: + path: /var/log/awx-deploy-ledger.log + line: >- + {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }} + DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }} + environment={{ environment_name }} owner={{ owner }} + cost_center={{ cost_center }} status=success + create: true + mode: "0644" diff --git a/ansible/roles/base-config/tasks/main.yml b/ansible/roles/base-config/tasks/main.yml new file mode 100644 index 0000000..1519eed --- /dev/null +++ b/ansible/roles/base-config/tasks/main.yml @@ -0,0 +1,53 @@ +--- +# base-config — Minimal post-deploy configuration for new instances +# +# Variables (passed from post-deploy.yml): +# hostname — desired hostname +# deploy_environment — dev/staging/prod +# deploy_owner — owner tag +# packages — list of packages to install + +- name: Set hostname + ansible.builtin.hostname: + name: "{{ hostname }}" + +- name: Add hostname to /etc/hosts + ansible.builtin.lineinfile: + path: /etc/hosts + regexp: '^127\.0\.1\.1' + line: "127.0.1.1 {{ hostname }}" + +- name: Update apt cache + ansible.builtin.apt: + update_cache: true + cache_valid_time: 3600 + when: ansible_os_family == "Debian" + +- name: Install base packages + ansible.builtin.apt: + name: "{{ packages }}" + state: present + when: ansible_os_family == "Debian" + +- name: Set timezone to UTC + community.general.timezone: + name: UTC + +- name: Enable unattended security upgrades + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + content: | + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Unattended-Upgrade "1"; + mode: "0644" + when: ansible_os_family == "Debian" + +- name: Add deployment metadata to /etc/deploy-info + ansible.builtin.copy: + dest: /etc/deploy-info + content: | + hostname={{ hostname }} + environment={{ deploy_environment }} + owner={{ deploy_owner }} + deployed={{ ansible_date_time.iso8601 }} + mode: "0644" diff --git a/incusos/awx-manifests/base/awx.yaml b/incusos/awx-manifests/base/awx.yaml new file mode 100644 index 0000000..c1f8d81 --- /dev/null +++ b/incusos/awx-manifests/base/awx.yaml @@ -0,0 +1,59 @@ +--- +apiVersion: awx.ansible.com/v1beta1 +kind: AWX +metadata: + name: awx + namespace: awx +spec: + # NodePort — expose on port 30080 (HTTP) and 30443 (HTTPS) + # Simpler than ingress for direct IP access without DNS + service_type: nodeport + nodeport_port: 30080 + + # Resource limits — tuned for 8 GiB VM + # Web needs ~1.5 GiB (5 uWSGI workers + daphne + nginx + redis sidecar) + # Task needs ~2 GiB (dispatcher + callback receiver + workers) + web_resource_requirements: + requests: + cpu: 200m + memory: 1Gi + limits: + cpu: "2" + memory: 2Gi + task_resource_requirements: + requests: + cpu: 200m + memory: 1Gi + limits: + cpu: "2" + memory: 3Gi + ee_resource_requirements: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + + # PostgreSQL — bundled (operator-managed) + postgres_storage_class: local-path + postgres_storage_requirements: + requests: + storage: 8Gi + postgres_resource_requirements: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi + + # Projects — persistent storage for SCM checkouts + # access_mode must be ReadWriteOnce for local-path (K3s default provisioner) + projects_persistence: true + projects_storage_class: local-path + projects_storage_size: 4Gi + projects_storage_access_mode: ReadWriteOnce + + # Admin account + admin_user: admin diff --git a/incusos/awx-manifests/base/kustomization.yaml b/incusos/awx-manifests/base/kustomization.yaml new file mode 100644 index 0000000..cc3e385 --- /dev/null +++ b/incusos/awx-manifests/base/kustomization.yaml @@ -0,0 +1,8 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: awx + +resources: + - awx.yaml diff --git a/incusos/awx-manifests/operator/kustomization.yaml b/incusos/awx-manifests/operator/kustomization.yaml new file mode 100644 index 0000000..9a3a010 --- /dev/null +++ b/incusos/awx-manifests/operator/kustomization.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - github.com/ansible/awx-operator/config/default?ref=2.19.1 + +images: + - name: quay.io/ansible/awx-operator + newTag: 2.19.1 + +namespace: awx diff --git a/incusos/deploy-awx b/incusos/deploy-awx new file mode 100755 index 0000000..510397f --- /dev/null +++ b/incusos/deploy-awx @@ -0,0 +1,1137 @@ +#!/usr/bin/env bash +# deploy-awx - Deploy and manage AWX on an Incus cluster +# +# Deploys a Debian 12 VM with K3s + AWX Operator, configures AWX for +# Aether integration (project, inventory, job templates), and provides +# ongoing health checks and healing. +# +# Usage: deploy-awx [OPTIONS] +# Run 'deploy-awx --help' for full usage information. + +set -euo pipefail + +readonly VERSION="0.1.0" +readonly SCRIPT_NAME="deploy-awx" +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly MANIFESTS_DIR="${SCRIPT_DIR}/awx-manifests" +readonly AWX_NAMESPACE="awx" + +# --------------------------------------------------------------------------- +# Defaults (overridden by config file or CLI flags) +# --------------------------------------------------------------------------- + +VM_NAME="awx" +TARGET_REMOTE="oc-node-02" +AWX_IP="192.168.102.161" +AWX_PREFIX="22" +AWX_GATEWAY="192.168.100.1" +AWX_DNS="192.168.100.1" +AWX_CPU="4" +AWX_MEMORY="8GiB" +AWX_DISK="40GiB" +GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git" +GIT_BRANCH="master" +PLAYBOOK_DIR="ansible/playbooks" +AETHER_URL="https://192.168.102.160:8443" +AETHER_CLUSTER_ID="52" + +# Runtime flags +DRY_RUN=false +VERBOSE=false +QUIET=false +CONFIG_FILE="" +ACTION="" + +# --------------------------------------------------------------------------- +# Color and formatting +# --------------------------------------------------------------------------- + +setup_colors() { + if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then + RED='\033[0;31m' + GREEN='\033[0;32m' + YELLOW='\033[0;33m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + BOLD='\033[1m' + DIM='\033[2m' + RESET='\033[0m' + else + RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET='' + fi +} + +info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; } +success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; } +warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } +error() { echo -e "${RED}[error]${RESET} $*" >&2; } +step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } + +dry_run_guard() { + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) $*" + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +usage() { + cat </dev/null 2>&1 +} + +vm_running() { + local state + state=$(incus info "$(vm_ref)" 2>/dev/null | grep -i '^status:' | awk '{print $2}') + [[ "$state" == "RUNNING" || "$state" == "Running" ]] +} + +vm_exec() { + incus exec "$(vm_ref)" -- "$@" +} + +awx_api() { + local method="$1" path="$2" + shift 2 + curl -sk -X "$method" "https://${AWX_IP}${path}" \ + -H "Content-Type: application/json" \ + "$@" +} + +awx_api_auth() { + local method="$1" path="$2" + shift 2 + local token + token=$(get_awx_admin_password) + curl -sk -X "$method" "https://${AWX_IP}${path}" \ + --user "admin:${token}" \ + -H "Content-Type: application/json" \ + "$@" +} + +get_awx_admin_password() { + vm_exec kubectl -n "$AWX_NAMESPACE" get secret awx-admin-password \ + -o jsonpath='{.data.password}' 2>/dev/null | base64 -d +} + +wait_for_vm_agent() { + local timeout="${1:-120}" + local elapsed=0 + info "Waiting for VM agent (up to ${timeout}s)..." + while ! vm_exec true &>/dev/null 2>&1; do + sleep 5 + elapsed=$((elapsed + 5)) + if [[ $elapsed -ge $timeout ]]; then + error "VM agent not available after ${timeout}s" + return 1 + fi + done + success "VM agent connected (${elapsed}s)" +} + +wait_for_awx_web() { + local timeout="${1:-600}" + local elapsed=0 + info "Waiting for AWX web interface (up to ${timeout}s)..." + while ! curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | grep -q '200'; do + sleep 10 + elapsed=$((elapsed + 10)) + if [[ $elapsed -ge $timeout ]]; then + error "AWX web not responding after ${timeout}s" + return 1 + fi + [[ "$VERBOSE" == true ]] && detail "Still waiting... (${elapsed}s)" + done + success "AWX web interface ready (${elapsed}s)" +} + +# --------------------------------------------------------------------------- +# Doctor — prerequisite checks +# --------------------------------------------------------------------------- + +action_doctor() { + step "Checking prerequisites" + local ok=true + + # incus CLI + if command -v incus &>/dev/null; then + success "incus CLI: $(incus version 2>/dev/null || echo 'installed')" + else + error "incus CLI not found — install the Incus client" + ok=false + fi + + # curl + if command -v curl &>/dev/null; then + success "curl: $(curl --version | head -1 | awk '{print $2}')" + else + error "curl not found" + ok=false + fi + + # python3 + if command -v python3 &>/dev/null; then + success "python3: $(python3 --version 2>&1 | awk '{print $2}')" + else + error "python3 not found" + ok=false + fi + + # base64 + if command -v base64 &>/dev/null; then + success "base64: available" + else + error "base64 not found" + ok=false + fi + + # Manifest files + if [[ -d "$MANIFESTS_DIR" ]]; then + success "AWX manifests: $MANIFESTS_DIR" + else + error "AWX manifests not found at $MANIFESTS_DIR" + ok=false + fi + + # Target remote + if incus remote list -f csv 2>/dev/null | grep -q "^${TARGET_REMOTE},"; then + success "Incus remote '${TARGET_REMOTE}': reachable" + else + warn "Incus remote '${TARGET_REMOTE}' not found (needed for deployment)" + fi + + # Existing VM + if vm_exists; then + info "AWX VM '$(vm_ref)' exists" + if vm_running; then + success " VM is running" + # Check K3s + if vm_exec kubectl get nodes &>/dev/null 2>&1; then + success " K3s is responding" + else + warn " K3s not responding" + fi + # Check AWX pods + local pod_count + pod_count=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods --no-headers 2>/dev/null | wc -l || echo 0) + info " AWX pods: ${pod_count}" + else + warn " VM is not running" + fi + else + info "AWX VM '$(vm_ref)' does not exist (--deploy to create)" + fi + + echo + if [[ "$ok" == true ]]; then + success "All prerequisites satisfied" + else + error "Some prerequisites missing — fix the issues above" + return 1 + fi +} + +# --------------------------------------------------------------------------- +# Deploy — full deployment pipeline +# --------------------------------------------------------------------------- + +action_deploy() { + step "Deploying AWX on $(vm_ref)" + echo + info " Target: ${TARGET_REMOTE}" + info " VM: ${VM_NAME}" + info " IP: ${AWX_IP}/${AWX_PREFIX}" + info " CPU: ${AWX_CPU}" + info " Memory: ${AWX_MEMORY}" + info " Disk: ${AWX_DISK}" + echo + + if vm_exists; then + if vm_running; then + error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first." + return 1 + else + warn "AWX VM exists but is not running — starting it" + if dry_run_guard "incus start $(vm_ref)"; then + incus start "$(vm_ref)" + fi + return 0 + fi + fi + + phase_create_vm + phase_configure_network + phase_install_k3s + phase_deploy_awx_operator + phase_deploy_awx_instance + phase_verify + + echo + success "AWX deployment complete!" + info " Web UI: https://${AWX_IP}/" + info " API: https://${AWX_IP}/api/v2/ping/" + local pw + pw=$(get_awx_admin_password 2>/dev/null || echo "") + info " Admin: admin / ${pw}" + echo + info "Next steps:" + info " ${SCRIPT_NAME} --configure # set up project, inventory, templates" + info " ${SCRIPT_NAME} --join-aether # register with Aether" +} + +phase_create_vm() { + step "Phase 1: Create VM" + + if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then + return 0 + fi + + incus launch images:debian/12 "$(vm_ref)" --vm \ + --target "$TARGET_REMOTE" \ + -c "limits.cpu=${AWX_CPU}" \ + -c "limits.memory=${AWX_MEMORY}" \ + -d "root,size=${AWX_DISK}" + + success "VM created" + detail "Waiting 10s for initial boot..." + sleep 10 + + # Switch to macvlan for direct VLAN access + info "Configuring macvlan NIC (parent=mgmt)" + incus config device remove "$(vm_ref)" eth0 2>/dev/null || true + incus config device add "$(vm_ref)" eth0 nic \ + nictype=macvlan parent=mgmt + + success "NIC switched to macvlan — VM will restart networking" +} + +phase_configure_network() { + step "Phase 2: Configure static IP + hostname" + + wait_for_vm_agent 120 + + if ! dry_run_guard "Configure networking on $(vm_ref)"; then + return 0 + fi + + vm_exec bash -c " + hostnamectl set-hostname '${VM_NAME}' + + cat > /etc/netplan/50-static.yaml << 'NETPLAN' +network: + version: 2 + ethernets: + enp5s0: + addresses: [${AWX_IP}/${AWX_PREFIX}] + routes: + - to: default + via: ${AWX_GATEWAY} + nameservers: + addresses: [${AWX_DNS}] +NETPLAN + chmod 600 /etc/netplan/50-static.yaml + netplan apply + " + + success "Static IP ${AWX_IP}/${AWX_PREFIX} configured" + + # Verify connectivity + detail "Verifying network connectivity..." + sleep 3 + if vm_exec ping -c 1 -W 3 "${AWX_GATEWAY}" &>/dev/null; then + success "Gateway reachable from VM" + else + warn "Cannot reach gateway — check VLAN/network config" + fi +} + +phase_install_k3s() { + step "Phase 3: Install K3s" + + if ! dry_run_guard "Install K3s on $(vm_ref)"; then + return 0 + fi + + vm_exec bash -c " + curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 + " + success "K3s installed" + + info "Waiting for K3s node to be ready..." + vm_exec kubectl wait --for=condition=Ready node --all --timeout=120s + success "K3s node ready" +} + +phase_deploy_awx_operator() { + step "Phase 4: Deploy AWX Operator" + + if ! dry_run_guard "Deploy AWX Operator on $(vm_ref)"; then + return 0 + fi + + # Create namespace + vm_exec kubectl create namespace "$AWX_NAMESPACE" 2>/dev/null || true + + # Push kustomization files + info "Pushing operator manifests..." + incus file push "${MANIFESTS_DIR}/operator/kustomization.yaml" \ + "$(vm_ref)/tmp/awx-operator-kustomization.yaml" + + vm_exec bash -c " + mkdir -p /opt/awx/operator + cp /tmp/awx-operator-kustomization.yaml /opt/awx/operator/kustomization.yaml + kubectl apply -k /opt/awx/operator/ + " + + info "Waiting for AWX Operator (up to 5 min)..." + vm_exec kubectl -n "$AWX_NAMESPACE" wait \ + --for=condition=Available \ + deployment/awx-operator-controller-manager \ + --timeout=300s + + success "AWX Operator deployed" +} + +phase_deploy_awx_instance() { + step "Phase 5: Deploy AWX instance" + + if ! dry_run_guard "Deploy AWX instance on $(vm_ref)"; then + return 0 + fi + + # Push AWX CR manifests + info "Pushing AWX instance manifests..." + incus file push "${MANIFESTS_DIR}/base/awx.yaml" \ + "$(vm_ref)/tmp/awx-cr.yaml" + incus file push "${MANIFESTS_DIR}/base/kustomization.yaml" \ + "$(vm_ref)/tmp/awx-kustomization.yaml" + + vm_exec bash -c " + mkdir -p /opt/awx/base + cp /tmp/awx-cr.yaml /opt/awx/base/awx.yaml + cp /tmp/awx-kustomization.yaml /opt/awx/base/kustomization.yaml + kubectl apply -k /opt/awx/base/ + " + + info "AWX instance deploying — this takes 5-10 minutes..." + info "Waiting for AWX pods to start..." + + # Wait for AWX deployment to exist and become available + local elapsed=0 + local timeout=600 + while ! vm_exec kubectl -n "$AWX_NAMESPACE" get deployment awx-web &>/dev/null 2>&1; do + sleep 15 + elapsed=$((elapsed + 15)) + if [[ $elapsed -ge $timeout ]]; then + error "AWX web deployment not created after ${timeout}s" + warn "Check pod status: incus exec $(vm_ref) -- kubectl -n awx get pods" + return 1 + fi + [[ "$VERBOSE" == true ]] && detail "Waiting for AWX web deployment... (${elapsed}s)" + done + + vm_exec kubectl -n "$AWX_NAMESPACE" wait \ + --for=condition=Available \ + deployment/awx-web \ + --timeout=300s || { + warn "AWX web deployment not fully available yet — checking pods" + vm_exec kubectl -n "$AWX_NAMESPACE" get pods + } + + success "AWX instance deployed" +} + +phase_verify() { + step "Phase 6: Verify AWX" + + wait_for_awx_web 300 + + local admin_pw + admin_pw=$(get_awx_admin_password) + success "Admin password retrieved" + + # Test API ping + local ping_result + ping_result=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null) + if echo "$ping_result" | python3 -c "import sys,json; json.load(sys.stdin)" &>/dev/null; then + success "AWX API responding" + else + warn "AWX API returned unexpected response" + fi +} + +# --------------------------------------------------------------------------- +# Status — health checks +# --------------------------------------------------------------------------- + +action_status() { + step "AWX Status for $(vm_ref)" + echo + + # VM existence and state + if ! vm_exists; then + error "AWX VM does not exist — use --deploy to create" + return 1 + fi + + if ! vm_running; then + error "AWX VM exists but is not running" + return 1 + fi + success "VM: running" + + # K3s + if vm_exec kubectl get nodes &>/dev/null 2>&1; then + success "K3s: healthy" + if [[ "$VERBOSE" == true ]]; then + vm_exec kubectl get nodes -o wide + fi + else + error "K3s: not responding" + return 1 + fi + + # AWX pods + echo + info "AWX pods:" + vm_exec kubectl -n "$AWX_NAMESPACE" get pods -o wide 2>/dev/null || warn "Cannot list pods" + + # Pod health summary + local total running + total=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods --no-headers 2>/dev/null | wc -l || echo 0) + running=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods --no-headers 2>/dev/null | grep -c 'Running' || echo 0) + echo + if [[ "$total" -gt 0 && "$running" -eq "$total" ]]; then + success "All ${total} pods running" + else + warn "Pods: ${running}/${total} running" + fi + + # AWX web + local http_code + http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000") + if [[ "$http_code" == "200" ]]; then + success "AWX API: responding (HTTP ${http_code})" + else + error "AWX API: not responding (HTTP ${http_code})" + fi + + # Admin password + local admin_pw + admin_pw=$(get_awx_admin_password 2>/dev/null || echo "") + if [[ -n "$admin_pw" ]]; then + success "Admin password: ${admin_pw}" + else + warn "Cannot retrieve admin password" + fi + + # AWX version + local version + version=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown") + info "AWX version: ${version}" + + # Aether connectivity + echo + local aether_code + aether_code=$(curl -sk -o /dev/null -w '%{http_code}' "${AETHER_URL}/api/health" 2>/dev/null || echo "000") + if [[ "$aether_code" == "200" || "$aether_code" == "401" || "$aether_code" == "302" ]]; then + success "Aether: reachable at ${AETHER_URL}" + else + warn "Aether: not reachable (HTTP ${aether_code})" + fi +} + +# --------------------------------------------------------------------------- +# Heal — fix common issues +# --------------------------------------------------------------------------- + +action_heal() { + step "Healing AWX on $(vm_ref)" + echo + + if ! vm_exists || ! vm_running; then + error "AWX VM is not running" + return 1 + fi + + # 1. K3s service + info "Checking K3s service..." + if vm_exec systemctl is-active k3s &>/dev/null; then + success "K3s service: active" + else + warn "K3s service not active — restarting" + if dry_run_guard "systemctl restart k3s"; then + vm_exec systemctl restart k3s + sleep 10 + fi + fi + + # 2. Unhealthy pods + info "Checking for unhealthy pods..." + local unhealthy + unhealthy=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods --no-headers 2>/dev/null \ + | grep -vE 'Running|Completed' | awk '{print $1}' || true) + if [[ -n "$unhealthy" ]]; then + warn "Unhealthy pods found — deleting for operator to recreate" + for pod in $unhealthy; do + info " Deleting pod: $pod" + if dry_run_guard "kubectl delete pod $pod"; then + vm_exec kubectl -n "$AWX_NAMESPACE" delete pod "$pod" --grace-period=30 + fi + done + info "Waiting 30s for pods to restart..." + sleep 30 + else + success "All pods healthy" + fi + + # 3. AWX web responsiveness + info "Checking AWX web..." + local http_code + http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000") + if [[ "$http_code" == "200" ]]; then + success "AWX web: responding" + else + warn "AWX web not responding — restarting deployment" + if dry_run_guard "kubectl rollout restart deployment/awx-web"; then + vm_exec kubectl -n "$AWX_NAMESPACE" rollout restart deployment/awx-web 2>/dev/null || true + vm_exec kubectl -n "$AWX_NAMESPACE" rollout restart deployment/awx-task 2>/dev/null || true + wait_for_awx_web 300 + fi + fi + + # 4. Project sync + info "Triggering project sync..." + local project_id + project_id=$(awx_api_auth GET "/api/v2/projects/?name=incus-contrib" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + if [[ -n "$project_id" ]]; then + if dry_run_guard "POST /api/v2/projects/${project_id}/update/"; then + awx_api_auth POST "/api/v2/projects/${project_id}/update/" 2>/dev/null || true + success "Project sync triggered (ID: ${project_id})" + fi + else + info "No project configured yet — use --configure" + fi + + echo + success "Heal complete" +} + +# --------------------------------------------------------------------------- +# Configure — set up AWX project, inventory, templates +# --------------------------------------------------------------------------- + +action_configure() { + step "Configuring AWX" + echo + + if ! vm_running; then + error "AWX VM is not running" + return 1 + fi + + wait_for_awx_web 60 + + local admin_pw + admin_pw=$(get_awx_admin_password) + detail "Admin password: ${admin_pw}" + + # Helper for authenticated API calls with password + awx_auth() { + local method="$1" path="$2" + shift 2 + curl -sk -X "$method" "https://${AWX_IP}${path}" \ + --user "admin:${admin_pw}" \ + -H "Content-Type: application/json" \ + "$@" + } + + # 1. Create SCM credential for private Git repo + step "Creating SCM credential" + local ssh_key="" + local ssh_key_path="${HOME}/.ssh/id_ed25519" + [[ ! -f "$ssh_key_path" ]] && ssh_key_path="${HOME}/.ssh/id_rsa" + if [[ -f "$ssh_key_path" ]]; then + ssh_key=$(cat "$ssh_key_path") + info "Using SSH key: ${ssh_key_path}" + else + warn "No SSH key found at ~/.ssh/id_ed25519 or ~/.ssh/id_rsa" + warn "SCM credential will be created without a key — update manually in AWX" + fi + + local scm_cred_id + scm_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-contrib-scm" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + if [[ -z "$scm_cred_id" ]]; then + if [[ -n "$ssh_key" ]]; then + scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \ + -d "$(python3 -c " +import json, sys +print(json.dumps({ + 'name': 'incus-contrib-scm', + 'organization': 1, + 'credential_type': 2, + 'inputs': {'ssh_key_data': sys.stdin.read()} +}))" <<< "$ssh_key")" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + else + scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \ + -d '{"name":"incus-contrib-scm","organization":1,"credential_type":2,"inputs":{}}' 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + fi + [[ -n "$scm_cred_id" ]] && success "SCM credential created (ID: ${scm_cred_id})" || warn "Failed to create SCM credential" + else + success "SCM credential exists (ID: ${scm_cred_id})" + fi + + # 2. Create project + step "Creating project" + local project_id + project_id=$(awx_auth GET "/api/v2/projects/?name=incus-contrib" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + if [[ -z "$project_id" ]]; then + local project_payload + project_payload=$(python3 -c " +import json +d = { + 'name': 'incus-contrib', + 'organization': 1, + 'scm_type': 'git', + 'scm_url': '${GIT_REPO}', + 'scm_branch': '${GIT_BRANCH}', + 'scm_update_on_launch': True +} +print(json.dumps(d))") + [[ -n "$scm_cred_id" ]] && project_payload=$(echo "$project_payload" | python3 -c " +import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(json.dumps(d))") + + project_id=$(awx_auth POST "/api/v2/projects/" -d "$project_payload" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + [[ -n "$project_id" ]] && success "Project created (ID: ${project_id})" || warn "Failed to create project" + else + success "Project exists (ID: ${project_id})" + fi + + # Wait for initial project sync + if [[ -n "$project_id" ]]; then + info "Waiting for project sync..." + local sync_elapsed=0 + while [[ $sync_elapsed -lt 120 ]]; do + local status + status=$(awx_auth GET "/api/v2/projects/${project_id}/" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || true) + if [[ "$status" == "successful" ]]; then + success "Project synced" + break + elif [[ "$status" == "failed" || "$status" == "error" ]]; then + warn "Project sync failed — check SCM credential and repo URL" + break + fi + sleep 5 + sync_elapsed=$((sync_elapsed + 5)) + done + fi + + # 3. Create inventory + step "Creating inventory" + local inventory_id + inventory_id=$(awx_auth GET "/api/v2/inventories/?name=incus-instances" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + if [[ -z "$inventory_id" ]]; then + inventory_id=$(awx_auth POST "/api/v2/inventories/" \ + -d '{"name":"incus-instances","organization":1}' 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + [[ -n "$inventory_id" ]] && success "Inventory created (ID: ${inventory_id})" || warn "Failed to create inventory" + else + success "Inventory exists (ID: ${inventory_id})" + fi + + # 4. Create machine credential (SSH to instances) + step "Creating machine credential" + local machine_cred_id + machine_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-instances" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + if [[ -z "$machine_cred_id" ]]; then + local machine_cred_payload='{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}' + if [[ -n "$ssh_key" ]]; then + machine_cred_payload=$(python3 -c " +import json, sys +key = sys.stdin.read() +print(json.dumps({ + 'name': 'incus-instances', + 'organization': 1, + 'credential_type': 1, + 'inputs': {'username': 'root', 'ssh_key_data': key} +}))" <<< "$ssh_key") + fi + machine_cred_id=$(awx_auth POST "/api/v2/credentials/" -d "$machine_cred_payload" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + [[ -n "$machine_cred_id" ]] && success "Machine credential created (ID: ${machine_cred_id})" || warn "Failed to create machine credential" + else + success "Machine credential exists (ID: ${machine_cred_id})" + fi + + # 5. Create job templates + step "Creating job templates" + for tmpl_name in post-deploy decommission; do + local tmpl_id + tmpl_id=$(awx_auth GET "/api/v2/job_templates/?name=${tmpl_name}" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + if [[ -z "$tmpl_id" ]]; then + local tmpl_payload + tmpl_payload=$(python3 -c " +import json +d = { + 'name': '${tmpl_name}', + 'organization': 1, + 'job_type': 'run', + 'ask_variables_on_launch': True, + 'playbook': '${PLAYBOOK_DIR}/${tmpl_name}.yml' +} +pid = '${project_id}' +iid = '${inventory_id}' +cid = '${machine_cred_id}' +if pid: d['project'] = int(pid) +if iid: d['inventory'] = int(iid) +print(json.dumps(d))") + + tmpl_id=$(awx_auth POST "/api/v2/job_templates/" -d "$tmpl_payload" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true) + + # Associate credential separately (templates use credential list) + if [[ -n "$tmpl_id" && -n "$machine_cred_id" ]]; then + awx_auth POST "/api/v2/job_templates/${tmpl_id}/credentials/" \ + -d "{\"id\":${machine_cred_id}}" 2>/dev/null || true + fi + + [[ -n "$tmpl_id" ]] && success "Template '${tmpl_name}' created (ID: ${tmpl_id})" || warn "Failed to create template '${tmpl_name}'" + else + success "Template '${tmpl_name}' exists (ID: ${tmpl_id})" + fi + done + + echo + success "AWX configuration complete" + echo + info "Template IDs (for Aether binding):" + for tmpl_name in post-deploy decommission; do + local tid + tid=$(awx_auth GET "/api/v2/job_templates/?name=${tmpl_name}" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + info " ${tmpl_name}: ${tid:-unknown}" + done + echo + info "Next: ${SCRIPT_NAME} --join-aether" +} + +# --------------------------------------------------------------------------- +# Join Aether — register AWX endpoint + bind to cluster +# --------------------------------------------------------------------------- + +action_join_aether() { + step "Registering AWX with Aether" + echo + + if ! vm_running; then + error "AWX VM is not running" + return 1 + fi + + # 1. Create Personal Access Token + step "Creating AWX Personal Access Token" + local admin_pw + admin_pw=$(get_awx_admin_password) + + local pat_response + pat_response=$(curl -sk -X POST "https://${AWX_IP}/api/v2/users/1/personal_tokens/" \ + --user "admin:${admin_pw}" \ + -H "Content-Type: application/json" \ + -d '{"description": "Aether integration", "scope": "write"}' 2>/dev/null) + + local awx_token + awx_token=$(echo "$pat_response" | python3 -c "import sys,json; print(json.load(sys.stdin).get('token',''))" 2>/dev/null || true) + + if [[ -z "$awx_token" ]]; then + error "Failed to create PAT — check AWX admin credentials" + detail "Response: ${pat_response}" + return 1 + fi + success "PAT created" + + # 2. Get template IDs + local post_deploy_id decommission_id + post_deploy_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=post-deploy" \ + --user "admin:${admin_pw}" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + decommission_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=decommission" \ + --user "admin:${admin_pw}" 2>/dev/null \ + | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true) + + info "Post-deploy template ID: ${post_deploy_id:-not found}" + info "Decommission template ID: ${decommission_id:-not found}" + + echo + info "To complete Aether integration:" + echo + info "1. Log into Aether at ${AETHER_URL}" + info "2. Navigate to Settings → Ansible Automation (/awx-endpoints)" + info "3. Add Endpoint:" + info " Name: lab-awx" + info " URL: https://${AWX_IP}" + info " Token: ${awx_token}" + info " Verify SSL: unchecked" + info "4. Configure cluster AWX binding:" + info " Post-deploy template ID: ${post_deploy_id:-}" + info " Decommission template ID: ${decommission_id:-}" + info " Job timeout: 600 seconds" + echo + + # Print API commands for scripted registration + if [[ "$VERBOSE" == true ]]; then + detail "API commands for scripted registration:" + detail "" + detail "# Register endpoint:" + detail "curl -sSk -b cookies.txt -H 'X-CSRF-Token: \$CSRF' \\" + detail " -H 'Referer: ${AETHER_URL}/' \\" + detail " -X POST ${AETHER_URL}/api/awx/endpoints \\" + detail " -H 'Content-Type: application/json' \\" + detail " -d '{\"name\":\"lab-awx\",\"url\":\"https://${AWX_IP}\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'" + detail "" + detail "# Bind cluster (replace ENDPOINT_ID):" + detail "curl -sSk -b cookies.txt -H 'X-CSRF-Token: \$CSRF' \\" + detail " -H 'Referer: ${AETHER_URL}/' \\" + detail " -X PUT ${AETHER_URL}/api/clusters/${AETHER_CLUSTER_ID}/awx-config \\" + detail " -H 'Content-Type: application/json' \\" + detail " -d '{\"cluster_id\":${AETHER_CLUSTER_ID},\"awx_endpoint_id\":ENDPOINT_ID,\"awx_post_deploy_template_id\":${post_deploy_id:-0},\"awx_decommission_template_id\":${decommission_id:-0},\"awx_job_timeout_seconds\":600}'" + fi +} + +# --------------------------------------------------------------------------- +# Cleanup — destroy AWX VM +# --------------------------------------------------------------------------- + +action_cleanup() { + step "Cleaning up AWX VM" + + if ! vm_exists; then + info "AWX VM does not exist — nothing to clean up" + return 0 + fi + + echo + warn "This will permanently destroy the AWX VM '$(vm_ref)'" + warn "All AWX data (jobs, templates, credentials) will be lost." + echo + + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) Would destroy $(vm_ref)" + return 0 + fi + + read -r -p "Type 'yes' to confirm: " confirm + if [[ "$confirm" != "yes" ]]; then + info "Aborted" + return 0 + fi + + if vm_running; then + info "Stopping VM..." + incus stop "$(vm_ref)" --force 2>/dev/null || true + sleep 3 + fi + + info "Deleting VM..." + incus delete "$(vm_ref)" --force 2>/dev/null || true + + success "AWX VM destroyed" +} + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + --deploy) ACTION="deploy";; + --status) ACTION="status";; + --heal) ACTION="heal";; + --configure) ACTION="configure";; + --join-aether) ACTION="join-aether";; + --cleanup) ACTION="cleanup";; + --doctor) ACTION="doctor";; + -c|--config) CONFIG_FILE="$2"; shift;; + -n|--dry-run) DRY_RUN=true;; + -v|--verbose) VERBOSE=true;; + -q|--quiet) QUIET=true;; + -h|--help) usage; exit 0;; + *) + error "Unknown option: $1" + echo "Run '${SCRIPT_NAME} --help' for usage." + exit 1 + ;; + esac + shift + done + + if [[ -z "$ACTION" ]]; then + error "No action specified" + echo "Run '${SCRIPT_NAME} --help' for usage." + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +main() { + setup_colors + parse_args "$@" + + # Load config file if provided + if [[ -n "$CONFIG_FILE" ]]; then + parse_config "$CONFIG_FILE" + fi + + echo + case "$ACTION" in + deploy) action_deploy;; + status) action_status;; + heal) action_heal;; + configure) action_configure;; + join-aether) action_join_aether;; + cleanup) action_cleanup;; + doctor) action_doctor;; + esac +} + +main "$@" diff --git a/notes/awx-guide.md b/notes/awx-guide.md new file mode 100644 index 0000000..ea6af8c --- /dev/null +++ b/notes/awx-guide.md @@ -0,0 +1,624 @@ +# AWX — Ansible Automation for Aether Lifecycle Hooks + +AWX (the open-source upstream of Ansible Tower) provides a web UI and REST +API for running Ansible playbooks. Aether integrates with AWX to run +post-deploy and decommission playbooks as lifecycle hooks on every instance +create/delete. + +This guide covers deploying AWX on the Incus cluster, connecting it to +Aether, writing lifecycle playbooks, and using the `deploy-awx` management +script. + +## Architecture + +``` + User deploys VM via Aether + | + v + Aether creates instance on Incus cluster + | + v + Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/ + with extra_vars: vm_name, vm_ip, environment, owner, cost_center + | + v + AWX runs post-deploy.yml playbook + 1. SSH to new instance (vm_ip) + 2. Set hostname, install packages, configure timezone + 3. Log to deployment ledger + | + v + Instance ready, user notified +``` + +Decommission works in reverse — Aether triggers the decommission template +before deleting the instance. + +### Separation of concerns + +| Layer | Tool | Responsibility | +|-------|------|----------------| +| Infrastructure | Aether + Incus | VM creation, networking, resources, lifecycle | +| Configuration | AWX + Ansible | OS config, packages, services, integrations | +| Automation glue | Aether AWX binding | Triggers playbooks at create/delete | + +### What Aether passes to AWX + +Every AWX job launched by Aether receives these extra vars: + +| Variable | Description | +|----------|-------------| +| `vm_name` | Instance hostname | +| `vm_ip` | IP address assigned by Aether | +| `environment` | Deployment environment (dev, staging, prod) | +| `owner` | VM owner or team | +| `cost_center` | Billing/cost allocation | +| (custom tags) | All blueprint tags and instance metadata | + +### AWX API endpoints Aether uses + +| Endpoint | Purpose | +|----------|---------| +| `GET /api/v2/ping/` | Health check | +| `GET /api/v2/job_templates/{id}/` | Validate template exists | +| `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars | +| `GET /api/v2/jobs/{id}/` | Poll job status | + +## Lab deployment + +### Resource requirements + +| Resource | Recommended | Minimum | +|----------|-------------|---------| +| vCPU | 4 | 2 | +| RAM | 8 GiB | 4 GiB | +| Disk | 40 GiB | 20 GiB | + +K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM +can run AWX but leaves no headroom for execution environments. + +### Lab details + +| Setting | Value | +|---------|-------| +| VM name | `awx` | +| Location | oc-node-02 | +| IP | 192.168.102.161/22 (VLAN 69) | +| Gateway | 192.168.100.1 | +| DNS | 192.168.100.1 | +| Port | 443 (HTTPS via K3s Traefik) | +| OS | Debian 12 | +| K8s | K3s (single-node) | +| AWX Operator | 2.19.1 | + +IP is adjacent to Aether at .160 on the same VLAN. + +### Quick deploy + +```bash +# Check prerequisites +./incusos/deploy-awx --doctor + +# Full deployment (~10-15 minutes) +./incusos/deploy-awx --deploy + +# Configure AWX (project, inventory, templates) +./incusos/deploy-awx --configure + +# Register with Aether +./incusos/deploy-awx --join-aether +``` + +### Manual deployment + +#### Step 1: Create VM + +```bash +# Launch Debian 12 VM on the cluster +incus launch images:debian/12 oc-node-02:awx --vm \ + --target oc-node-02 \ + -c limits.cpu=4 -c limits.memory=8GiB \ + -d root,size=40GiB + +# Switch to macvlan for direct VLAN access +incus config device remove oc-node-02:awx eth0 +incus config device add oc-node-02:awx eth0 nic \ + nictype=macvlan parent=mgmt +``` + +#### Step 2: Configure static IP + +```bash +incus exec oc-node-02:awx -- bash -c " + hostnamectl set-hostname awx + cat > /etc/netplan/50-static.yaml << 'NETPLAN' +network: + version: 2 + ethernets: + enp5s0: + addresses: [192.168.102.161/22] + routes: + - to: default + via: 192.168.100.1 + nameservers: + addresses: [192.168.100.1] +NETPLAN + chmod 600 /etc/netplan/50-static.yaml + netplan apply +" +``` + +#### Step 3: Install K3s + +```bash +incus exec oc-node-02:awx -- bash -c " + curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 + kubectl wait --for=condition=Ready node --all --timeout=120s +" +``` + +#### Step 4: Deploy AWX Operator + +```bash +incus exec oc-node-02:awx -- bash -c " + kubectl create namespace awx + + mkdir -p /opt/awx/operator + cat > /opt/awx/operator/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - github.com/ansible/awx-operator/config/default?ref=2.19.1 +images: + - name: quay.io/ansible/awx-operator + newTag: 2.19.1 +namespace: awx +EOF + kubectl apply -k /opt/awx/operator/ + + # Wait for operator (~2-3 min) + kubectl -n awx wait --for=condition=Available \ + deployment/awx-operator-controller-manager --timeout=300s +" +``` + +#### Step 5: Deploy AWX instance + +The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`. +Push and apply: + +```bash +incus file push incusos/awx-manifests/base/awx.yaml \ + oc-node-02:awx/opt/awx/base/awx.yaml +incus file push incusos/awx-manifests/base/kustomization.yaml \ + oc-node-02:awx/opt/awx/base/kustomization.yaml + +incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/ + +# Wait 5-10 min for all pods to start +incus exec oc-node-02:awx -- kubectl -n awx get pods -w +``` + +#### Step 6: Verify + +```bash +# Get admin password +ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \ + awx-admin-password -o jsonpath='{.data.password}' | base64 -d) +echo "Admin password: $ADMIN_PW" + +# Test API +curl -sk https://192.168.102.161/api/v2/ping/ +``` + +Web UI: `https://192.168.102.161/` — login with `admin` / password above. + +## AWX configuration + +All configuration is API-driven so it can be reproduced. The `deploy-awx +--configure` command automates these steps. + +### Project (Git repository) + +Create a project pointing to the `ansible/` directory in this repo: + +```bash +curl -sk -X POST https://192.168.102.161/api/v2/projects/ \ + --user "admin:${ADMIN_PW}" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "incus-contrib", + "organization": 1, + "scm_type": "git", + "scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git", + "scm_branch": "master", + "scm_update_on_launch": true + }' +``` + +AWX clones the repo and discovers playbooks under `ansible/playbooks/`. + +### Inventory + +Aether passes `vm_ip` as an extra var. Playbooks use `add_host` to +dynamically create an in-memory inventory entry for the target instance. +The AWX inventory is a placeholder: + +```bash +curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \ + --user "admin:${ADMIN_PW}" \ + -H "Content-Type: application/json" \ + -d '{"name": "incus-instances", "organization": 1}' +``` + +### Credentials + +Two credentials: + +1. **SCM credential** (type 2) — SSH key for cloning the private Git repo +2. **Machine credential** (type 1) — SSH key for connecting to managed instances + +```bash +# Machine credential +curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \ + --user "admin:${ADMIN_PW}" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "incus-instances", + "organization": 1, + "credential_type": 1, + "inputs": {"username": "root", "ssh_key_data": "..."} + }' +``` + +### Job templates + +Two templates matching Aether's lifecycle hooks: + +| Template | Playbook | Purpose | +|----------|----------|---------| +| `post-deploy` | `ansible/playbooks/post-deploy.yml` | Runs after instance creation | +| `decommission` | `ansible/playbooks/decommission.yml` | Runs before instance deletion | + +Both must have `ask_variables_on_launch: true` so Aether can inject extra vars. + +```bash +curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \ + --user "admin:${ADMIN_PW}" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "post-deploy", + "organization": 1, + "project": , + "playbook": "ansible/playbooks/post-deploy.yml", + "inventory": , + "ask_variables_on_launch": true + }' +``` + +### Personal Access Token (for Aether) + +```bash +AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \ + --user "admin:${ADMIN_PW}" \ + -H "Content-Type: application/json" \ + -d '{"description": "Aether integration", "scope": "write"}' \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") +``` + +## Aether integration + +### Register AWX endpoint + +In the Aether web UI: Settings → Ansible Automation (`/awx-endpoints`) → +Add Endpoint: + +| Field | Value | +|-------|-------| +| Name | `lab-awx` | +| URL | `https://192.168.102.161` | +| Token | (PAT from above) | +| Verify SSL | unchecked (self-signed) | + +Or via API: +```bash +curl -sSk -b cookies.txt \ + -H "X-CSRF-Token: $CSRF" \ + -H "Referer: https://192.168.102.160:8443/" \ + -X POST https://192.168.102.160:8443/api/awx/endpoints \ + -H "Content-Type: application/json" \ + -d '{ + "name": "lab-awx", + "url": "https://192.168.102.161", + "token": "'$AWX_TOKEN'", + "verify_ssl": false + }' +``` + +### Bind cluster to AWX + +In the Aether UI: AWX endpoint → cluster config, or via API: + +```bash +curl -sSk -b cookies.txt \ + -H "X-CSRF-Token: $CSRF" \ + -H "Referer: https://192.168.102.160:8443/" \ + -X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \ + -H "Content-Type: application/json" \ + -d '{ + "cluster_id": 52, + "awx_endpoint_id": , + "awx_post_deploy_template_id": , + "awx_decommission_template_id": , + "awx_job_timeout_seconds": 600 + }' +``` + +### Verify integration + +- Aether Health page (`/health`) shows AWX endpoint as "Healthy" +- Deploy a test instance via Aether — AWX post-deploy job should trigger +- Check AWX job history for the completed run +- Delete the test instance — decommission job should trigger + +### Lifecycle behavior + +| Hook | Trigger | On failure | +|------|---------|------------| +| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) | +| Decommission | Before instance deletion | Failure does NOT block deletion | + +Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal +state (successful, failed, error) or the timeout expires. + +## Writing playbooks + +### Directory structure + +``` +ansible/ +├── ansible.cfg # Project-level config +├── playbooks/ +│ ├── post-deploy.yml # Aether post-deploy hook +│ └── decommission.yml # Aether decommission hook +└── roles/ + └── base-config/ + ├── tasks/main.yml # Base OS configuration + └── templates/ # Jinja2 templates (future) +``` + +### Pattern: dynamic host from extra vars + +Aether passes `vm_ip` — playbooks use `add_host` to target it: + +```yaml +- name: Add target host + hosts: localhost + gather_facts: false + tasks: + - ansible.builtin.add_host: + name: "{{ vm_ip }}" + groups: target + ansible_user: root + +- name: Configure target + hosts: target + tasks: + - ansible.builtin.wait_for_connection: + delay: 5 + timeout: 300 + # ... configuration tasks +``` + +### Pattern: best-effort decommission + +Decommission playbooks should never block instance deletion: + +```yaml +- name: Attempt cleanup on target + hosts: target + ignore_unreachable: true + tasks: + - name: Check if host is reachable + ansible.builtin.wait_for_connection: + timeout: 15 + ignore_errors: true + register: host_check + + - name: Cleanup tasks + # ... + when: host_check is succeeded +``` + +### The base-config role + +The included `base-config` role performs minimal post-deploy setup: + +1. Set hostname +2. Update apt cache +3. Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades) +4. Set timezone to UTC +5. Enable unattended security upgrades +6. Write deployment metadata to `/etc/deploy-info` + +### Extending playbooks + +Common additions for production environments: + +| Task | Method | Module | +|------|--------|--------| +| DNS registration | API from AWX controller | `community.general.nsupdate` or REST | +| Monitoring agent | SSH to target | `ansible.builtin.package` + config | +| SSH key distribution | SSH to target | `ansible.posix.authorized_key` | +| Certificate provisioning | API from controller | `community.crypto.acme_certificate` | +| Backup enrollment | API from controller | REST calls to backup system | + +External system tasks use `delegate_to: localhost` to run from the AWX +controller rather than on the target instance. + +## The deploy-awx script + +### Usage + +``` +deploy-awx [OPTIONS] + +Actions: + --deploy Full deploy: create VM → K3s → AWX → configure + --status Check AWX health, K3s pods, Aether connection + --heal Restart failed pods, re-sync project, re-validate + --configure (Re-)configure AWX: project, inventory, templates + --join-aether Register AWX with Aether + print binding instructions + --cleanup Destroy the AWX VM + --doctor Check prerequisites + +Options: + -c, --config FILE Configuration file (YAML with awx: section) + -n, --dry-run Preview actions without executing + -v, --verbose Show detailed output + -q, --quiet Suppress informational output +``` + +### Configuration + +Defaults are built into the script for the lab environment. Override with +a YAML config file: + +```yaml +awx: + vm_name: awx + target_node: oc-node-02 + ip: 192.168.102.161/22 + gateway: 192.168.100.1 + dns: 192.168.100.1 + cpu: 4 + memory: 8GiB + disk: 40GiB + git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git + git_branch: master + aether_url: https://192.168.102.160:8443 + aether_cluster_id: 52 +``` + +### Deployment phases + +1. **Create VM** — `incus launch images:debian/12` with macvlan NIC +2. **Configure network** — static IP via netplan, hostname +3. **Install K3s** — single-node K3s with kubeconfig mode 644 +4. **Deploy AWX Operator** — kustomize from `awx-manifests/operator/` +5. **Deploy AWX instance** — kustomize from `awx-manifests/base/` +6. **Verify** — API ping, admin password retrieval + +### Heal logic + +The `--heal` command checks and fixes: + +1. K3s service running → restart if not +2. All AWX pods healthy → delete unhealthy (operator recreates) +3. AWX web responding → restart deployments if not +4. Project sync current → trigger re-sync + +### Manifests + +K8s manifests stored in `incusos/awx-manifests/`: + +``` +awx-manifests/ +├── operator/ +│ └── kustomization.yaml # AWX Operator 2.19.1 via kustomize +└── base/ + ├── kustomization.yaml # AWX instance resources + └── awx.yaml # AWX custom resource (pods, storage, ingress) +``` + +The AWX CR in `awx.yaml` defines resource limits tuned for the 8 GiB VM: +- Web pod: 512 MiB → 1 GiB +- Task pod: 512 MiB → 2 GiB +- EE pod: 256 MiB → 1 GiB +- PostgreSQL: 256 MiB → 512 MiB + +## Troubleshooting + +### Pod status + +```bash +# List all AWX pods +incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide + +# Describe a failing pod +incus exec oc-node-02:awx -- kubectl -n awx describe pod + +# View pod logs +incus exec oc-node-02:awx -- kubectl -n awx logs +``` + +### Common issues + +**AWX pods stuck in Pending**: insufficient resources. Check node capacity: +```bash +incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources" +``` + +**AWX web returns 502**: task or web pod crashed. Restart: +```bash +incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web +incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task +``` + +**Project sync fails**: SCM credential or network issue. Check: +```bash +incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm +``` + +**Job fails with "Host unreachable"**: target instance not booted yet or +SSH not available. Increase the `wait_for_connection` timeout in the playbook +or add a delay in Aether's job timeout setting. + +**K3s not starting**: VM resources exhausted. Check: +```bash +incus exec oc-node-02:awx -- systemctl status k3s +incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50 +``` + +**Cannot reach AWX from outside**: macvlan NIC misconfigured. Verify: +```bash +incus exec oc-node-02:awx -- ip addr show enp5s0 +incus exec oc-node-02:awx -- ip route +``` + +### Useful commands + +```bash +# AWX version +curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool + +# List job templates +curl -sk https://192.168.102.161/api/v2/job_templates/ \ + --user "admin:$PW" | python3 -m json.tool + +# List recent jobs +curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \ + --user "admin:$PW" | python3 -c " +import sys,json +for j in json.load(sys.stdin)['results']: + print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}')" + +# Manually trigger post-deploy +curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \ + --user "admin:$PW" \ + -H "Content-Type: application/json" \ + -d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}' +``` + +### Rollback + +If AWX is broken beyond repair: +```bash +./incusos/deploy-awx --cleanup +./incusos/deploy-awx --deploy +./incusos/deploy-awx --configure +./incusos/deploy-awx --join-aether +``` + +The AWX VM is a standalone workload with no cluster dependencies. Destroying +and redeploying has zero impact on the Incus cluster or other workloads.