Validate AWX guide against real infrastructure, fix all bugs

End-to-end tested: Aether deploy triggers AWX post-deploy job,
container gets configured (hostname, packages, timezone, deploy-info),
decommission job runs on delete. All playbooks validated working.

Key fixes from testing:
- Playbooks: use Incus REST API (not SSH) — bridge subnet not routable
  from management VLAN, nftables blocks inbound forwarding
- Aether vars: ffsdn_* prefix (not vm_name/vm_ip/environment/owner)
- AWX EE paths: /runner/project/ (not /var/lib/awx/projects/)
- AWX access: NodePort 30080 (not Traefik ingress — 404 for IP access)
- deploy-awx: systemd-networkd (not netplan), manual project (not git
  SCM — EE can't reach private repo), Incus cert push, prereq install
- Remove unused base-config role (tasks inlined as setup script)
- Simplify ansible.cfg (no SSH settings needed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-24 01:53:54 +01:00
parent 6a74925fc4
commit 0b82b9ce3b
8 changed files with 837 additions and 633 deletions

View File

@ -15,12 +15,9 @@ incu-contrib/
├── .gitignore
├── ansible/ # Ansible playbooks for AWX/Aether lifecycle hooks
│ ├── ansible.cfg # Project-level Ansible config
│ ├── playbooks/
│ │ ├── post-deploy.yml # Runs after Aether creates an instance
│ │ └── decommission.yml # Runs before Aether deletes an instance
│ └── roles/
│ └── base-config/ # Base OS configuration role
│ └── tasks/main.yml
│ └── playbooks/
│ ├── post-deploy.yml # Runs after Aether creates an instance (Incus API)
│ └── decommission.yml # Runs before Aether deletes an instance
├── incusos/ # IncusOS installation tooling
│ ├── README.md # Detailed usage docs
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
@ -826,22 +823,40 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
- **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
- **AWX URL**: `http://192.168.102.161:30080` — exposed via K3s NodePort.
Traefik ingress returns 404 for IP-based access; use NodePort directly.
- **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
`--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
- **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
- **Ansible playbooks** in `ansible/` directory:
- `playbooks/post-deploy.yml` -- runs after Aether creates an instance
- `playbooks/decommission.yml` -- runs before Aether deletes an instance
- `roles/base-config/` -- hostname, packages, timezone, deploy metadata
- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with
a Personal Access Token. Per-cluster config binds post-deploy and
decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/`
with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`).
- **Aether extra vars use `ffsdn_` prefix**: Aether passes `ffsdn_instance_name`,
`ffsdn_instance_ip`, `ffsdn_cluster_id`, `ffsdn_cluster_name`,
`ffsdn_deployed_by`, `ffsdn_image_os`, `ffsdn_image_release`,
`ffsdn_image_alias`. It does NOT pass `vm_name`, `vm_ip`, `environment`,
`owner`, or `cost_center` (the original plan assumed these).
- **Playbook pattern — Incus REST API, not SSH**: AWX cannot SSH to containers
on incusbr0 (bridge subnet not routable from management VLAN, IncusOS
nftables blocks inbound forwarding). Playbooks use the Incus REST API
(`uri` module with client cert) for file push + exec. The cert is at
`/runner/project/incus-client.crt` during job execution (AWX EE mounts
projects at `/runner/project/`, NOT `/var/lib/awx/projects/`).
- **Manual project (local_path)**: AWX EE containers cannot reach the private
git repo. Use `scm_type: ""` with `local_path: "incus-contrib"` and push
playbooks directly to the AWX task pod at
`/var/lib/awx/projects/incus-contrib/playbooks/`.
- **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
deleted). Decommission failure does NOT block deletion.
- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to
create dynamic inventory → SSH to target for configuration. External system
tasks use `delegate_to: localhost`.
- **Aether cluster AWX config API bug**: `PUT /api/clusters/{id}/awx-config`
returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL
UPDATE on the `clusters` table from within the Aether container.
- **Self-referencing vars cause infinite recursion in Ansible**: patterns like
`vm_ip: "{{ vm_ip | default('') }}"` cause `AnsibleUndefinedVariable`
recursive loop when the variable is not provided. Use `ffsdn_*` vars
directly without redefining them.
- **Ansible `environment` is a reserved keyword**: using it as an extra var
resolves to `[]` instead of the string value.
- See `notes/awx-guide.md` for full deployment guide and troubleshooting.
## Git workflow

View File

@ -31,13 +31,13 @@ See [`incusos/README.md`](incusos/README.md) for full documentation.
### [`ansible/`](ansible/)
Ansible playbooks and roles for AWX/Aether lifecycle hooks:
Ansible playbooks for AWX/Aether lifecycle hooks. Playbooks use the Incus REST
API (not SSH) for all instance operations — works regardless of network topology.
- **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance.
Configures hostname, installs base packages, sets timezone, logs deployment.
- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
Graceful service shutdown and decommission logging (best-effort).
- **`roles/base-config/`** -- Base OS configuration role used by post-deploy.
### [`notes/`](notes/)

View File

@ -2,10 +2,7 @@
# Inventory — AWX manages inventory; this is for local testing only
inventory = localhost,
# Roles path
roles_path = roles
# Reduce SSH overhead
# Reduce overhead
forks = 10
timeout = 30
@ -18,11 +15,7 @@ stdout_callback = yaml
# Host key checking — disabled for dynamic lab instances
host_key_checking = False
[privilege_escalation]
become = True
become_method = sudo
[ssh_connection]
# Speed up SSH connections
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no
# Note: playbooks use the Incus REST API (uri module with client cert)
# for all instance operations — no SSH connection to instances is needed.
# The [ssh_connection] and [privilege_escalation] sections are therefore
# not required.

View File

@ -1,78 +1,77 @@
---
# decommission.yml — Runs before Aether deletes an instance
#
# Aether passes the same extra vars as post-deploy:
# vm_name, vm_ip, environment, owner, cost_center
# Aether passes the same ffsdn_ extra vars as post-deploy:
# ffsdn_instance_name, ffsdn_instance_ip, ffsdn_cluster_id,
# ffsdn_cluster_name, ffsdn_deployed_by
#
# This playbook runs best-effort: failures do NOT block instance
# deletion in Aether. The playbook should be defensive and handle
# unreachable hosts gracefully.
# deletion in Aether. Uses the Incus REST API for all operations
# (no SSH needed).
- name: Decommission — prepare instance for deletion
- name: Decommission — graceful shutdown and logging
hosts: localhost
gather_facts: false
connection: local
vars:
vm_name: "{{ vm_name | default('unknown') }}"
vm_ip: "{{ vm_ip | default('') }}"
environment_name: "{{ environment | default('lab') }}"
owner: "{{ owner | default('unassigned') }}"
cost_center: "{{ cost_center | default('') }}"
ssh_user: root
incus_api: "https://192.168.102.140:8443"
# AWX runner mounts project at /runner/project/ during job execution
incus_cert: "/runner/project/incus-client.crt"
incus_key: "/runner/project/incus-client.key"
tasks:
- name: Add target host to in-memory inventory
ansible.builtin.add_host:
name: "{{ vm_ip }}"
groups: target
ansible_user: "{{ ssh_user }}"
ansible_host: "{{ vm_ip }}"
when: vm_ip | length > 0
- name: Attempt graceful shutdown on target
hosts: target
gather_facts: false
ignore_unreachable: true
tasks:
- name: Check if host is reachable
ansible.builtin.wait_for_connection:
delay: 0
timeout: 15
register: host_check
ignore_errors: true
- name: Check if instance still exists
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
method: GET
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
return_content: true
status_code: [200, 404]
register: instance_check
when: ffsdn_instance_name is defined
- name: Stop application services (best-effort)
ansible.builtin.shell: |
# Stop user-facing services before deletion
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
method: POST
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
body_format: json
body:
command:
- "/bin/bash"
- "-c"
- |
systemctl list-units --type=service --state=running --no-legend \
| awk '{print $1}' \
| grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
| head -20 \
| xargs -r -n1 systemctl stop
register: service_stop
| xargs -r -n1 systemctl stop 2>/dev/null || true
journalctl --flush 2>/dev/null || true
record-output: false
interactive: false
wait-for-websocket: false
status_code: [202]
ignore_errors: true
when: host_check is succeeded
when:
- instance_check is defined
- instance_check.status == 200
- instance_check.json.metadata.status is defined
- instance_check.json.metadata.status == "Running"
- name: Flush logs to persistent storage
ansible.builtin.command: journalctl --flush
ignore_errors: true
when: host_check is succeeded
- name: Log decommission
hosts: localhost
gather_facts: false
connection: local
tasks:
- name: Record decommission in ledger
ansible.builtin.lineinfile:
path: /var/log/awx-deploy-ledger.log
path: /tmp/awx-deploy-ledger.log
line: >-
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }}
environment={{ environment_name }} owner={{ owner }}
cost_center={{ cost_center }} status=completed
{{ lookup('pipe', 'date -Iseconds') }}
DECOMMISSION instance={{ ffsdn_instance_name | default('unknown') }}
ip={{ ffsdn_instance_ip | default('') }}
cluster={{ ffsdn_cluster_name | default('unknown') }}
deployed_by={{ ffsdn_deployed_by | default('unknown') }}
status=completed
create: true
mode: "0644"

View File

@ -1,88 +1,200 @@
---
# post-deploy.yml — Runs after Aether creates an instance
#
# Aether passes extra vars:
# vm_name — instance hostname
# vm_ip — IP address assigned by Aether
# environment — deployment environment (dev, staging, prod)
# owner — VM owner or team
# cost_center — billing/cost allocation
# (plus any custom tags from the blueprint)
# Aether passes these extra vars (ffsdn_ prefix):
# ffsdn_instance_name — instance name
# ffsdn_instance_ip — IP address assigned by Aether
# ffsdn_cluster_id — numeric cluster ID
# ffsdn_cluster_name — cluster display name
# ffsdn_deployed_by — Aether username who triggered the deploy
# ffsdn_image_os — image OS (e.g., "Debian")
# ffsdn_image_release — image release (e.g., "bookworm")
# ffsdn_image_alias — image alias (may be empty)
#
# AWX job template must have "ask_variables_on_launch: true" so Aether
# can inject these variables at runtime.
# This playbook configures new instances via the Incus REST API
# (file push + exec), which works regardless of network topology.
# AWX does NOT need SSH access to the container — it talks to the
# Incus cluster API at the node level.
#
# Requirements:
# - Incus client cert/key at /var/lib/awx/projects/incus-contrib/
# - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443)
- name: Post-deploy — configure newly provisioned instance
- name: Post-deploy — configure instance via Incus API
hosts: localhost
gather_facts: false
connection: local
vars:
# Defaults (overridden by Aether extra vars)
vm_name: "{{ vm_name | default('unknown') }}"
vm_ip: "{{ vm_ip | default('') }}"
environment_name: "{{ environment | default('lab') }}"
owner: "{{ owner | default('unassigned') }}"
cost_center: "{{ cost_center | default('') }}"
ssh_user: root
base_packages:
- curl
- vim
- htop
- jq
- tmux
- rsync
- unattended-upgrades
# Incus cluster API (any cluster member works)
incus_api: "https://192.168.102.140:8443"
# AWX runner mounts project at /runner/project/ during job execution
incus_cert: "/runner/project/incus-client.crt"
incus_key: "/runner/project/incus-client.key"
base_packages: "curl vim htop jq tmux rsync unattended-upgrades"
pre_tasks:
tasks:
- name: Validate required variables
ansible.builtin.assert:
that:
- vm_ip | length > 0
- vm_name | length > 0
fail_msg: "vm_ip and vm_name must be provided by Aether"
- ffsdn_instance_name is defined
- ffsdn_instance_name | length > 0
fail_msg: "ffsdn_instance_name must be provided by Aether"
- name: Add target host to in-memory inventory
ansible.builtin.add_host:
name: "{{ vm_ip }}"
groups: target
ansible_user: "{{ ssh_user }}"
ansible_host: "{{ vm_ip }}"
- name: Wait for instance and apply base configuration
hosts: target
gather_facts: false
pre_tasks:
- name: Wait for SSH connectivity (up to 5 minutes)
ansible.builtin.wait_for_connection:
- name: Wait for instance to be running (up to 60s)
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
method: GET
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
return_content: true
register: instance_state
until: instance_state.json.metadata.status == "Running"
retries: 12
delay: 5
timeout: 300
- name: Gather facts after connection
ansible.builtin.setup:
- name: Generate setup script
ansible.builtin.set_fact:
setup_script: |
#!/bin/bash
set -e
roles:
- role: base-config
vars:
hostname: "{{ hostvars['localhost']['vm_name'] }}"
deploy_environment: "{{ hostvars['localhost']['environment_name'] }}"
deploy_owner: "{{ hostvars['localhost']['owner'] }}"
packages: "{{ hostvars['localhost']['base_packages'] }}"
echo "=== Post-deploy configuration starting ==="
- name: Log deployment
hosts: localhost
gather_facts: false
connection: local
# Set hostname
hostname {{ ffsdn_instance_name }}
echo "{{ ffsdn_instance_name }}" > /etc/hostname
sed -i '/^127\.0\.1\.1/d' /etc/hosts
echo "127.0.1.1 {{ ffsdn_instance_name }}" >> /etc/hosts
echo "[OK] Hostname set to {{ ffsdn_instance_name }}"
# Update package cache and install base packages
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq {{ base_packages }} 2>&1 | tail -1
echo "[OK] Base packages installed"
# Set timezone
if command -v timedatectl >/dev/null 2>&1; then
timedatectl set-timezone UTC
else
ln -sf /usr/share/zoneinfo/UTC /etc/localtime
echo "UTC" > /etc/timezone
fi
echo "[OK] Timezone set to UTC"
# Configure unattended upgrades
cat > /etc/apt/apt.conf.d/20auto-upgrades << 'AUTOUPGRADE'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
AUTOUPGRADE
echo "[OK] Unattended upgrades configured"
# Write deployment metadata
cat > /etc/deploy-info << DEPLOYINFO
hostname={{ ffsdn_instance_name }}
ip={{ ffsdn_instance_ip | default('unknown') }}
cluster={{ ffsdn_cluster_name | default('unknown') }}
deployed_by={{ ffsdn_deployed_by | default('unknown') }}
image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
deployed=$(date -Iseconds)
DEPLOYINFO
echo "[OK] Deployment metadata written to /etc/deploy-info"
echo "=== Post-deploy configuration complete ==="
- name: Push setup script to instance
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/post-deploy-setup.sh"
method: POST
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
body: "{{ setup_script }}"
headers:
Content-Type: "application/octet-stream"
X-Incus-type: "file"
X-Incus-mode: "0755"
status_code: [200]
- name: Execute setup script via Incus API
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
method: POST
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
body_format: json
body:
command: ["/bin/bash", "/tmp/post-deploy-setup.sh"]
record-output: true
interactive: false
wait-for-websocket: false
environment:
DEBIAN_FRONTEND: "noninteractive"
status_code: [202]
return_content: true
register: exec_result
- name: Wait for setup script to complete (up to 5 minutes)
ansible.builtin.uri:
url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
method: GET
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
return_content: true
register: exec_wait
failed_when: exec_wait.json.metadata.status != "Success"
- name: Verify exit code
ansible.builtin.assert:
that:
- exec_wait.json.metadata.metadata.return | int == 0
fail_msg: "Setup script exited with code {{ exec_wait.json.metadata.metadata.return }}"
success_msg: "Setup completed successfully (exit code 0)"
- name: Verify deployment metadata was written
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
method: GET
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
return_content: true
register: deploy_info
- name: Display deployment metadata
ansible.builtin.debug:
msg: "{{ deploy_info.content }}"
- name: Clean up setup script
ansible.builtin.uri:
url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
method: POST
client_cert: "{{ incus_cert }}"
client_key: "{{ incus_key }}"
validate_certs: false
body_format: json
body:
command: ["/bin/rm", "-f", "/tmp/post-deploy-setup.sh"]
record-output: false
interactive: false
wait-for-websocket: false
status_code: [202]
ignore_errors: true
tasks:
- name: Record deployment in ledger
ansible.builtin.lineinfile:
path: /var/log/awx-deploy-ledger.log
path: /tmp/awx-deploy-ledger.log
line: >-
{{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }}
environment={{ environment_name }} owner={{ owner }}
cost_center={{ cost_center }} status=success
{{ lookup('pipe', 'date -Iseconds') }}
DEPLOY instance={{ ffsdn_instance_name | default('unknown') }}
ip={{ ffsdn_instance_ip | default('') }}
cluster={{ ffsdn_cluster_name | default('unknown') }}
deployed_by={{ ffsdn_deployed_by | default('unknown') }}
image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
status=success
create: true
mode: "0644"

View File

@ -1,53 +0,0 @@
---
# base-config — Minimal post-deploy configuration for new instances
#
# Variables (passed from post-deploy.yml):
# hostname — desired hostname
# deploy_environment — dev/staging/prod
# deploy_owner — owner tag
# packages — list of packages to install
- name: Set hostname
ansible.builtin.hostname:
name: "{{ hostname }}"
- name: Add hostname to /etc/hosts
ansible.builtin.lineinfile:
path: /etc/hosts
regexp: '^127\.0\.1\.1'
line: "127.0.1.1 {{ hostname }}"
- name: Update apt cache
ansible.builtin.apt:
update_cache: true
cache_valid_time: 3600
when: ansible_os_family == "Debian"
- name: Install base packages
ansible.builtin.apt:
name: "{{ packages }}"
state: present
when: ansible_os_family == "Debian"
- name: Set timezone to UTC
community.general.timezone:
name: UTC
- name: Enable unattended security upgrades
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
content: |
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
mode: "0644"
when: ansible_os_family == "Debian"
- name: Add deployment metadata to /etc/deploy-info
ansible.builtin.copy:
dest: /etc/deploy-info
content: |
hostname={{ hostname }}
environment={{ deploy_environment }}
owner={{ deploy_owner }}
deployed={{ ansible_date_time.iso8601 }}
mode: "0644"

View File

@ -212,10 +212,16 @@ vm_exec() {
incus exec "$(vm_ref)" -- "$@"
}
awx_url() {
# AWX is exposed via K3s NodePort on port 30080 (HTTP)
# Traefik ingress returns 404 for IP-based access
echo "http://${AWX_IP}:30080"
}
awx_api() {
local method="$1" path="$2"
shift 2
curl -sk -X "$method" "https://${AWX_IP}${path}" \
curl -sk -X "$method" "$(awx_url)${path}" \
-H "Content-Type: application/json" \
"$@"
}
@ -225,7 +231,7 @@ awx_api_auth() {
shift 2
local token
token=$(get_awx_admin_password)
curl -sk -X "$method" "https://${AWX_IP}${path}" \
curl -sk -X "$method" "$(awx_url)${path}" \
--user "admin:${token}" \
-H "Content-Type: application/json" \
"$@"
@ -255,7 +261,7 @@ wait_for_awx_web() {
local timeout="${1:-600}"
local elapsed=0
info "Waiting for AWX web interface (up to ${timeout}s)..."
while ! curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | grep -q '200'; do
while ! curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null | grep -q '200'; do
sleep 10
elapsed=$((elapsed + 10))
if [[ $elapsed -ge $timeout ]]; then
@ -390,8 +396,8 @@ action_deploy() {
echo
success "AWX deployment complete!"
info " Web UI: https://${AWX_IP}/"
info " API: https://${AWX_IP}/api/v2/ping/"
info " Web UI: $(awx_url)/"
info " API: $(awx_url)/api/v2/ping/"
local pw
pw=$(get_awx_admin_password 2>/dev/null || echo "<retrieve with --status>")
info " Admin: admin / ${pw}"
@ -436,23 +442,28 @@ phase_configure_network() {
return 0
fi
# Debian 12 images use systemd-networkd (not netplan)
# The macvlan NIC appears as enp5s0 in Incus VMs
vm_exec bash -c "
hostnamectl set-hostname '${VM_NAME}'
cat > /etc/netplan/50-static.yaml << 'NETPLAN'
network:
version: 2
ethernets:
enp5s0:
addresses: [${AWX_IP}/${AWX_PREFIX}]
routes:
- to: default
via: ${AWX_GATEWAY}
nameservers:
addresses: [${AWX_DNS}]
NETPLAN
chmod 600 /etc/netplan/50-static.yaml
netplan apply
# Disable any existing DHCP config
rm -f /etc/network/interfaces.d/* 2>/dev/null || true
# Configure static IP via systemd-networkd
mkdir -p /etc/systemd/network
cat > /etc/systemd/network/10-static.network << 'NETWORKD'
[Match]
Name=enp5s0
[Network]
Address=${AWX_IP}/${AWX_PREFIX}
Gateway=${AWX_GATEWAY}
DNS=${AWX_DNS}
NETWORKD
systemctl enable systemd-networkd
systemctl restart systemd-networkd
"
success "Static IP ${AWX_IP}/${AWX_PREFIX} configured"
@ -474,6 +485,14 @@ phase_install_k3s() {
return 0
fi
# Ensure curl is available (minimal Debian 12 may not have it)
info "Installing prerequisites..."
vm_exec bash -c "
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq curl python3 2>&1 | tail -1
"
vm_exec bash -c "
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
"
@ -572,9 +591,9 @@ phase_verify() {
admin_pw=$(get_awx_admin_password)
success "Admin password retrieved"
# Test API ping
# Test API ping (NodePort 30080)
local ping_result
ping_result=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null)
ping_result=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null)
if echo "$ping_result" | python3 -c "import sys,json; json.load(sys.stdin)" &>/dev/null; then
success "AWX API responding"
else
@ -629,11 +648,11 @@ action_status() {
warn "Pods: ${running}/${total} running"
fi
# AWX web
# AWX web (NodePort 30080)
local http_code
http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
if [[ "$http_code" == "200" ]]; then
success "AWX API: responding (HTTP ${http_code})"
success "AWX API: responding at $(awx_url) (HTTP ${http_code})"
else
error "AWX API: not responding (HTTP ${http_code})"
fi
@ -649,7 +668,7 @@ action_status() {
# AWX version
local version
version=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
version=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
info "AWX version: ${version}"
# Aether connectivity
@ -707,10 +726,10 @@ action_heal() {
success "All pods healthy"
fi
# 3. AWX web responsiveness
# 3. AWX web responsiveness (NodePort 30080)
info "Checking AWX web..."
local http_code
http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
if [[ "$http_code" == "200" ]]; then
success "AWX web: responding"
else
@ -763,98 +782,85 @@ action_configure() {
awx_auth() {
local method="$1" path="$2"
shift 2
curl -sk -X "$method" "https://${AWX_IP}${path}" \
curl -sk -X "$method" "$(awx_url)${path}" \
--user "admin:${admin_pw}" \
-H "Content-Type: application/json" \
"$@"
}
# 1. Create SCM credential for private Git repo
step "Creating SCM credential"
local ssh_key=""
local ssh_key_path="${HOME}/.ssh/id_ed25519"
[[ ! -f "$ssh_key_path" ]] && ssh_key_path="${HOME}/.ssh/id_rsa"
if [[ -f "$ssh_key_path" ]]; then
ssh_key=$(cat "$ssh_key_path")
info "Using SSH key: ${ssh_key_path}"
# 1. Push Incus client cert/key to AWX project directory
# Playbooks use Incus REST API (not SSH) — cert is required.
# AWX EE mounts project dir at /runner/project/ during job execution.
step "Pushing Incus client certificate to AWX"
local incus_cert="${HOME}/.config/incus/client.crt"
local incus_key="${HOME}/.config/incus/client.key"
if [[ -f "$incus_cert" && -f "$incus_key" ]]; then
local task_pod
task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
-l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
if [[ -n "$task_pod" ]]; then
local project_dir="/var/lib/awx/projects/incus-contrib"
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "$project_dir"
# Push cert via base64 to avoid file push permission issues
local cert_b64 key_b64
cert_b64=$(base64 -w0 "$incus_cert")
key_b64=$(base64 -w0 "$incus_key")
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
bash -c "echo '${cert_b64}' | base64 -d > ${project_dir}/incus-client.crt"
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
bash -c "echo '${key_b64}' | base64 -d > ${project_dir}/incus-client.key"
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
chmod 600 "${project_dir}/incus-client.key"
success "Incus client cert/key pushed to AWX task pod"
else
warn "No SSH key found at ~/.ssh/id_ed25519 or ~/.ssh/id_rsa"
warn "SCM credential will be created without a key — update manually in AWX"
warn "Cannot find AWX task pod — push cert manually later"
fi
else
warn "Incus client cert not found at ${incus_cert}"
warn "Playbooks need the cert to talk to Incus API — push it manually"
fi
local scm_cred_id
scm_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-contrib-scm" 2>/dev/null \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
if [[ -z "$scm_cred_id" ]]; then
if [[ -n "$ssh_key" ]]; then
scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-d "$(python3 -c "
import json, sys
print(json.dumps({
'name': 'incus-contrib-scm',
'organization': 1,
'credential_type': 2,
'inputs': {'ssh_key_data': sys.stdin.read()}
}))" <<< "$ssh_key")" 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
else
scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-d '{"name":"incus-contrib-scm","organization":1,"credential_type":2,"inputs":{}}' 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
fi
[[ -n "$scm_cred_id" ]] && success "SCM credential created (ID: ${scm_cred_id})" || warn "Failed to create SCM credential"
else
success "SCM credential exists (ID: ${scm_cred_id})"
fi
# 2. Create project
# 2. Create manual project (local_path)
# AWX EE containers cannot reach the private git repo, so we use a
# manual project with files pushed directly to the task pod.
step "Creating project"
local project_id
project_id=$(awx_auth GET "/api/v2/projects/?name=incus-contrib" 2>/dev/null \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
if [[ -z "$project_id" ]]; then
local project_payload
project_payload=$(python3 -c "
import json
d = {
'name': 'incus-contrib',
'organization': 1,
'scm_type': 'git',
'scm_url': '${GIT_REPO}',
'scm_branch': '${GIT_BRANCH}',
'scm_update_on_launch': True
}
print(json.dumps(d))")
[[ -n "$scm_cred_id" ]] && project_payload=$(echo "$project_payload" | python3 -c "
import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(json.dumps(d))")
project_id=$(awx_auth POST "/api/v2/projects/" -d "$project_payload" 2>/dev/null \
project_id=$(awx_auth POST "/api/v2/projects/" \
-d '{"name":"incus-contrib","organization":1,"scm_type":"","local_path":"incus-contrib"}' 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
[[ -n "$project_id" ]] && success "Project created (ID: ${project_id})" || warn "Failed to create project"
[[ -n "$project_id" ]] && success "Project created (ID: ${project_id}, local_path)" || warn "Failed to create project"
else
success "Project exists (ID: ${project_id})"
fi
# Wait for initial project sync
# Push playbooks to AWX task pod project directory
if [[ -n "$project_id" ]]; then
info "Waiting for project sync..."
local sync_elapsed=0
while [[ $sync_elapsed -lt 120 ]]; do
local status
status=$(awx_auth GET "/api/v2/projects/${project_id}/" 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || true)
if [[ "$status" == "successful" ]]; then
success "Project synced"
break
elif [[ "$status" == "failed" || "$status" == "error" ]]; then
warn "Project sync failed — check SCM credential and repo URL"
break
local task_pod
task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
-l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
if [[ -n "$task_pod" ]]; then
local project_dir="/var/lib/awx/projects/incus-contrib"
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "${project_dir}/playbooks"
for playbook in post-deploy.yml decommission.yml; do
local pb_path="${SCRIPT_DIR}/../ansible/playbooks/${playbook}"
if [[ -f "$pb_path" ]]; then
local pb_b64
pb_b64=$(base64 -w0 "$pb_path")
vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
bash -c "echo '${pb_b64}' | base64 -d > ${project_dir}/playbooks/${playbook}"
detail "Pushed ${playbook}"
else
warn "Playbook not found: ${pb_path}"
fi
sleep 5
sync_elapsed=$((sync_elapsed + 5))
done
success "Playbooks pushed to AWX project directory"
else
warn "Cannot find AWX task pod — push playbooks manually"
fi
fi
# 3. Create inventory
@ -872,26 +878,16 @@ import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(j
success "Inventory exists (ID: ${inventory_id})"
fi
# 4. Create machine credential (SSH to instances)
# 4. Create machine credential (required by AWX templates, though
# playbooks use Incus API not SSH)
step "Creating machine credential"
local machine_cred_id
machine_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-instances" 2>/dev/null \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
if [[ -z "$machine_cred_id" ]]; then
local machine_cred_payload='{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}'
if [[ -n "$ssh_key" ]]; then
machine_cred_payload=$(python3 -c "
import json, sys
key = sys.stdin.read()
print(json.dumps({
'name': 'incus-instances',
'organization': 1,
'credential_type': 1,
'inputs': {'username': 'root', 'ssh_key_data': key}
}))" <<< "$ssh_key")
fi
machine_cred_id=$(awx_auth POST "/api/v2/credentials/" -d "$machine_cred_payload" 2>/dev/null \
machine_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-d '{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}' 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
[[ -n "$machine_cred_id" ]] && success "Machine credential created (ID: ${machine_cred_id})" || warn "Failed to create machine credential"
else
@ -971,7 +967,7 @@ action_join_aether() {
admin_pw=$(get_awx_admin_password)
local pat_response
pat_response=$(curl -sk -X POST "https://${AWX_IP}/api/v2/users/1/personal_tokens/" \
pat_response=$(curl -sk -X POST "$(awx_url)/api/v2/users/1/personal_tokens/" \
--user "admin:${admin_pw}" \
-H "Content-Type: application/json" \
-d '{"description": "Aether integration", "scope": "write"}' 2>/dev/null)
@ -988,10 +984,10 @@ action_join_aether() {
# 2. Get template IDs
local post_deploy_id decommission_id
post_deploy_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=post-deploy" \
post_deploy_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=post-deploy" \
--user "admin:${admin_pw}" 2>/dev/null \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
decommission_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=decommission" \
decommission_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=decommission" \
--user "admin:${admin_pw}" 2>/dev/null \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
@ -1005,7 +1001,7 @@ action_join_aether() {
info "2. Navigate to Settings → Ansible Automation (/awx-endpoints)"
info "3. Add Endpoint:"
info " Name: lab-awx"
info " URL: https://${AWX_IP}"
info " URL: $(awx_url)"
info " Token: ${awx_token}"
info " Verify SSL: unchecked"
info "4. Configure cluster AWX binding:"
@ -1023,7 +1019,7 @@ action_join_aether() {
detail " -H 'Referer: ${AETHER_URL}/' \\"
detail " -X POST ${AETHER_URL}/api/awx/endpoints \\"
detail " -H 'Content-Type: application/json' \\"
detail " -d '{\"name\":\"lab-awx\",\"url\":\"https://${AWX_IP}\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
detail " -d '{\"name\":\"lab-awx\",\"url\":\"http://${AWX_IP}:30080\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
detail ""
detail "# Bind cluster (replace ENDPOINT_ID):"
detail "curl -sSk -b cookies.txt -H 'X-CSRF-Token: \$CSRF' \\"

File diff suppressed because it is too large Load Diff