From 0b82b9ce3b3137b61e3ee722e0322038231f6e7c Mon Sep 17 00:00:00 2001
From: Maarten <maarten@lambda-m.nl>
Date: Tue, 24 Feb 2026 01:53:54 +0100
Subject: [PATCH] Validate AWX guide against real infrastructure, fix all bugs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

End-to-end tested: Aether deploy triggers AWX post-deploy job,
container gets configured (hostname, packages, timezone, deploy-info),
decommission job runs on delete. All playbooks validated working.

Key fixes from testing:
- Playbooks: use Incus REST API (not SSH) — bridge subnet not routable
  from management VLAN, nftables blocks inbound forwarding
- Aether vars: ffsdn_* prefix (not vm_name/vm_ip/environment/owner)
- AWX EE paths: /runner/project/ (not /var/lib/awx/projects/)
- AWX access: NodePort 30080 (not Traefik ingress — 404 for IP access)
- deploy-awx: systemd-networkd (not netplan), manual project (not git
  SCM — EE can't reach private repo), Incus cert push, prereq install
- Remove unused base-config role (tasks inlined as setup script)
- Simplify ansible.cfg (no SSH settings needed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 CLAUDE.md                                |  43 +-
 README.md                                |   4 +-
 ansible/ansible.cfg                      |  17 +-
 ansible/playbooks/decommission.yml       | 111 ++--
 ansible/playbooks/post-deploy.yml        | 236 +++++--
 ansible/roles/base-config/tasks/main.yml |  53 --
 incusos/deploy-awx                       | 228 ++++---
 notes/awx-guide.md                       | 778 ++++++++++++++---------
 8 files changed, 837 insertions(+), 633 deletions(-)
 delete mode 100644 ansible/roles/base-config/tasks/main.yml

diff --git a/CLAUDE.md b/CLAUDE.md
index 67035b4..c3ff078 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -15,12 +15,9 @@ incu-contrib/
 ├── .gitignore
 ├── ansible/               # Ansible playbooks for AWX/Aether lifecycle hooks
 │   ├── ansible.cfg        # Project-level Ansible config
-│   ├── playbooks/
-│   │   ├── post-deploy.yml    # Runs after Aether creates an instance
-│   │   └── decommission.yml   # Runs before Aether deletes an instance
-│   └── roles/
-│       └── base-config/       # Base OS configuration role
-│           └── tasks/main.yml
+│   └── playbooks/
+│       ├── post-deploy.yml    # Runs after Aether creates an instance (Incus API)
+│       └── decommission.yml   # Runs before Aether deletes an instance
 ├── incusos/               # IncusOS installation tooling
 │   ├── README.md          # Detailed usage docs
 │   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
@@ -826,22 +823,40 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
   Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
 - **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
   to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
+- **AWX URL**: `http://192.168.102.161:30080` — exposed via K3s NodePort.
+  Traefik ingress returns 404 for IP-based access; use NodePort directly.
 - **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
   `--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
 - **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
 - **Ansible playbooks** in `ansible/` directory:
   - `playbooks/post-deploy.yml` -- runs after Aether creates an instance
   - `playbooks/decommission.yml` -- runs before Aether deletes an instance
-  - `roles/base-config/` -- hostname, packages, timezone, deploy metadata
-- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with
-  a Personal Access Token. Per-cluster config binds post-deploy and
-  decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/`
-  with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`).
+- **Aether extra vars use `ffsdn_` prefix**: Aether passes `ffsdn_instance_name`,
+  `ffsdn_instance_ip`, `ffsdn_cluster_id`, `ffsdn_cluster_name`,
+  `ffsdn_deployed_by`, `ffsdn_image_os`, `ffsdn_image_release`,
+  `ffsdn_image_alias`. It does NOT pass `vm_name`, `vm_ip`, `environment`,
+  `owner`, or `cost_center` (the original plan assumed these).
+- **Playbook pattern — Incus REST API, not SSH**: AWX cannot SSH to containers
+  on incusbr0 (bridge subnet not routable from management VLAN, IncusOS
+  nftables blocks inbound forwarding). Playbooks use the Incus REST API
+  (`uri` module with client cert) for file push + exec. The cert is at
+  `/runner/project/incus-client.crt` during job execution (AWX EE mounts
+  projects at `/runner/project/`, NOT `/var/lib/awx/projects/`).
+- **Manual project (local_path)**: AWX EE containers cannot reach the private
+  git repo. Use `scm_type: ""` with `local_path: "incus-contrib"` and push
+  playbooks directly to the AWX task pod at
+  `/var/lib/awx/projects/incus-contrib/playbooks/`.
 - **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
   deleted). Decommission failure does NOT block deletion.
-- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to
-  create dynamic inventory → SSH to target for configuration. External system
-  tasks use `delegate_to: localhost`.
+- **Aether cluster AWX config API bug**: `PUT /api/clusters/{id}/awx-config`
+  returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL
+  UPDATE on the `clusters` table from within the Aether container.
+- **Self-referencing vars cause infinite recursion in Ansible**: patterns like
+  `vm_ip: "{{ vm_ip | default('') }}"` cause `AnsibleUndefinedVariable`
+  recursive loop when the variable is not provided. Use `ffsdn_*` vars
+  directly without redefining them.
+- **Ansible `environment` is a reserved keyword**: using it as an extra var
+  resolves to `[]` instead of the string value.
 - See `notes/awx-guide.md` for full deployment guide and troubleshooting.
 
 ## Git workflow
diff --git a/README.md b/README.md
index 389a22f..299c144 100644
--- a/README.md
+++ b/README.md
@@ -31,13 +31,13 @@ See [`incusos/README.md`](incusos/README.md) for full documentation.
 
 ### [`ansible/`](ansible/)
 
-Ansible playbooks and roles for AWX/Aether lifecycle hooks:
+Ansible playbooks for AWX/Aether lifecycle hooks. Playbooks use the Incus REST
+API (not SSH) for all instance operations — works regardless of network topology.
 
 - **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance.
   Configures hostname, installs base packages, sets timezone, logs deployment.
 - **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
   Graceful service shutdown and decommission logging (best-effort).
-- **`roles/base-config/`** -- Base OS configuration role used by post-deploy.
 
 ### [`notes/`](notes/)
 
diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg
index 1a62528..5523a00 100644
--- a/ansible/ansible.cfg
+++ b/ansible/ansible.cfg
@@ -2,10 +2,7 @@
 # Inventory — AWX manages inventory; this is for local testing only
 inventory = localhost,
 
-# Roles path
-roles_path = roles
-
-# Reduce SSH overhead
+# Reduce overhead
 forks = 10
 timeout = 30
 
@@ -18,11 +15,7 @@ stdout_callback = yaml
 # Host key checking — disabled for dynamic lab instances
 host_key_checking = False
 
-[privilege_escalation]
-become = True
-become_method = sudo
-
-[ssh_connection]
-# Speed up SSH connections
-pipelining = True
-ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no
+# Note: playbooks use the Incus REST API (uri module with client cert)
+# for all instance operations — no SSH connection to instances is needed.
+# The [ssh_connection] and [privilege_escalation] sections are therefore
+# not required.
diff --git a/ansible/playbooks/decommission.yml b/ansible/playbooks/decommission.yml
index 91f0813..65faa38 100644
--- a/ansible/playbooks/decommission.yml
+++ b/ansible/playbooks/decommission.yml
@@ -1,78 +1,77 @@
 ---
 # decommission.yml — Runs before Aether deletes an instance
 #
-# Aether passes the same extra vars as post-deploy:
-#   vm_name, vm_ip, environment, owner, cost_center
+# Aether passes the same ffsdn_ extra vars as post-deploy:
+#   ffsdn_instance_name, ffsdn_instance_ip, ffsdn_cluster_id,
+#   ffsdn_cluster_name, ffsdn_deployed_by
 #
 # This playbook runs best-effort: failures do NOT block instance
-# deletion in Aether. The playbook should be defensive and handle
-# unreachable hosts gracefully.
+# deletion in Aether. Uses the Incus REST API for all operations
+# (no SSH needed).
 
-- name: Decommission — prepare instance for deletion
+- name: Decommission — graceful shutdown and logging
   hosts: localhost
   gather_facts: false
   connection: local
 
   vars:
-    vm_name: "{{ vm_name | default('unknown') }}"
-    vm_ip: "{{ vm_ip | default('') }}"
-    environment_name: "{{ environment | default('lab') }}"
-    owner: "{{ owner | default('unassigned') }}"
-    cost_center: "{{ cost_center | default('') }}"
-    ssh_user: root
+    incus_api: "https://192.168.102.140:8443"
+    # AWX runner mounts project at /runner/project/ during job execution
+    incus_cert: "/runner/project/incus-client.crt"
+    incus_key: "/runner/project/incus-client.key"
 
   tasks:
-    - name: Add target host to in-memory inventory
-      ansible.builtin.add_host:
-        name: "{{ vm_ip }}"
-        groups: target
-        ansible_user: "{{ ssh_user }}"
-        ansible_host: "{{ vm_ip }}"
-      when: vm_ip | length > 0
-
-- name: Attempt graceful shutdown on target
-  hosts: target
-  gather_facts: false
-  ignore_unreachable: true
-
-  tasks:
-    - name: Check if host is reachable
-      ansible.builtin.wait_for_connection:
-        delay: 0
-        timeout: 15
-      register: host_check
-      ignore_errors: true
+    - name: Check if instance still exists
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+        status_code: [200, 404]
+      register: instance_check
+      when: ffsdn_instance_name is defined
 
     - name: Stop application services (best-effort)
-      ansible.builtin.shell: |
-        # Stop user-facing services before deletion
-        systemctl list-units --type=service --state=running --no-legend \
-          | awk '{print $1}' \
-          | grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
-          | head -20 \
-          | xargs -r -n1 systemctl stop
-      register: service_stop
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command:
+            - "/bin/bash"
+            - "-c"
+            - |
+              systemctl list-units --type=service --state=running --no-legend \
+                | awk '{print $1}' \
+                | grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
+                | head -20 \
+                | xargs -r -n1 systemctl stop 2>/dev/null || true
+              journalctl --flush 2>/dev/null || true
+          record-output: false
+          interactive: false
+          wait-for-websocket: false
+        status_code: [202]
       ignore_errors: true
-      when: host_check is succeeded
+      when:
+        - instance_check is defined
+        - instance_check.status == 200
+        - instance_check.json.metadata.status is defined
+        - instance_check.json.metadata.status == "Running"
 
-    - name: Flush logs to persistent storage
-      ansible.builtin.command: journalctl --flush
-      ignore_errors: true
-      when: host_check is succeeded
-
-- name: Log decommission
-  hosts: localhost
-  gather_facts: false
-  connection: local
-
-  tasks:
     - name: Record decommission in ledger
       ansible.builtin.lineinfile:
-        path: /var/log/awx-deploy-ledger.log
+        path: /tmp/awx-deploy-ledger.log
         line: >-
-          {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
-          DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }}
-          environment={{ environment_name }} owner={{ owner }}
-          cost_center={{ cost_center }} status=completed
+          {{ lookup('pipe', 'date -Iseconds') }}
+          DECOMMISSION instance={{ ffsdn_instance_name | default('unknown') }}
+          ip={{ ffsdn_instance_ip | default('') }}
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          status=completed
         create: true
         mode: "0644"
diff --git a/ansible/playbooks/post-deploy.yml b/ansible/playbooks/post-deploy.yml
index bcbe505..b42181c 100644
--- a/ansible/playbooks/post-deploy.yml
+++ b/ansible/playbooks/post-deploy.yml
@@ -1,88 +1,200 @@
 ---
 # post-deploy.yml — Runs after Aether creates an instance
 #
-# Aether passes extra vars:
-#   vm_name       — instance hostname
-#   vm_ip         — IP address assigned by Aether
-#   environment   — deployment environment (dev, staging, prod)
-#   owner         — VM owner or team
-#   cost_center   — billing/cost allocation
-#   (plus any custom tags from the blueprint)
+# Aether passes these extra vars (ffsdn_ prefix):
+#   ffsdn_instance_name  — instance name
+#   ffsdn_instance_ip    — IP address assigned by Aether
+#   ffsdn_cluster_id     — numeric cluster ID
+#   ffsdn_cluster_name   — cluster display name
+#   ffsdn_deployed_by    — Aether username who triggered the deploy
+#   ffsdn_image_os       — image OS (e.g., "Debian")
+#   ffsdn_image_release  — image release (e.g., "bookworm")
+#   ffsdn_image_alias    — image alias (may be empty)
 #
-# AWX job template must have "ask_variables_on_launch: true" so Aether
-# can inject these variables at runtime.
+# This playbook configures new instances via the Incus REST API
+# (file push + exec), which works regardless of network topology.
+# AWX does NOT need SSH access to the container — it talks to the
+# Incus cluster API at the node level.
+#
+# Requirements:
+#   - Incus client cert/key at /var/lib/awx/projects/incus-contrib/
+#   - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443)
 
-- name: Post-deploy — configure newly provisioned instance
+- name: Post-deploy — configure instance via Incus API
   hosts: localhost
   gather_facts: false
   connection: local
 
   vars:
-    # Defaults (overridden by Aether extra vars)
-    vm_name: "{{ vm_name | default('unknown') }}"
-    vm_ip: "{{ vm_ip | default('') }}"
-    environment_name: "{{ environment | default('lab') }}"
-    owner: "{{ owner | default('unassigned') }}"
-    cost_center: "{{ cost_center | default('') }}"
-    ssh_user: root
-    base_packages:
-      - curl
-      - vim
-      - htop
-      - jq
-      - tmux
-      - rsync
-      - unattended-upgrades
+    # Incus cluster API (any cluster member works)
+    incus_api: "https://192.168.102.140:8443"
+    # AWX runner mounts project at /runner/project/ during job execution
+    incus_cert: "/runner/project/incus-client.crt"
+    incus_key: "/runner/project/incus-client.key"
+    base_packages: "curl vim htop jq tmux rsync unattended-upgrades"
 
-  pre_tasks:
+  tasks:
     - name: Validate required variables
       ansible.builtin.assert:
         that:
-          - vm_ip | length > 0
-          - vm_name | length > 0
-        fail_msg: "vm_ip and vm_name must be provided by Aether"
+          - ffsdn_instance_name is defined
+          - ffsdn_instance_name | length > 0
+        fail_msg: "ffsdn_instance_name must be provided by Aether"
 
-    - name: Add target host to in-memory inventory
-      ansible.builtin.add_host:
-        name: "{{ vm_ip }}"
-        groups: target
-        ansible_user: "{{ ssh_user }}"
-        ansible_host: "{{ vm_ip }}"
+    - name: Wait for instance to be running (up to 60s)
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: instance_state
+      until: instance_state.json.metadata.status == "Running"
+      retries: 12
+      delay: 5
 
-- name: Wait for instance and apply base configuration
-  hosts: target
-  gather_facts: false
+    - name: Generate setup script
+      ansible.builtin.set_fact:
+        setup_script: |
+          #!/bin/bash
+          set -e
 
-  pre_tasks:
-    - name: Wait for SSH connectivity (up to 5 minutes)
-      ansible.builtin.wait_for_connection:
-        delay: 5
-        timeout: 300
+          echo "=== Post-deploy configuration starting ==="
 
-    - name: Gather facts after connection
-      ansible.builtin.setup:
+          # Set hostname
+          hostname {{ ffsdn_instance_name }}
+          echo "{{ ffsdn_instance_name }}" > /etc/hostname
+          sed -i '/^127\.0\.1\.1/d' /etc/hosts
+          echo "127.0.1.1 {{ ffsdn_instance_name }}" >> /etc/hosts
+          echo "[OK] Hostname set to {{ ffsdn_instance_name }}"
 
-  roles:
-    - role: base-config
-      vars:
-        hostname: "{{ hostvars['localhost']['vm_name'] }}"
-        deploy_environment: "{{ hostvars['localhost']['environment_name'] }}"
-        deploy_owner: "{{ hostvars['localhost']['owner'] }}"
-        packages: "{{ hostvars['localhost']['base_packages'] }}"
+          # Update package cache and install base packages
+          export DEBIAN_FRONTEND=noninteractive
+          apt-get update -qq
+          apt-get install -y -qq {{ base_packages }} 2>&1 | tail -1
+          echo "[OK] Base packages installed"
 
-- name: Log deployment
-  hosts: localhost
-  gather_facts: false
-  connection: local
+          # Set timezone
+          if command -v timedatectl >/dev/null 2>&1; then
+            timedatectl set-timezone UTC
+          else
+            ln -sf /usr/share/zoneinfo/UTC /etc/localtime
+            echo "UTC" > /etc/timezone
+          fi
+          echo "[OK] Timezone set to UTC"
+
+          # Configure unattended upgrades
+          cat > /etc/apt/apt.conf.d/20auto-upgrades << 'AUTOUPGRADE'
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          AUTOUPGRADE
+          echo "[OK] Unattended upgrades configured"
+
+          # Write deployment metadata
+          cat > /etc/deploy-info << DEPLOYINFO
+          hostname={{ ffsdn_instance_name }}
+          ip={{ ffsdn_instance_ip | default('unknown') }}
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
+          deployed=$(date -Iseconds)
+          DEPLOYINFO
+          echo "[OK] Deployment metadata written to /etc/deploy-info"
+
+          echo "=== Post-deploy configuration complete ==="
+
+    - name: Push setup script to instance
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/post-deploy-setup.sh"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body: "{{ setup_script }}"
+        headers:
+          Content-Type: "application/octet-stream"
+          X-Incus-type: "file"
+          X-Incus-mode: "0755"
+        status_code: [200]
+
+    - name: Execute setup script via Incus API
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command: ["/bin/bash", "/tmp/post-deploy-setup.sh"]
+          record-output: true
+          interactive: false
+          wait-for-websocket: false
+          environment:
+            DEBIAN_FRONTEND: "noninteractive"
+        status_code: [202]
+        return_content: true
+      register: exec_result
+
+    - name: Wait for setup script to complete (up to 5 minutes)
+      ansible.builtin.uri:
+        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: exec_wait
+      failed_when: exec_wait.json.metadata.status != "Success"
+
+    - name: Verify exit code
+      ansible.builtin.assert:
+        that:
+          - exec_wait.json.metadata.metadata.return | int == 0
+        fail_msg: "Setup script exited with code {{ exec_wait.json.metadata.metadata.return }}"
+        success_msg: "Setup completed successfully (exit code 0)"
+
+    - name: Verify deployment metadata was written
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: deploy_info
+
+    - name: Display deployment metadata
+      ansible.builtin.debug:
+        msg: "{{ deploy_info.content }}"
+
+    - name: Clean up setup script
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command: ["/bin/rm", "-f", "/tmp/post-deploy-setup.sh"]
+          record-output: false
+          interactive: false
+          wait-for-websocket: false
+        status_code: [202]
+      ignore_errors: true
 
-  tasks:
     - name: Record deployment in ledger
       ansible.builtin.lineinfile:
-        path: /var/log/awx-deploy-ledger.log
+        path: /tmp/awx-deploy-ledger.log
         line: >-
-          {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
-          DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }}
-          environment={{ environment_name }} owner={{ owner }}
-          cost_center={{ cost_center }} status=success
+          {{ lookup('pipe', 'date -Iseconds') }}
+          DEPLOY instance={{ ffsdn_instance_name | default('unknown') }}
+          ip={{ ffsdn_instance_ip | default('') }}
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
+          status=success
         create: true
         mode: "0644"
diff --git a/ansible/roles/base-config/tasks/main.yml b/ansible/roles/base-config/tasks/main.yml
deleted file mode 100644
index 1519eed..0000000
--- a/ansible/roles/base-config/tasks/main.yml
+++ /dev/null
@@ -1,53 +0,0 @@
----
-# base-config — Minimal post-deploy configuration for new instances
-#
-# Variables (passed from post-deploy.yml):
-#   hostname          — desired hostname
-#   deploy_environment — dev/staging/prod
-#   deploy_owner      — owner tag
-#   packages          — list of packages to install
-
-- name: Set hostname
-  ansible.builtin.hostname:
-    name: "{{ hostname }}"
-
-- name: Add hostname to /etc/hosts
-  ansible.builtin.lineinfile:
-    path: /etc/hosts
-    regexp: '^127\.0\.1\.1'
-    line: "127.0.1.1 {{ hostname }}"
-
-- name: Update apt cache
-  ansible.builtin.apt:
-    update_cache: true
-    cache_valid_time: 3600
-  when: ansible_os_family == "Debian"
-
-- name: Install base packages
-  ansible.builtin.apt:
-    name: "{{ packages }}"
-    state: present
-  when: ansible_os_family == "Debian"
-
-- name: Set timezone to UTC
-  community.general.timezone:
-    name: UTC
-
-- name: Enable unattended security upgrades
-  ansible.builtin.copy:
-    dest: /etc/apt/apt.conf.d/20auto-upgrades
-    content: |
-      APT::Periodic::Update-Package-Lists "1";
-      APT::Periodic::Unattended-Upgrade "1";
-    mode: "0644"
-  when: ansible_os_family == "Debian"
-
-- name: Add deployment metadata to /etc/deploy-info
-  ansible.builtin.copy:
-    dest: /etc/deploy-info
-    content: |
-      hostname={{ hostname }}
-      environment={{ deploy_environment }}
-      owner={{ deploy_owner }}
-      deployed={{ ansible_date_time.iso8601 }}
-    mode: "0644"
diff --git a/incusos/deploy-awx b/incusos/deploy-awx
index 510397f..37e3613 100755
--- a/incusos/deploy-awx
+++ b/incusos/deploy-awx
@@ -212,10 +212,16 @@ vm_exec() {
     incus exec "$(vm_ref)" -- "$@"
 }
 
+awx_url() {
+    # AWX is exposed via K3s NodePort on port 30080 (HTTP)
+    # Traefik ingress returns 404 for IP-based access
+    echo "http://${AWX_IP}:30080"
+}
+
 awx_api() {
     local method="$1" path="$2"
     shift 2
-    curl -sk -X "$method" "https://${AWX_IP}${path}" \
+    curl -sk -X "$method" "$(awx_url)${path}" \
         -H "Content-Type: application/json" \
         "$@"
 }
@@ -225,7 +231,7 @@ awx_api_auth() {
     shift 2
     local token
     token=$(get_awx_admin_password)
-    curl -sk -X "$method" "https://${AWX_IP}${path}" \
+    curl -sk -X "$method" "$(awx_url)${path}" \
         --user "admin:${token}" \
         -H "Content-Type: application/json" \
         "$@"
@@ -255,7 +261,7 @@ wait_for_awx_web() {
     local timeout="${1:-600}"
     local elapsed=0
     info "Waiting for AWX web interface (up to ${timeout}s)..."
-    while ! curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | grep -q '200'; do
+    while ! curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null | grep -q '200'; do
         sleep 10
         elapsed=$((elapsed + 10))
         if [[ $elapsed -ge $timeout ]]; then
@@ -390,8 +396,8 @@ action_deploy() {
 
     echo
     success "AWX deployment complete!"
-    info "  Web UI:   https://${AWX_IP}/"
-    info "  API:      https://${AWX_IP}/api/v2/ping/"
+    info "  Web UI:   $(awx_url)/"
+    info "  API:      $(awx_url)/api/v2/ping/"
     local pw
     pw=$(get_awx_admin_password 2>/dev/null || echo "<retrieve with --status>")
     info "  Admin:    admin / ${pw}"
@@ -436,23 +442,28 @@ phase_configure_network() {
         return 0
     fi
 
+    # Debian 12 images use systemd-networkd (not netplan)
+    # The macvlan NIC appears as enp5s0 in Incus VMs
     vm_exec bash -c "
         hostnamectl set-hostname '${VM_NAME}'
 
-        cat > /etc/netplan/50-static.yaml << 'NETPLAN'
-network:
-  version: 2
-  ethernets:
-    enp5s0:
-      addresses: [${AWX_IP}/${AWX_PREFIX}]
-      routes:
-        - to: default
-          via: ${AWX_GATEWAY}
-      nameservers:
-        addresses: [${AWX_DNS}]
-NETPLAN
-        chmod 600 /etc/netplan/50-static.yaml
-        netplan apply
+        # Disable any existing DHCP config
+        rm -f /etc/network/interfaces.d/* 2>/dev/null || true
+
+        # Configure static IP via systemd-networkd
+        mkdir -p /etc/systemd/network
+        cat > /etc/systemd/network/10-static.network << 'NETWORKD'
+[Match]
+Name=enp5s0
+
+[Network]
+Address=${AWX_IP}/${AWX_PREFIX}
+Gateway=${AWX_GATEWAY}
+DNS=${AWX_DNS}
+NETWORKD
+
+        systemctl enable systemd-networkd
+        systemctl restart systemd-networkd
     "
 
     success "Static IP ${AWX_IP}/${AWX_PREFIX} configured"
@@ -474,6 +485,14 @@ phase_install_k3s() {
         return 0
     fi
 
+    # Ensure curl is available (minimal Debian 12 may not have it)
+    info "Installing prerequisites..."
+    vm_exec bash -c "
+        export DEBIAN_FRONTEND=noninteractive
+        apt-get update -qq
+        apt-get install -y -qq curl python3 2>&1 | tail -1
+    "
+
     vm_exec bash -c "
         curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
     "
@@ -572,9 +591,9 @@ phase_verify() {
     admin_pw=$(get_awx_admin_password)
     success "Admin password retrieved"
 
-    # Test API ping
+    # Test API ping (NodePort 30080)
     local ping_result
-    ping_result=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null)
+    ping_result=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null)
     if echo "$ping_result" | python3 -c "import sys,json; json.load(sys.stdin)" &>/dev/null; then
         success "AWX API responding"
     else
@@ -629,11 +648,11 @@ action_status() {
         warn "Pods: ${running}/${total} running"
     fi
 
-    # AWX web
+    # AWX web (NodePort 30080)
     local http_code
-    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
+    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
     if [[ "$http_code" == "200" ]]; then
-        success "AWX API: responding (HTTP ${http_code})"
+        success "AWX API: responding at $(awx_url) (HTTP ${http_code})"
     else
         error "AWX API: not responding (HTTP ${http_code})"
     fi
@@ -649,7 +668,7 @@ action_status() {
 
     # AWX version
     local version
-    version=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
+    version=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
     info "AWX version: ${version}"
 
     # Aether connectivity
@@ -707,10 +726,10 @@ action_heal() {
         success "All pods healthy"
     fi
 
-    # 3. AWX web responsiveness
+    # 3. AWX web responsiveness (NodePort 30080)
     info "Checking AWX web..."
     local http_code
-    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
+    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
     if [[ "$http_code" == "200" ]]; then
         success "AWX web: responding"
     else
@@ -763,98 +782,85 @@ action_configure() {
     awx_auth() {
         local method="$1" path="$2"
         shift 2
-        curl -sk -X "$method" "https://${AWX_IP}${path}" \
+        curl -sk -X "$method" "$(awx_url)${path}" \
             --user "admin:${admin_pw}" \
             -H "Content-Type: application/json" \
             "$@"
     }
 
-    # 1. Create SCM credential for private Git repo
-    step "Creating SCM credential"
-    local ssh_key=""
-    local ssh_key_path="${HOME}/.ssh/id_ed25519"
-    [[ ! -f "$ssh_key_path" ]] && ssh_key_path="${HOME}/.ssh/id_rsa"
-    if [[ -f "$ssh_key_path" ]]; then
-        ssh_key=$(cat "$ssh_key_path")
-        info "Using SSH key: ${ssh_key_path}"
-    else
-        warn "No SSH key found at ~/.ssh/id_ed25519 or ~/.ssh/id_rsa"
-        warn "SCM credential will be created without a key — update manually in AWX"
-    fi
-
-    local scm_cred_id
-    scm_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-contrib-scm" 2>/dev/null \
-        | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
-
-    if [[ -z "$scm_cred_id" ]]; then
-        if [[ -n "$ssh_key" ]]; then
-            scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-                -d "$(python3 -c "
-import json, sys
-print(json.dumps({
-    'name': 'incus-contrib-scm',
-    'organization': 1,
-    'credential_type': 2,
-    'inputs': {'ssh_key_data': sys.stdin.read()}
-}))" <<< "$ssh_key")" 2>/dev/null \
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
+    # 1. Push Incus client cert/key to AWX project directory
+    #    Playbooks use Incus REST API (not SSH) — cert is required.
+    #    AWX EE mounts project dir at /runner/project/ during job execution.
+    step "Pushing Incus client certificate to AWX"
+    local incus_cert="${HOME}/.config/incus/client.crt"
+    local incus_key="${HOME}/.config/incus/client.key"
+    if [[ -f "$incus_cert" && -f "$incus_key" ]]; then
+        local task_pod
+        task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
+            -l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
+        if [[ -n "$task_pod" ]]; then
+            local project_dir="/var/lib/awx/projects/incus-contrib"
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "$project_dir"
+            # Push cert via base64 to avoid file push permission issues
+            local cert_b64 key_b64
+            cert_b64=$(base64 -w0 "$incus_cert")
+            key_b64=$(base64 -w0 "$incus_key")
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
+                bash -c "echo '${cert_b64}' | base64 -d > ${project_dir}/incus-client.crt"
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
+                bash -c "echo '${key_b64}' | base64 -d > ${project_dir}/incus-client.key"
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
+                chmod 600 "${project_dir}/incus-client.key"
+            success "Incus client cert/key pushed to AWX task pod"
         else
-            scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-                -d '{"name":"incus-contrib-scm","organization":1,"credential_type":2,"inputs":{}}' 2>/dev/null \
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
+            warn "Cannot find AWX task pod — push cert manually later"
         fi
-        [[ -n "$scm_cred_id" ]] && success "SCM credential created (ID: ${scm_cred_id})" || warn "Failed to create SCM credential"
     else
-        success "SCM credential exists (ID: ${scm_cred_id})"
+        warn "Incus client cert not found at ${incus_cert}"
+        warn "Playbooks need the cert to talk to Incus API — push it manually"
     fi
 
-    # 2. Create project
+    # 2. Create manual project (local_path)
+    #    AWX EE containers cannot reach the private git repo, so we use a
+    #    manual project with files pushed directly to the task pod.
     step "Creating project"
     local project_id
     project_id=$(awx_auth GET "/api/v2/projects/?name=incus-contrib" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
     if [[ -z "$project_id" ]]; then
-        local project_payload
-        project_payload=$(python3 -c "
-import json
-d = {
-    'name': 'incus-contrib',
-    'organization': 1,
-    'scm_type': 'git',
-    'scm_url': '${GIT_REPO}',
-    'scm_branch': '${GIT_BRANCH}',
-    'scm_update_on_launch': True
-}
-print(json.dumps(d))")
-        [[ -n "$scm_cred_id" ]] && project_payload=$(echo "$project_payload" | python3 -c "
-import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(json.dumps(d))")
-
-        project_id=$(awx_auth POST "/api/v2/projects/" -d "$project_payload" 2>/dev/null \
+        project_id=$(awx_auth POST "/api/v2/projects/" \
+            -d '{"name":"incus-contrib","organization":1,"scm_type":"","local_path":"incus-contrib"}' 2>/dev/null \
             | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
-        [[ -n "$project_id" ]] && success "Project created (ID: ${project_id})" || warn "Failed to create project"
+        [[ -n "$project_id" ]] && success "Project created (ID: ${project_id}, local_path)" || warn "Failed to create project"
     else
         success "Project exists (ID: ${project_id})"
     fi
 
-    # Wait for initial project sync
+    # Push playbooks to AWX task pod project directory
     if [[ -n "$project_id" ]]; then
-        info "Waiting for project sync..."
-        local sync_elapsed=0
-        while [[ $sync_elapsed -lt 120 ]]; do
-            local status
-            status=$(awx_auth GET "/api/v2/projects/${project_id}/" 2>/dev/null \
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || true)
-            if [[ "$status" == "successful" ]]; then
-                success "Project synced"
-                break
-            elif [[ "$status" == "failed" || "$status" == "error" ]]; then
-                warn "Project sync failed — check SCM credential and repo URL"
-                break
-            fi
-            sleep 5
-            sync_elapsed=$((sync_elapsed + 5))
-        done
+        local task_pod
+        task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
+            -l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
+        if [[ -n "$task_pod" ]]; then
+            local project_dir="/var/lib/awx/projects/incus-contrib"
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "${project_dir}/playbooks"
+            for playbook in post-deploy.yml decommission.yml; do
+                local pb_path="${SCRIPT_DIR}/../ansible/playbooks/${playbook}"
+                if [[ -f "$pb_path" ]]; then
+                    local pb_b64
+                    pb_b64=$(base64 -w0 "$pb_path")
+                    vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
+                        bash -c "echo '${pb_b64}' | base64 -d > ${project_dir}/playbooks/${playbook}"
+                    detail "Pushed ${playbook}"
+                else
+                    warn "Playbook not found: ${pb_path}"
+                fi
+            done
+            success "Playbooks pushed to AWX project directory"
+        else
+            warn "Cannot find AWX task pod — push playbooks manually"
+        fi
     fi
 
     # 3. Create inventory
@@ -872,26 +878,16 @@ import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(j
         success "Inventory exists (ID: ${inventory_id})"
     fi
 
-    # 4. Create machine credential (SSH to instances)
+    # 4. Create machine credential (required by AWX templates, though
+    #    playbooks use Incus API not SSH)
     step "Creating machine credential"
     local machine_cred_id
     machine_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-instances" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
     if [[ -z "$machine_cred_id" ]]; then
-        local machine_cred_payload='{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}'
-        if [[ -n "$ssh_key" ]]; then
-            machine_cred_payload=$(python3 -c "
-import json, sys
-key = sys.stdin.read()
-print(json.dumps({
-    'name': 'incus-instances',
-    'organization': 1,
-    'credential_type': 1,
-    'inputs': {'username': 'root', 'ssh_key_data': key}
-}))" <<< "$ssh_key")
-        fi
-        machine_cred_id=$(awx_auth POST "/api/v2/credentials/" -d "$machine_cred_payload" 2>/dev/null \
+        machine_cred_id=$(awx_auth POST "/api/v2/credentials/" \
+            -d '{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}' 2>/dev/null \
             | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
         [[ -n "$machine_cred_id" ]] && success "Machine credential created (ID: ${machine_cred_id})" || warn "Failed to create machine credential"
     else
@@ -971,7 +967,7 @@ action_join_aether() {
     admin_pw=$(get_awx_admin_password)
 
     local pat_response
-    pat_response=$(curl -sk -X POST "https://${AWX_IP}/api/v2/users/1/personal_tokens/" \
+    pat_response=$(curl -sk -X POST "$(awx_url)/api/v2/users/1/personal_tokens/" \
         --user "admin:${admin_pw}" \
         -H "Content-Type: application/json" \
         -d '{"description": "Aether integration", "scope": "write"}' 2>/dev/null)
@@ -988,10 +984,10 @@ action_join_aether() {
 
     # 2. Get template IDs
     local post_deploy_id decommission_id
-    post_deploy_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=post-deploy" \
+    post_deploy_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=post-deploy" \
         --user "admin:${admin_pw}" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
-    decommission_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=decommission" \
+    decommission_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=decommission" \
         --user "admin:${admin_pw}" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
@@ -1005,7 +1001,7 @@ action_join_aether() {
     info "2. Navigate to Settings → Ansible Automation (/awx-endpoints)"
     info "3. Add Endpoint:"
     info "     Name:       lab-awx"
-    info "     URL:        https://${AWX_IP}"
+    info "     URL:        $(awx_url)"
     info "     Token:      ${awx_token}"
     info "     Verify SSL: unchecked"
     info "4. Configure cluster AWX binding:"
@@ -1023,7 +1019,7 @@ action_join_aether() {
         detail "  -H 'Referer: ${AETHER_URL}/' \\"
         detail "  -X POST ${AETHER_URL}/api/awx/endpoints \\"
         detail "  -H 'Content-Type: application/json' \\"
-        detail "  -d '{\"name\":\"lab-awx\",\"url\":\"https://${AWX_IP}\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
+        detail "  -d '{\"name\":\"lab-awx\",\"url\":\"http://${AWX_IP}:30080\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
         detail ""
         detail "# Bind cluster (replace ENDPOINT_ID):"
         detail "curl -sSk -b cookies.txt -H 'X-CSRF-Token: \$CSRF' \\"
diff --git a/notes/awx-guide.md b/notes/awx-guide.md
index ea6af8c..d52611d 100644
--- a/notes/awx-guide.md
+++ b/notes/awx-guide.md
@@ -6,54 +6,60 @@ post-deploy and decommission playbooks as lifecycle hooks on every instance
 create/delete.
 
 This guide covers deploying AWX on the Incus cluster, connecting it to
-Aether, writing lifecycle playbooks, and using the `deploy-awx` management
-script.
+Aether, writing lifecycle playbooks, and troubleshooting.
 
 ## Architecture
 
 ```
-  User deploys VM via Aether
+  User deploys instance via Aether UI
           |
           v
   Aether creates instance on Incus cluster
           |
           v
   Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
-    with extra_vars: vm_name, vm_ip, environment, owner, cost_center
+    with extra_vars: ffsdn_instance_name, ffsdn_instance_ip, ...
           |
           v
   AWX runs post-deploy.yml playbook
-    1. SSH to new instance (vm_ip)
-    2. Set hostname, install packages, configure timezone
-    3. Log to deployment ledger
+    1. Pushes setup script to instance (Incus file API)
+    2. Executes script (Incus exec API)
+    3. Verifies /etc/deploy-info was written
+    4. Logs to deployment ledger
           |
           v
-  Instance ready, user notified
+  Aether marks deployment complete (or rolls back on failure)
 ```
 
 Decommission works in reverse — Aether triggers the decommission template
-before deleting the instance.
+before deleting the instance. Decommission failures do NOT block deletion.
 
 ### Separation of concerns
 
 | Layer | Tool | Responsibility |
 |-------|------|----------------|
-| Infrastructure | Aether + Incus | VM creation, networking, resources, lifecycle |
+| Infrastructure | Aether + Incus | Instance creation, networking, resources, lifecycle |
 | Configuration | AWX + Ansible | OS config, packages, services, integrations |
 | Automation glue | Aether AWX binding | Triggers playbooks at create/delete |
 
-### What Aether passes to AWX
+### What Aether passes to AWX (validated)
 
-Every AWX job launched by Aether receives these extra vars:
+Every AWX job launched by Aether receives these `ffsdn_` prefixed extra vars:
 
-| Variable | Description |
-|----------|-------------|
-| `vm_name` | Instance hostname |
-| `vm_ip` | IP address assigned by Aether |
-| `environment` | Deployment environment (dev, staging, prod) |
-| `owner` | VM owner or team |
-| `cost_center` | Billing/cost allocation |
-| (custom tags) | All blueprint tags and instance metadata |
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `ffsdn_instance_name` | Instance name | `awx-e2e-test` |
+| `ffsdn_instance_ip` | IP address assigned by Aether | `10.207.217.5` |
+| `ffsdn_cluster_id` | Numeric cluster ID | `52` |
+| `ffsdn_cluster_name` | Cluster display name | `oc-lab-cluster` |
+| `ffsdn_deployed_by` | Aether username who triggered deploy | `admin` |
+| `ffsdn_image_os` | Image OS | `Debian` |
+| `ffsdn_image_release` | Image release | `bookworm` |
+| `ffsdn_image_alias` | Image alias (may be empty) | `""` |
+
+**Important**: Aether does NOT pass `vm_name`, `vm_ip`, `environment`,
+`owner`, or `cost_center`. The original plan assumed these names, but
+real testing revealed Aether uses the `ffsdn_` prefix exclusively.
 
 ### AWX API endpoints Aether uses
 
@@ -64,6 +70,18 @@ Every AWX job launched by Aether receives these extra vars:
 | `POST /api/v2/job_templates/{id}/launch/` | Trigger job with extra_vars |
 | `GET /api/v2/jobs/{id}/` | Poll job status |
 
+### Lifecycle behavior
+
+| Hook | Trigger | On failure |
+|------|---------|------------|
+| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
+| Decommission | Before instance deletion | Failure does NOT block deletion |
+
+Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
+state (successful, failed, error) or the timeout expires. Post-deploy
+rollback means the instance is deleted — the user sees "Ansible job N
+finished with status: failed. Instance has been deleted."
+
 ## Lab deployment
 
 ### Resource requirements
@@ -74,8 +92,9 @@ Every AWX job launched by Aether receives these extra vars:
 | RAM | 8 GiB | 4 GiB |
 | Disk | 40 GiB | 20 GiB |
 
-K3s uses ~600 MiB, AWX pods ~2.5 GiB, PostgreSQL ~400 MiB. A 4 GiB VM
-can run AWX but leaves no headroom for execution environments.
+K3s uses ~600 MiB, AWX pods ~3 GiB, PostgreSQL ~400 MiB. The web pod
+was originally set to 1 GiB limit but OOMKilled; 2 GiB is the tested
+minimum.
 
 ### Lab details
 
@@ -86,30 +105,17 @@ can run AWX but leaves no headroom for execution environments.
 | IP | 192.168.102.161/22 (VLAN 69) |
 | Gateway | 192.168.100.1 |
 | DNS | 192.168.100.1 |
-| Port | 443 (HTTPS via K3s Traefik) |
+| AWX API port | 30080 (K3s NodePort) |
 | OS | Debian 12 |
 | K8s | K3s (single-node) |
+| AWX | 24.6.1 |
 | AWX Operator | 2.19.1 |
 
-IP is adjacent to Aether at .160 on the same VLAN.
+**Note**: AWX is exposed via K3s NodePort on port 30080, NOT via Traefik
+ingress. The ingress returns 404 when accessed by IP (requires hostname).
+NodePort works reliably with IP-based access.
 
-### Quick deploy
-
-```bash
-# Check prerequisites
-./incusos/deploy-awx --doctor
-
-# Full deployment (~10-15 minutes)
-./incusos/deploy-awx --deploy
-
-# Configure AWX (project, inventory, templates)
-./incusos/deploy-awx --configure
-
-# Register with Aether
-./incusos/deploy-awx --join-aether
-```
-
-### Manual deployment
+### Manual deployment (tested)
 
 #### Step 1: Create VM
 
@@ -120,38 +126,51 @@ incus launch images:debian/12 oc-node-02:awx --vm \
     -c limits.cpu=4 -c limits.memory=8GiB \
     -d root,size=40GiB
 
-# Switch to macvlan for direct VLAN access
-incus config device remove oc-node-02:awx eth0
+# Replace default NIC with macvlan for direct VLAN access
+# IMPORTANT: use 'config device add' not 'profile device remove'
+# Profile device removal fails on cluster members
 incus config device add oc-node-02:awx eth0 nic \
     nictype=macvlan parent=mgmt
 ```
 
+**Bug found**: `incus profile device remove` fails on cluster members
+with "Profile device eth0 not found" because profiles are cluster-wide.
+Use `incus config device add` instead, which creates an instance-level
+override that takes priority over the profile.
+
 #### Step 2: Configure static IP
 
+Debian 12 uses `systemd-networkd`, **not netplan**. The original plan
+assumed netplan but that's Ubuntu-only.
+
 ```bash
 incus exec oc-node-02:awx -- bash -c "
     hostnamectl set-hostname awx
-    cat > /etc/netplan/50-static.yaml << 'NETPLAN'
-network:
-  version: 2
-  ethernets:
-    enp5s0:
-      addresses: [192.168.102.161/22]
-      routes:
-        - to: default
-          via: 192.168.100.1
-      nameservers:
-        addresses: [192.168.100.1]
-NETPLAN
-    chmod 600 /etc/netplan/50-static.yaml
-    netplan apply
+
+    # Debian 12 uses systemd-networkd
+    cat > /etc/systemd/network/10-static.network << 'NETCFG'
+[Match]
+Name=enp5s0
+
+[Network]
+Address=192.168.102.161/22
+Gateway=192.168.100.1
+DNS=192.168.100.1
+NETCFG
+
+    systemctl restart systemd-networkd
 "
 ```
 
-#### Step 3: Install K3s
+After IP change, wait a few seconds for the new IP to take effect.
+
+#### Step 3: Install prerequisites and K3s
 
 ```bash
+# Debian 12 minimal doesn't have curl or git
 incus exec oc-node-02:awx -- bash -c "
+    apt-get update && apt-get install -y curl git
+
     curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
     kubectl wait --for=condition=Ready node --all --timeout=120s
 "
@@ -185,20 +204,32 @@ EOF
 #### Step 5: Deploy AWX instance
 
 The AWX custom resource is defined in `incusos/awx-manifests/base/awx.yaml`.
-Push and apply:
+Key settings discovered during testing:
+
+- **PVC**: must use `ReadWriteOnce` (not `ReadWriteMany`) for K3s local-path
+- **Web pod memory**: 2 GiB limit minimum (1 GiB causes OOMKill)
+- **Service type**: `NodePort` on port 30080 (ingress returns 404 for IP access)
 
 ```bash
-incus file push incusos/awx-manifests/base/awx.yaml \
-    oc-node-02:awx/opt/awx/base/awx.yaml
-incus file push incusos/awx-manifests/base/kustomization.yaml \
-    oc-node-02:awx/opt/awx/base/kustomization.yaml
+# Push manifests to VM and apply
+incus exec oc-node-02:awx -- mkdir -p /opt/awx/base
 
+# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/)
+# Then apply:
 incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/
 
 # Wait 5-10 min for all pods to start
 incus exec oc-node-02:awx -- kubectl -n awx get pods -w
 ```
 
+Expected final pod state:
+```
+awx-operator-controller-manager-xxx   2/2     Running
+awx-postgres-15-0                     1/1     Running
+awx-task-xxx                          4/4     Running
+awx-web-xxx                           3/3     Running
+```
+
 #### Step 6: Verify
 
 ```bash
@@ -207,99 +238,129 @@ ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
     awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
 echo "Admin password: $ADMIN_PW"
 
-# Test API
-curl -sk https://192.168.102.161/api/v2/ping/
+# Test API via NodePort
+curl -sk http://192.168.102.161:30080/api/v2/ping/
 ```
 
-Web UI: `https://192.168.102.161/` — login with `admin` / password above.
+AWX web UI: `http://192.168.102.161:30080/` — login with `admin` / password above.
 
-## AWX configuration
+## AWX configuration (tested)
 
-All configuration is API-driven so it can be reproduced. The `deploy-awx
---configure` command automates these steps.
+### Project (manual, no Git dependency)
 
-### Project (Git repository)
-
-Create a project pointing to the `ansible/` directory in this repo:
+AWX Git-based projects require SSH key configuration and network access
+to the Git server. For simplicity, use a **manual project** — playbook
+files are pushed directly to the AWX task pod:
 
 ```bash
-curl -sk -X POST https://192.168.102.161/api/v2/projects/ \
-    --user "admin:${ADMIN_PW}" \
+# Create manual project via API
+AWX_TOKEN="..."  # from PAT creation below
+curl -sk -X POST http://192.168.102.161:30080/api/v2/projects/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
         "name": "incus-contrib",
         "organization": 1,
-        "scm_type": "git",
-        "scm_url": "ssh://git@192.168.1.200:2222/maarten/incus-contrib.git",
-        "scm_branch": "master",
-        "scm_update_on_launch": true
+        "scm_type": "",
+        "local_path": "incus-contrib"
     }'
 ```
 
-AWX clones the repo and discovers playbooks under `ansible/playbooks/`.
+Then push playbooks to the task pod:
+```bash
+AWX_POD=$(incus exec oc-node-02:awx -- kubectl -n awx get pods \
+    -l app.kubernetes.io/name=awx-task -o jsonpath='{.items[0].metadata.name}')
+
+# Create project directory structure
+incus exec oc-node-02:awx -- kubectl -n awx exec $AWX_POD -- \
+    mkdir -p /var/lib/awx/projects/incus-contrib/playbooks
+
+# Push playbooks (via base64 to handle stdin issues)
+B64=$(base64 -w0 ansible/playbooks/post-deploy.yml)
+incus exec oc-node-02:awx -- bash -c "echo '$B64' | base64 -d > /root/post-deploy.yml"
+incus exec oc-node-02:awx -- kubectl -n awx cp /root/post-deploy.yml \
+    $AWX_POD:/var/lib/awx/projects/incus-contrib/playbooks/post-deploy.yml
+```
+
+**Important**: also push the Incus client certificate and key to the
+project directory — the playbooks use these for Incus API calls:
+```bash
+incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.crt \
+    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.crt
+incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.key \
+    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.key
+```
 
 ### Inventory
 
-Aether passes `vm_ip` as an extra var. Playbooks use `add_host` to
-dynamically create an in-memory inventory entry for the target instance.
-The AWX inventory is a placeholder:
+The playbooks use `localhost` exclusively (no SSH, Incus API only).
+The inventory is a placeholder required by AWX:
 
 ```bash
-curl -sk -X POST https://192.168.102.161/api/v2/inventories/ \
-    --user "admin:${ADMIN_PW}" \
+curl -sk -X POST http://192.168.102.161:30080/api/v2/inventories/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name": "incus-instances", "organization": 1}'
 ```
 
 ### Credentials
 
-Two credentials:
-
-1. **SCM credential** (type 2) — SSH key for cloning the private Git repo
-2. **Machine credential** (type 1) — SSH key for connecting to managed instances
+A machine credential (SSH key) is created but only used as a fallback.
+The primary playbook approach uses the Incus REST API with client
+certificates, not SSH.
 
 ```bash
-# Machine credential
-curl -sk -X POST https://192.168.102.161/api/v2/credentials/ \
-    --user "admin:${ADMIN_PW}" \
+# Generate SSH key on AWX VM
+incus exec oc-node-02:awx -- ssh-keygen -t ed25519 -f /root/.ssh/awx_key -N ""
+
+# Create machine credential in AWX
+PRIV_KEY=$(incus exec oc-node-02:awx -- cat /root/.ssh/awx_key)
+curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
-    -d '{
-        "name": "incus-instances",
-        "organization": 1,
-        "credential_type": 1,
-        "inputs": {"username": "root", "ssh_key_data": "..."}
-    }'
+    -d "{
+        \"name\": \"incus-instances\",
+        \"organization\": 1,
+        \"credential_type\": 1,
+        \"inputs\": {\"username\": \"root\", \"ssh_key_data\": $(echo "$PRIV_KEY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}
+    }"
 ```
 
 ### Job templates
 
-Two templates matching Aether's lifecycle hooks:
+| Template | Playbook | Template ID |
+|----------|----------|-------------|
+| `post-deploy` | `playbooks/post-deploy.yml` | 10 |
+| `decommission` | `playbooks/decommission.yml` | 11 |
 
-| Template | Playbook | Purpose |
-|----------|----------|---------|
-| `post-deploy` | `ansible/playbooks/post-deploy.yml` | Runs after instance creation |
-| `decommission` | `ansible/playbooks/decommission.yml` | Runs before instance deletion |
-
-Both must have `ask_variables_on_launch: true` so Aether can inject extra vars.
+Both must have `ask_variables_on_launch: true`:
 
 ```bash
-curl -sk -X POST https://192.168.102.161/api/v2/job_templates/ \
-    --user "admin:${ADMIN_PW}" \
+curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
         "name": "post-deploy",
         "organization": 1,
-        "project": <PROJECT_ID>,
-        "playbook": "ansible/playbooks/post-deploy.yml",
-        "inventory": <INVENTORY_ID>,
+        "project": 6,
+        "playbook": "playbooks/post-deploy.yml",
+        "inventory": 2,
         "ask_variables_on_launch": true
     }'
 ```
 
-### Personal Access Token (for Aether)
+**Attach credentials** to templates (required even if playbook doesn't use SSH):
+```bash
+curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"id": 4}'
+```
+
+### Personal Access Token
 
 ```bash
-AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tokens/ \
+AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/personal_tokens/ \
     --user "admin:${ADMIN_PW}" \
     -H "Content-Type: application/json" \
     -d '{"description": "Aether integration", "scope": "write"}' \
@@ -310,232 +371,300 @@ AWX_TOKEN=$(curl -sk -X POST https://192.168.102.161/api/v2/users/1/personal_tok
 
 ### Register AWX endpoint
 
-In the Aether web UI: Settings → Ansible Automation (`/awx-endpoints`) →
-Add Endpoint:
+Via Aether API (requires session cookies + CSRF token):
 
-| Field | Value |
-|-------|-------|
-| Name | `lab-awx` |
-| URL | `https://192.168.102.161` |
-| Token | (PAT from above) |
-| Verify SSL | unchecked (self-signed) |
-
-Or via API:
 ```bash
-curl -sSk -b cookies.txt \
+curl -sSk -b /tmp/aether-cookies.txt \
     -H "X-CSRF-Token: $CSRF" \
     -H "Referer: https://192.168.102.160:8443/" \
     -X POST https://192.168.102.160:8443/api/awx/endpoints \
     -H "Content-Type: application/json" \
     -d '{
         "name": "lab-awx",
-        "url": "https://192.168.102.161",
+        "url": "http://192.168.102.161:30080",
         "token": "'$AWX_TOKEN'",
         "verify_ssl": false
     }'
 ```
 
+**Tested result**: endpoint ID 2 created, AWX health shows "healthy"
+at `/api/health/awx`:
+```json
+{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}
+```
+
 ### Bind cluster to AWX
 
-In the Aether UI: AWX endpoint → cluster config, or via API:
+**Known bug**: `PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid
+cluster ID"}` for all valid cluster IDs. This Aether API endpoint appears
+non-functional in v6.4.317.
 
+**Workaround**: direct PostgreSQL update on the Aether database:
 ```bash
-curl -sSk -b cookies.txt \
-    -H "X-CSRF-Token: $CSRF" \
-    -H "Referer: https://192.168.102.160:8443/" \
-    -X PUT https://192.168.102.160:8443/api/clusters/52/awx-config \
-    -H "Content-Type: application/json" \
-    -d '{
-        "cluster_id": 52,
-        "awx_endpoint_id": <ENDPOINT_ID>,
-        "awx_post_deploy_template_id": <POST_DEPLOY_TEMPLATE_ID>,
-        "awx_decommission_template_id": <DECOMMISSION_TEMPLATE_ID>,
-        "awx_job_timeout_seconds": 600
-    }'
+incus exec oc-node-01:aether -- bash -c "
+    docker exec aether-postgres psql -U aether -d incusovnsdnc -c \"
+        UPDATE incus_ovn_clusters SET
+            awx_endpoint_id = 2,
+            awx_post_deploy_template_id = 10,
+            awx_decommission_template_id = 11,
+            awx_job_timeout_seconds = 600
+        WHERE cluster_id = 52;
+    \"
+"
 ```
 
 ### Verify integration
 
-- Aether Health page (`/health`) shows AWX endpoint as "Healthy"
-- Deploy a test instance via Aether — AWX post-deploy job should trigger
-- Check AWX job history for the completed run
-- Delete the test instance — decommission job should trigger
+```bash
+# Check Aether's AWX health endpoint
+curl -sSk -b /tmp/aether-cookies.txt \
+    https://192.168.102.160:8443/api/health/awx
 
-### Lifecycle behavior
-
-| Hook | Trigger | On failure |
-|------|---------|------------|
-| Post-deploy | After instance creation | Aether auto-rollbacks (deletes instance) |
-| Decommission | Before instance deletion | Failure does NOT block deletion |
-
-Aether polls `GET /api/v2/jobs/{id}/` until the job reaches a terminal
-state (successful, failed, error) or the timeout expires.
+# Deploy a test instance via Aether → AWX job triggers automatically
+# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"
+```
 
 ## Writing playbooks
 
-### Directory structure
+### Key design decision: Incus API, not SSH
 
-```
-ansible/
-├── ansible.cfg                    # Project-level config
-├── playbooks/
-│   ├── post-deploy.yml            # Aether post-deploy hook
-│   └── decommission.yml           # Aether decommission hook
-└── roles/
-    └── base-config/
-        ├── tasks/main.yml         # Base OS configuration
-        └── templates/             # Jinja2 templates (future)
-```
+The playbooks use the **Incus REST API** (file push + exec) instead of
+SSH for all instance configuration. This is necessary because:
 
-### Pattern: dynamic host from extra vars
+1. **Bridge network isolation**: containers on incusbr0 (10.207.217.0/24)
+   are not routable from AWX (192.168.102.161). The bridge is NAT'd and
+   IncusOS nodes don't forward inbound traffic to the bridge.
+2. **No SSH dependency**: fresh Debian 12 containers don't have python3
+   installed. With SSH, a raw bootstrap step is needed first.
+3. **Works with any network**: the Incus API is accessible at the cluster
+   node level (192.168.102.140-142:8443), regardless of which overlay
+   or bridge network the container is on.
 
-Aether passes `vm_ip` — playbooks use `add_host` to target it:
+The trade-off: playbooks require an Incus client certificate in the AWX
+project directory, and the playbook pattern is less "Ansible-native" (no
+SSH connection plugins, no gather_facts, no modules running on the target).
+
+### File paths in AWX
+
+AWX runs playbooks inside an **Execution Environment** (EE) container via
+receptor. During job execution:
+
+| Path | Description |
+|------|-------------|
+| `/runner/project/` | Project root (maps to the manual project directory) |
+| `/runner/project/playbooks/` | Playbook directory |
+| `/runner/project/incus-client.crt` | Incus client certificate |
+| `/runner/project/incus-client.key` | Incus client key |
+| `/runner/artifacts/{job_id}/` | Job artifacts (logs, SSH keys) |
+
+**Important**: files at `/var/lib/awx/projects/` (the task pod) are NOT
+accessible at the same path during job execution. The EE mounts the
+project at `/runner/project/`.
+
+### post-deploy.yml pattern (tested, validated)
 
 ```yaml
-- name: Add target host
+- name: Post-deploy — configure instance via Incus API
   hosts: localhost
   gather_facts: false
-  tasks:
-    - ansible.builtin.add_host:
-        name: "{{ vm_ip }}"
-        groups: target
-        ansible_user: root
+  connection: local
+
+  vars:
+    incus_api: "https://192.168.102.140:8443"
+    incus_cert: "/runner/project/incus-client.crt"
+    incus_key: "/runner/project/incus-client.key"
 
-- name: Configure target
-  hosts: target
   tasks:
-    - ansible.builtin.wait_for_connection:
-        delay: 5
-        timeout: 300
-    # ... configuration tasks
+    # 1. Validate required ffsdn_* variables from Aether
+    - name: Validate required variables
+      ansible.builtin.assert:
+        that:
+          - ffsdn_instance_name is defined
+          - ffsdn_instance_name | length > 0
+
+    # 2. Wait for instance to be Running
+    - name: Wait for instance to be running
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+      register: state
+      until: state.json.metadata.status == "Running"
+      retries: 12
+      delay: 5
+
+    # 3. Push setup script via Incus file API
+    - name: Push setup script
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/setup.sh"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body: "{{ setup_script }}"
+        headers:
+          Content-Type: "application/octet-stream"
+          X-Incus-type: "file"
+          X-Incus-mode: "0755"
+
+    # 4. Execute via Incus exec API
+    - name: Execute setup script
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command: ["/bin/bash", "/tmp/setup.sh"]
+          record-output: true
+          interactive: false
+          wait-for-websocket: false
+        status_code: [202]
+      register: exec_result
+
+    # 5. Wait for completion via operation API
+    - name: Wait for script to complete
+      ansible.builtin.uri:
+        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+      register: exec_wait
+
+    # 6. Verify exit code
+    - name: Verify exit code
+      ansible.builtin.assert:
+        that:
+          - exec_wait.json.metadata.metadata.return | int == 0
+
+    # 7. Verify result via file API
+    - name: Read /etc/deploy-info
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: deploy_info
 ```
 
-### Pattern: best-effort decommission
+### Incus API endpoints used by playbooks
 
-Decommission playbooks should never block instance deletion:
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/1.0/instances/{name}/state` | GET | Check if instance is Running |
+| `/1.0/instances/{name}/files?path=...` | POST | Push files to instance |
+| `/1.0/instances/{name}/files?path=...` | GET | Read files from instance |
+| `/1.0/instances/{name}/exec` | POST | Execute commands in instance |
+| `/1.0/operations/{uuid}/wait` | GET | Wait for async operation |
+| `/1.0/instances/{name}` | GET | Check if instance exists |
+
+**Note**: exec output (stdout/stderr from `record-output: true`) is NOT
+retrievable via the `/1.0/instances/{name}/logs/` API. The log filename
+format (`exec_UUID.stdout`) is rejected as "not valid". Verify results by
+reading files written by the script instead.
+
+### decommission.yml pattern (tested, validated)
 
 ```yaml
-- name: Attempt cleanup on target
-  hosts: target
-  ignore_unreachable: true
+- name: Decommission — graceful shutdown and logging
+  hosts: localhost
+  gather_facts: false
+  connection: local
+
+  vars:
+    incus_api: "https://192.168.102.140:8443"
+    incus_cert: "/runner/project/incus-client.crt"
+    incus_key: "/runner/project/incus-client.key"
+
   tasks:
-    - name: Check if host is reachable
-      ansible.builtin.wait_for_connection:
-        timeout: 15
+    - name: Check if instance still exists
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        status_code: [200, 404]
+      register: instance_check
+
+    - name: Stop services (best-effort)
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        # ... service stop command
       ignore_errors: true
-      register: host_check
+      when: instance_check.status == 200
 
-    - name: Cleanup tasks
-      # ...
-      when: host_check is succeeded
+    - name: Record in ledger
+      ansible.builtin.lineinfile:
+        path: /tmp/awx-deploy-ledger.log
+        # ... log entry
 ```
 
-### The base-config role
+## Bugs discovered during validation
 
-The included `base-config` role performs minimal post-deploy setup:
+### Ansible reserved keyword: `environment`
 
-1. Set hostname
-2. Update apt cache
-3. Install base packages (curl, vim, htop, jq, tmux, rsync, unattended-upgrades)
-4. Set timezone to UTC
-5. Enable unattended security upgrades
-6. Write deployment metadata to `/etc/deploy-info`
+Ansible's `environment` keyword is reserved — it sets process environment
+variables for tasks. If Aether passed `environment: "lab"` as an extra var,
+Ansible would interpret it as the keyword, resolving to `[]` instead of
+the string value. The fix was to use `deploy_env` instead. In practice,
+Aether uses `ffsdn_*` prefixes which avoid this entirely.
 
-### Extending playbooks
-
-Common additions for production environments:
-
-| Task | Method | Module |
-|------|--------|--------|
-| DNS registration | API from AWX controller | `community.general.nsupdate` or REST |
-| Monitoring agent | SSH to target | `ansible.builtin.package` + config |
-| SSH key distribution | SSH to target | `ansible.posix.authorized_key` |
-| Certificate provisioning | API from controller | `community.crypto.acme_certificate` |
-| Backup enrollment | API from controller | REST calls to backup system |
-
-External system tasks use `delegate_to: localhost` to run from the AWX
-controller rather than on the target instance.
-
-## The deploy-awx script
-
-### Usage
-
-```
-deploy-awx [OPTIONS]
-
-Actions:
-    --deploy          Full deploy: create VM → K3s → AWX → configure
-    --status          Check AWX health, K3s pods, Aether connection
-    --heal            Restart failed pods, re-sync project, re-validate
-    --configure       (Re-)configure AWX: project, inventory, templates
-    --join-aether     Register AWX with Aether + print binding instructions
-    --cleanup         Destroy the AWX VM
-    --doctor          Check prerequisites
-
-Options:
-    -c, --config FILE     Configuration file (YAML with awx: section)
-    -n, --dry-run         Preview actions without executing
-    -v, --verbose         Show detailed output
-    -q, --quiet           Suppress informational output
-```
-
-### Configuration
-
-Defaults are built into the script for the lab environment. Override with
-a YAML config file:
+### Recursive template loop with self-referencing vars
 
 ```yaml
-awx:
-  vm_name: awx
-  target_node: oc-node-02
-  ip: 192.168.102.161/22
-  gateway: 192.168.100.1
-  dns: 192.168.100.1
-  cpu: 4
-  memory: 8GiB
-  disk: 40GiB
-  git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
-  git_branch: master
-  aether_url: https://192.168.102.160:8443
-  aether_cluster_id: 52
+# BROKEN — causes infinite recursion if vm_ip is not provided
+vars:
+  vm_ip: "{{ vm_ip | default('') }}"
 ```
 
-### Deployment phases
+When the variable name matches the template reference AND the variable
+is not provided as an extra var, Ansible enters infinite recursion.
+The fix: don't self-reference. Use the `ffsdn_*` variables directly
+without redeclaring them in vars blocks.
 
-1. **Create VM** — `incus launch images:debian/12` with macvlan NIC
-2. **Configure network** — static IP via netplan, hostname
-3. **Install K3s** — single-node K3s with kubeconfig mode 644
-4. **Deploy AWX Operator** — kustomize from `awx-manifests/operator/`
-5. **Deploy AWX instance** — kustomize from `awx-manifests/base/`
-6. **Verify** — API ping, admin password retrieval
+### AWX EE runs at /runner/project/, not /var/lib/awx/projects/
 
-### Heal logic
+Files pushed to the AWX task pod at `/var/lib/awx/projects/incus-contrib/`
+are mounted in the EE container at `/runner/project/`. All file paths in
+playbooks must use the `/runner/project/` prefix.
 
-The `--heal` command checks and fixes:
+### Bridge network not routable from AWX VM
 
-1. K3s service running → restart if not
-2. All AWX pods healthy → delete unhealthy (operator recreates)
-3. AWX web responding → restart deployments if not
-4. Project sync current → trigger re-sync
+Containers on incusbr0 (10.207.217.0/24) are NOT reachable from the AWX
+VM (192.168.102.161). The bridge is NAT'd outbound only; IncusOS nodes
+don't forward inbound traffic from the management network to the bridge.
+Adding a static route on the AWX VM (`ip route add 10.207.217.0/24 via
+192.168.102.140`) doesn't work because the node's nftables rules block
+non-established connections from external interfaces to the bridge.
 
-### Manifests
+**Solution**: use the Incus REST API instead of SSH. The cluster API
+(192.168.102.140:8443) is always reachable, and provides file push +
+exec endpoints that work regardless of instance network topology.
 
-K8s manifests stored in `incusos/awx-manifests/`:
+### Aether cluster AWX config API bug
 
-```
-awx-manifests/
-├── operator/
-│   └── kustomization.yaml    # AWX Operator 2.19.1 via kustomize
-└── base/
-    ├── kustomization.yaml    # AWX instance resources
-    └── awx.yaml              # AWX custom resource (pods, storage, ingress)
-```
+`PUT /api/clusters/{id}/awx-config` returns `{"error":"Invalid cluster ID"}`
+for all valid cluster IDs. Tested with different CSRF tokens, session
+cookies, request formats (JSON fields, POST vs PUT). The endpoint is not
+documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE.
 
-The AWX CR in `awx.yaml` defines resource limits tuned for the 8 GiB VM:
-- Web pod: 512 MiB → 1 GiB
-- Task pod: 512 MiB → 2 GiB
-- EE pod: 256 MiB → 1 GiB
-- PostgreSQL: 256 MiB → 512 MiB
+### AWX web pod OOMKill at 1 GiB
+
+The AWX web pod (nginx + uwsgi) needs at least 2 GiB memory limit.
+With 1 GiB, it starts successfully but gets OOMKilled under load.
+Set in the AWX CR: `web_resource_requirements.limits.memory: 2Gi`.
+
+### AWX PVC needs ReadWriteOnce
+
+K3s local-path provisioner creates `hostPath` volumes which only support
+`ReadWriteOnce`. If the AWX CR specifies `ReadWriteMany`, the PVC stays
+Pending and PostgreSQL can't start. Set: `postgres_storage_class: local-path`
+and ensure `ReadWriteOnce` access mode.
 
 ## Troubleshooting
 
@@ -552,72 +681,85 @@ incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
 incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>
 ```
 
+### Job output
+
+```bash
+# Get real-time job output (text format)
+curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/stdout/?format=txt \
+    -H "Authorization: Bearer $AWX_TOKEN"
+
+# Check job status
+curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; d=json.load(sys.stdin); \
+        print('status:', d['status'], 'failed:', d['failed'])"
+```
+
 ### Common issues
 
-**AWX pods stuck in Pending**: insufficient resources. Check node capacity:
+**AWX pods stuck in Pending**: insufficient resources or PVC issue. Check:
 ```bash
-incus exec oc-node-02:awx -- kubectl describe node | grep -A5 "Allocated resources"
+incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
+incus exec oc-node-02:awx -- kubectl -n awx get pvc
 ```
 
-**AWX web returns 502**: task or web pod crashed. Restart:
+**Job fails with recursive template error**: self-referencing var definition.
+Check playbook vars blocks for patterns like `var: "{{ var | default('') }}"`.
+
+**Job fails with "No such file or directory"**: cert/key not at the correct
+path. Remember: EE uses `/runner/project/`, not `/var/lib/awx/projects/`.
+
+**Job fails with "Connection failure"**: Incus API not reachable from AWX
+pod, or client cert not trusted by cluster. Test from inside the pod:
 ```bash
-incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-web
-incus exec oc-node-02:awx -- kubectl -n awx rollout restart deployment/awx-task
+incus exec oc-node-02:awx -- kubectl -n awx exec <task-pod> -c awx-ee -- \
+    curl -sk --cert /runner/project/incus-client.crt \
+    --key /runner/project/incus-client.key \
+    https://192.168.102.140:8443/1.0
 ```
 
-**Project sync fails**: SCM credential or network issue. Check:
-```bash
-incus exec oc-node-02:awx -- kubectl -n awx logs deployment/awx-task | grep -i scm
-```
-
-**Job fails with "Host unreachable"**: target instance not booted yet or
-SSH not available. Increase the `wait_for_connection` timeout in the playbook
-or add a delay in Aether's job timeout setting.
-
-**K3s not starting**: VM resources exhausted. Check:
-```bash
-incus exec oc-node-02:awx -- systemctl status k3s
-incus exec oc-node-02:awx -- journalctl -u k3s --no-pager -n 50
-```
-
-**Cannot reach AWX from outside**: macvlan NIC misconfigured. Verify:
-```bash
-incus exec oc-node-02:awx -- ip addr show enp5s0
-incus exec oc-node-02:awx -- ip route
-```
+**Aether deploy rolls back immediately**: AWX job failed. Check the AWX
+job output for the specific error. Common causes: template misconfiguration,
+missing credential attachment, playbook syntax errors.
 
 ### Useful commands
 
 ```bash
 # AWX version
-curl -sk https://192.168.102.161/api/v2/ping/ | python3 -m json.tool
+curl -sk http://192.168.102.161:30080/api/v2/ping/
 
 # List job templates
-curl -sk https://192.168.102.161/api/v2/job_templates/ \
-    --user "admin:$PW" | python3 -m json.tool
+curl -sk http://192.168.102.161:30080/api/v2/job_templates/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; [print(f'{t[\"id\"]:3d} {t[\"name\"]}') \
+        for t in json.load(sys.stdin)['results']]"
 
 # List recent jobs
-curl -sk https://192.168.102.161/api/v2/jobs/?order_by=-id\&page_size=5 \
-    --user "admin:$PW" | python3 -c "
-import sys,json
-for j in json.load(sys.stdin)['results']:
-    print(f'{j[\"id\"]:4d}  {j[\"status\"]:12s}  {j[\"name\"]}')"
+curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
+    -H "Authorization: Bearer $AWX_TOKEN" \
+    | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
+        for j in json.load(sys.stdin)['results']]"
 
 # Manually trigger post-deploy
-curl -sk -X POST https://192.168.102.161/api/v2/job_templates/1/launch/ \
-    --user "admin:$PW" \
+curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
+    -H "Authorization: Bearer $AWX_TOKEN" \
     -H "Content-Type: application/json" \
-    -d '{"extra_vars": {"vm_name": "test-vm", "vm_ip": "192.168.102.200", "environment": "lab", "owner": "admin"}}'
+    -d '{"extra_vars": {
+        "ffsdn_instance_name": "test-ct",
+        "ffsdn_instance_ip": "10.207.217.99",
+        "ffsdn_cluster_name": "oc-lab-cluster",
+        "ffsdn_deployed_by": "admin",
+        "ffsdn_image_os": "Debian",
+        "ffsdn_image_release": "bookworm"
+    }}'
 ```
 
 ### Rollback
 
 If AWX is broken beyond repair:
 ```bash
-./incusos/deploy-awx --cleanup
-./incusos/deploy-awx --deploy
-./incusos/deploy-awx --configure
-./incusos/deploy-awx --join-aether
+incus delete oc-node-02:awx --force
+# Then redeploy from Step 1
 ```
 
 The AWX VM is a standalone workload with no cluster dependencies. Destroying