Validate AWX guide against real infrastructure, fix all bugs

End-to-end tested: Aether deploy triggers AWX post-deploy job,
container gets configured (hostname, packages, timezone, deploy-info),
decommission job runs on delete. All playbooks validated working.

Key fixes from testing:
- Playbooks: use Incus REST API (not SSH) — bridge subnet not routable
  from management VLAN, nftables blocks inbound forwarding
- Aether vars: ffsdn_* prefix (not vm_name/vm_ip/environment/owner)
- AWX EE paths: /runner/project/ (not /var/lib/awx/projects/)
- AWX access: NodePort 30080 (not Traefik ingress — 404 for IP access)
- deploy-awx: systemd-networkd (not netplan), manual project (not git
  SCM — EE can't reach private repo), Incus cert push, prereq install
- Remove unused base-config role (tasks inlined as setup script)
- Simplify ansible.cfg (no SSH settings needed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -15,12 +15,9 @@ incu-contrib/
 ├── .gitignore
 ├── ansible/               # Ansible playbooks for AWX/Aether lifecycle hooks
 │   ├── ansible.cfg        # Project-level Ansible config
-│   ├── playbooks/
+│   └── playbooks/
-│   │   ├── post-deploy.yml    # Runs after Aether creates an instance
+│       ├── post-deploy.yml    # Runs after Aether creates an instance (Incus API)
-│   │   └── decommission.yml   # Runs before Aether deletes an instance
+│       └── decommission.yml   # Runs before Aether deletes an instance
-│   └── roles/
-│       └── base-config/       # Base OS configuration role
-│           └── tasks/main.yml
 ├── incusos/               # IncusOS installation tooling
 │   ├── README.md          # Detailed usage docs
 │   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
@@ -826,22 +823,40 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
   Tower). Deployed as a Debian 12 VM running K3s + AWX Operator on the cluster.
 - **Lab VM**: `awx` on oc-node-02, IP 192.168.102.161/22 (VLAN 69, adjacent
   to Aether at .160). 4 vCPU, 8 GiB RAM, 40 GiB disk.
+- **AWX URL**: `http://192.168.102.161:30080` — exposed via K3s NodePort.
+  Traefik ingress returns 404 for IP-based access; use NodePort directly.
 - **`deploy-awx`** script manages the full lifecycle: `--deploy`, `--status`,
   `--heal`, `--configure`, `--join-aether`, `--cleanup`, `--doctor`.
 - **K8s manifests** in `incusos/awx-manifests/` (operator + AWX CR via kustomize).
 - **Ansible playbooks** in `ansible/` directory:
   - `playbooks/post-deploy.yml` -- runs after Aether creates an instance
   - `playbooks/decommission.yml` -- runs before Aether deletes an instance
-  - `roles/base-config/` -- hostname, packages, timezone, deploy metadata
+- **Aether extra vars use `ffsdn_` prefix**: Aether passes `ffsdn_instance_name`,
-- **Aether integration**: AWX endpoint registered at `/awx-endpoints` with
+  `ffsdn_instance_ip`, `ffsdn_cluster_id`, `ffsdn_cluster_name`,
-  a Personal Access Token. Per-cluster config binds post-deploy and
+  `ffsdn_deployed_by`, `ffsdn_image_os`, `ffsdn_image_release`,
-  decommission template IDs. Aether calls `POST /api/v2/job_templates/{id}/launch/`
+  `ffsdn_image_alias`. It does NOT pass `vm_name`, `vm_ip`, `environment`,
-  with extra vars (`vm_name`, `vm_ip`, `environment`, `owner`, `cost_center`).
+  `owner`, or `cost_center` (the original plan assumed these).
+- **Playbook pattern — Incus REST API, not SSH**: AWX cannot SSH to containers
+  on incusbr0 (bridge subnet not routable from management VLAN, IncusOS
+  nftables blocks inbound forwarding). Playbooks use the Incus REST API
+  (`uri` module with client cert) for file push + exec. The cert is at
+  `/runner/project/incus-client.crt` during job execution (AWX EE mounts
+  projects at `/runner/project/`, NOT `/var/lib/awx/projects/`).
+- **Manual project (local_path)**: AWX EE containers cannot reach the private
+  git repo. Use `scm_type: ""` with `local_path: "incus-contrib"` and push
+  playbooks directly to the AWX task pod at
+  `/var/lib/awx/projects/incus-contrib/playbooks/`.
 - **Lifecycle hooks**: post-deploy failure triggers auto-rollback (instance
   deleted). Decommission failure does NOT block deletion.
-- **Playbook pattern**: Aether passes `vm_ip` → playbook uses `add_host` to
+- **Aether cluster AWX config API bug**: `PUT /api/clusters/{id}/awx-config`
-  create dynamic inventory → SSH to target for configuration. External system
+  returns "Invalid cluster ID" for valid IDs. Workaround: direct PostgreSQL
-  tasks use `delegate_to: localhost`.
+  UPDATE on the `clusters` table from within the Aether container.
+- **Self-referencing vars cause infinite recursion in Ansible**: patterns like
+  `vm_ip: "{{ vm_ip | default('') }}"` cause `AnsibleUndefinedVariable`
+  recursive loop when the variable is not provided. Use `ffsdn_*` vars
+  directly without redefining them.
+- **Ansible `environment` is a reserved keyword**: using it as an extra var
+  resolves to `[]` instead of the string value.
 - See `notes/awx-guide.md` for full deployment guide and troubleshooting.
 
 ## Git workflow

README.md

@@ -31,13 +31,13 @@ See [`incusos/README.md`](incusos/README.md) for full documentation.
 
 ### [`ansible/`](ansible/)
 
-Ansible playbooks and roles for AWX/Aether lifecycle hooks:
+Ansible playbooks for AWX/Aether lifecycle hooks. Playbooks use the Incus REST
+API (not SSH) for all instance operations — works regardless of network topology.
 
 - **`playbooks/post-deploy.yml`** -- Runs after Aether creates an instance.
   Configures hostname, installs base packages, sets timezone, logs deployment.
 - **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
   Graceful service shutdown and decommission logging (best-effort).
-- **`roles/base-config/`** -- Base OS configuration role used by post-deploy.
 
 ### [`notes/`](notes/)
 

ansible/ansible.cfg

@@ -2,10 +2,7 @@
 # Inventory — AWX manages inventory; this is for local testing only
 inventory = localhost,
 
-# Roles path
+# Reduce overhead
-roles_path = roles
-
-# Reduce SSH overhead
 forks = 10
 timeout = 30
 
@@ -18,11 +15,7 @@ stdout_callback = yaml
 # Host key checking — disabled for dynamic lab instances
 host_key_checking = False
 
-[privilege_escalation]
+# Note: playbooks use the Incus REST API (uri module with client cert)
-become = True
+# for all instance operations — no SSH connection to instances is needed.
-become_method = sudo
+# The [ssh_connection] and [privilege_escalation] sections are therefore
-
+# not required.
-[ssh_connection]
-# Speed up SSH connections
-pipelining = True
-ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no

ansible/playbooks/decommission.yml

@@ -1,78 +1,77 @@
 ---
 # decommission.yml — Runs before Aether deletes an instance
 #
-# Aether passes the same extra vars as post-deploy:
+# Aether passes the same ffsdn_ extra vars as post-deploy:
-#   vm_name, vm_ip, environment, owner, cost_center
+#   ffsdn_instance_name, ffsdn_instance_ip, ffsdn_cluster_id,
+#   ffsdn_cluster_name, ffsdn_deployed_by
 #
 # This playbook runs best-effort: failures do NOT block instance
-# deletion in Aether. The playbook should be defensive and handle
+# deletion in Aether. Uses the Incus REST API for all operations
-# unreachable hosts gracefully.
+# (no SSH needed).
 
-- name: Decommission — prepare instance for deletion
+- name: Decommission — graceful shutdown and logging
   hosts: localhost
   gather_facts: false
   connection: local
 
   vars:
-    vm_name: "{{ vm_name | default('unknown') }}"
+    incus_api: "https://192.168.102.140:8443"
-    vm_ip: "{{ vm_ip | default('') }}"
+    # AWX runner mounts project at /runner/project/ during job execution
-    environment_name: "{{ environment | default('lab') }}"
+    incus_cert: "/runner/project/incus-client.crt"
-    owner: "{{ owner | default('unassigned') }}"
+    incus_key: "/runner/project/incus-client.key"
-    cost_center: "{{ cost_center | default('') }}"
-    ssh_user: root
 
   tasks:
-    - name: Add target host to in-memory inventory
+    - name: Check if instance still exists
-      ansible.builtin.add_host:
+      ansible.builtin.uri:
-        name: "{{ vm_ip }}"
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
-        groups: target
+        method: GET
-        ansible_user: "{{ ssh_user }}"
+        client_cert: "{{ incus_cert }}"
-        ansible_host: "{{ vm_ip }}"
+        client_key: "{{ incus_key }}"
-      when: vm_ip | length > 0
+        validate_certs: false
-
+        return_content: true
-- name: Attempt graceful shutdown on target
+        status_code: [200, 404]
-  hosts: target
+      register: instance_check
-  gather_facts: false
+      when: ffsdn_instance_name is defined
-  ignore_unreachable: true
-
-  tasks:
-    - name: Check if host is reachable
-      ansible.builtin.wait_for_connection:
-        delay: 0
-        timeout: 15
-      register: host_check
-      ignore_errors: true
 
     - name: Stop application services (best-effort)
-      ansible.builtin.shell: |
+      ansible.builtin.uri:
-        # Stop user-facing services before deletion
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
-        systemctl list-units --type=service --state=running --no-legend \
+        method: POST
-          | awk '{print $1}' \
+        client_cert: "{{ incus_cert }}"
-          | grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
+        client_key: "{{ incus_key }}"
-          | head -20 \
+        validate_certs: false
-          | xargs -r -n1 systemctl stop
+        body_format: json
-      register: service_stop
+        body:
+          command:
+            - "/bin/bash"
+            - "-c"
+            - |
+              systemctl list-units --type=service --state=running --no-legend \
+                | awk '{print $1}' \
+                | grep -vE '^(ssh|systemd|dbus|cron|networking|rsyslog)' \
+                | head -20 \
+                | xargs -r -n1 systemctl stop 2>/dev/null || true
+              journalctl --flush 2>/dev/null || true
+          record-output: false
+          interactive: false
+          wait-for-websocket: false
+        status_code: [202]
       ignore_errors: true
-      when: host_check is succeeded
+      when:
+        - instance_check is defined
+        - instance_check.status == 200
+        - instance_check.json.metadata.status is defined
+        - instance_check.json.metadata.status == "Running"
 
-    - name: Flush logs to persistent storage
-      ansible.builtin.command: journalctl --flush
-      ignore_errors: true
-      when: host_check is succeeded
-
-- name: Log decommission
-  hosts: localhost
-  gather_facts: false
-  connection: local
-
-  tasks:
     - name: Record decommission in ledger
       ansible.builtin.lineinfile:
-        path: /var/log/awx-deploy-ledger.log
+        path: /tmp/awx-deploy-ledger.log
         line: >-
-          {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
+          {{ lookup('pipe', 'date -Iseconds') }}
-          DECOMMISSION vm_name={{ vm_name }} vm_ip={{ vm_ip }}
+          DECOMMISSION instance={{ ffsdn_instance_name | default('unknown') }}
-          environment={{ environment_name }} owner={{ owner }}
+          ip={{ ffsdn_instance_ip | default('') }}
-          cost_center={{ cost_center }} status=completed
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          status=completed
         create: true
         mode: "0644"

ansible/playbooks/post-deploy.yml

@@ -1,88 +1,200 @@
 ---
 # post-deploy.yml — Runs after Aether creates an instance
 #
-# Aether passes extra vars:
+# Aether passes these extra vars (ffsdn_ prefix):
-#   vm_name       — instance hostname
+#   ffsdn_instance_name  — instance name
-#   vm_ip         — IP address assigned by Aether
+#   ffsdn_instance_ip    — IP address assigned by Aether
-#   environment   — deployment environment (dev, staging, prod)
+#   ffsdn_cluster_id     — numeric cluster ID
-#   owner         — VM owner or team
+#   ffsdn_cluster_name   — cluster display name
-#   cost_center   — billing/cost allocation
+#   ffsdn_deployed_by    — Aether username who triggered the deploy
-#   (plus any custom tags from the blueprint)
+#   ffsdn_image_os       — image OS (e.g., "Debian")
+#   ffsdn_image_release  — image release (e.g., "bookworm")
+#   ffsdn_image_alias    — image alias (may be empty)
 #
-# AWX job template must have "ask_variables_on_launch: true" so Aether
+# This playbook configures new instances via the Incus REST API
-# can inject these variables at runtime.
+# (file push + exec), which works regardless of network topology.
+# AWX does NOT need SSH access to the container — it talks to the
+# Incus cluster API at the node level.
+#
+# Requirements:
+#   - Incus client cert/key at /var/lib/awx/projects/incus-contrib/
+#   - Cluster API reachable from AWX (e.g., https://192.168.102.140:8443)
 
-- name: Post-deploy — configure newly provisioned instance
+- name: Post-deploy — configure instance via Incus API
   hosts: localhost
   gather_facts: false
   connection: local
 
   vars:
-    # Defaults (overridden by Aether extra vars)
+    # Incus cluster API (any cluster member works)
-    vm_name: "{{ vm_name | default('unknown') }}"
+    incus_api: "https://192.168.102.140:8443"
-    vm_ip: "{{ vm_ip | default('') }}"
+    # AWX runner mounts project at /runner/project/ during job execution
-    environment_name: "{{ environment | default('lab') }}"
+    incus_cert: "/runner/project/incus-client.crt"
-    owner: "{{ owner | default('unassigned') }}"
+    incus_key: "/runner/project/incus-client.key"
-    cost_center: "{{ cost_center | default('') }}"
+    base_packages: "curl vim htop jq tmux rsync unattended-upgrades"
-    ssh_user: root
-    base_packages:
-      - curl
-      - vim
-      - htop
-      - jq
-      - tmux
-      - rsync
-      - unattended-upgrades
 
-  pre_tasks:
+  tasks:
     - name: Validate required variables
       ansible.builtin.assert:
         that:
-          - vm_ip | length > 0
+          - ffsdn_instance_name is defined
-          - vm_name | length > 0
+          - ffsdn_instance_name | length > 0
-        fail_msg: "vm_ip and vm_name must be provided by Aether"
+        fail_msg: "ffsdn_instance_name must be provided by Aether"
 
-    - name: Add target host to in-memory inventory
+    - name: Wait for instance to be running (up to 60s)
-      ansible.builtin.add_host:
+      ansible.builtin.uri:
-        name: "{{ vm_ip }}"
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
-        groups: target
+        method: GET
-        ansible_user: "{{ ssh_user }}"
+        client_cert: "{{ incus_cert }}"
-        ansible_host: "{{ vm_ip }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: instance_state
+      until: instance_state.json.metadata.status == "Running"
+      retries: 12
+      delay: 5
 
-- name: Wait for instance and apply base configuration
+    - name: Generate setup script
-  hosts: target
+      ansible.builtin.set_fact:
-  gather_facts: false
+        setup_script: |
+          #!/bin/bash
+          set -e
 
-  pre_tasks:
+          echo "=== Post-deploy configuration starting ==="
-    - name: Wait for SSH connectivity (up to 5 minutes)
-      ansible.builtin.wait_for_connection:
-        delay: 5
-        timeout: 300
 
-    - name: Gather facts after connection
+          # Set hostname
-      ansible.builtin.setup:
+          hostname {{ ffsdn_instance_name }}
+          echo "{{ ffsdn_instance_name }}" > /etc/hostname
+          sed -i '/^127\.0\.1\.1/d' /etc/hosts
+          echo "127.0.1.1 {{ ffsdn_instance_name }}" >> /etc/hosts
+          echo "[OK] Hostname set to {{ ffsdn_instance_name }}"
 
-  roles:
+          # Update package cache and install base packages
-    - role: base-config
+          export DEBIAN_FRONTEND=noninteractive
-      vars:
+          apt-get update -qq
-        hostname: "{{ hostvars['localhost']['vm_name'] }}"
+          apt-get install -y -qq {{ base_packages }} 2>&1 | tail -1
-        deploy_environment: "{{ hostvars['localhost']['environment_name'] }}"
+          echo "[OK] Base packages installed"
-        deploy_owner: "{{ hostvars['localhost']['owner'] }}"
-        packages: "{{ hostvars['localhost']['base_packages'] }}"
 
-- name: Log deployment
+          # Set timezone
-  hosts: localhost
+          if command -v timedatectl >/dev/null 2>&1; then
-  gather_facts: false
+            timedatectl set-timezone UTC
-  connection: local
+          else
+            ln -sf /usr/share/zoneinfo/UTC /etc/localtime
+            echo "UTC" > /etc/timezone
+          fi
+          echo "[OK] Timezone set to UTC"
+
+          # Configure unattended upgrades
+          cat > /etc/apt/apt.conf.d/20auto-upgrades << 'AUTOUPGRADE'
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          AUTOUPGRADE
+          echo "[OK] Unattended upgrades configured"
+
+          # Write deployment metadata
+          cat > /etc/deploy-info << DEPLOYINFO
+          hostname={{ ffsdn_instance_name }}
+          ip={{ ffsdn_instance_ip | default('unknown') }}
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
+          deployed=$(date -Iseconds)
+          DEPLOYINFO
+          echo "[OK] Deployment metadata written to /etc/deploy-info"
+
+          echo "=== Post-deploy configuration complete ==="
+
+    - name: Push setup script to instance
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/post-deploy-setup.sh"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body: "{{ setup_script }}"
+        headers:
+          Content-Type: "application/octet-stream"
+          X-Incus-type: "file"
+          X-Incus-mode: "0755"
+        status_code: [200]
+
+    - name: Execute setup script via Incus API
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command: ["/bin/bash", "/tmp/post-deploy-setup.sh"]
+          record-output: true
+          interactive: false
+          wait-for-websocket: false
+          environment:
+            DEBIAN_FRONTEND: "noninteractive"
+        status_code: [202]
+        return_content: true
+      register: exec_result
+
+    - name: Wait for setup script to complete (up to 5 minutes)
+      ansible.builtin.uri:
+        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: exec_wait
+      failed_when: exec_wait.json.metadata.status != "Success"
+
+    - name: Verify exit code
+      ansible.builtin.assert:
+        that:
+          - exec_wait.json.metadata.metadata.return | int == 0
+        fail_msg: "Setup script exited with code {{ exec_wait.json.metadata.metadata.return }}"
+        success_msg: "Setup completed successfully (exit code 0)"
+
+    - name: Verify deployment metadata was written
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
+        method: GET
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        return_content: true
+      register: deploy_info
+
+    - name: Display deployment metadata
+      ansible.builtin.debug:
+        msg: "{{ deploy_info.content }}"
+
+    - name: Clean up setup script
+      ansible.builtin.uri:
+        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
+        method: POST
+        client_cert: "{{ incus_cert }}"
+        client_key: "{{ incus_key }}"
+        validate_certs: false
+        body_format: json
+        body:
+          command: ["/bin/rm", "-f", "/tmp/post-deploy-setup.sh"]
+          record-output: false
+          interactive: false
+          wait-for-websocket: false
+        status_code: [202]
+      ignore_errors: true
 
-  tasks:
     - name: Record deployment in ledger
       ansible.builtin.lineinfile:
-        path: /var/log/awx-deploy-ledger.log
+        path: /tmp/awx-deploy-ledger.log
         line: >-
-          {{ ansible_date_time.iso8601 | default(lookup('pipe', 'date -Iseconds')) }}
+          {{ lookup('pipe', 'date -Iseconds') }}
-          DEPLOY vm_name={{ vm_name }} vm_ip={{ vm_ip }}
+          DEPLOY instance={{ ffsdn_instance_name | default('unknown') }}
-          environment={{ environment_name }} owner={{ owner }}
+          ip={{ ffsdn_instance_ip | default('') }}
-          cost_center={{ cost_center }} status=success
+          cluster={{ ffsdn_cluster_name | default('unknown') }}
+          deployed_by={{ ffsdn_deployed_by | default('unknown') }}
+          image={{ ffsdn_image_os | default('') }}/{{ ffsdn_image_release | default('') }}
+          status=success
         create: true
         mode: "0644"

ansible/roles/base-config/tasks/main.yml

@@ -1,53 +0,0 @@
----
-# base-config — Minimal post-deploy configuration for new instances
-#
-# Variables (passed from post-deploy.yml):
-#   hostname          — desired hostname
-#   deploy_environment — dev/staging/prod
-#   deploy_owner      — owner tag
-#   packages          — list of packages to install
-
-- name: Set hostname
-  ansible.builtin.hostname:
-    name: "{{ hostname }}"
-
-- name: Add hostname to /etc/hosts
-  ansible.builtin.lineinfile:
-    path: /etc/hosts
-    regexp: '^127\.0\.1\.1'
-    line: "127.0.1.1 {{ hostname }}"
-
-- name: Update apt cache
-  ansible.builtin.apt:
-    update_cache: true
-    cache_valid_time: 3600
-  when: ansible_os_family == "Debian"
-
-- name: Install base packages
-  ansible.builtin.apt:
-    name: "{{ packages }}"
-    state: present
-  when: ansible_os_family == "Debian"
-
-- name: Set timezone to UTC
-  community.general.timezone:
-    name: UTC
-
-- name: Enable unattended security upgrades
-  ansible.builtin.copy:
-    dest: /etc/apt/apt.conf.d/20auto-upgrades
-    content: |
-      APT::Periodic::Update-Package-Lists "1";
-      APT::Periodic::Unattended-Upgrade "1";
-    mode: "0644"
-  when: ansible_os_family == "Debian"
-
-- name: Add deployment metadata to /etc/deploy-info
-  ansible.builtin.copy:
-    dest: /etc/deploy-info
-    content: |
-      hostname={{ hostname }}
-      environment={{ deploy_environment }}
-      owner={{ deploy_owner }}
-      deployed={{ ansible_date_time.iso8601 }}
-    mode: "0644"

incusos/deploy-awx

@@ -212,10 +212,16 @@ vm_exec() {
     incus exec "$(vm_ref)" -- "$@"
 }
 
+awx_url() {
+    # AWX is exposed via K3s NodePort on port 30080 (HTTP)
+    # Traefik ingress returns 404 for IP-based access
+    echo "http://${AWX_IP}:30080"
+}
+
 awx_api() {
     local method="$1" path="$2"
     shift 2
-    curl -sk -X "$method" "https://${AWX_IP}${path}" \
+    curl -sk -X "$method" "$(awx_url)${path}" \
         -H "Content-Type: application/json" \
         "$@"
 }
@@ -225,7 +231,7 @@ awx_api_auth() {
     shift 2
     local token
     token=$(get_awx_admin_password)
-    curl -sk -X "$method" "https://${AWX_IP}${path}" \
+    curl -sk -X "$method" "$(awx_url)${path}" \
         --user "admin:${token}" \
         -H "Content-Type: application/json" \
         "$@"
@@ -255,7 +261,7 @@ wait_for_awx_web() {
     local timeout="${1:-600}"
     local elapsed=0
     info "Waiting for AWX web interface (up to ${timeout}s)..."
-    while ! curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | grep -q '200'; do
+    while ! curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null | grep -q '200'; do
         sleep 10
         elapsed=$((elapsed + 10))
         if [[ $elapsed -ge $timeout ]]; then
@@ -390,8 +396,8 @@ action_deploy() {
 
     echo
     success "AWX deployment complete!"
-    info "  Web UI:   https://${AWX_IP}/"
+    info "  Web UI:   $(awx_url)/"
-    info "  API:      https://${AWX_IP}/api/v2/ping/"
+    info "  API:      $(awx_url)/api/v2/ping/"
     local pw
     pw=$(get_awx_admin_password 2>/dev/null || echo "<retrieve with --status>")
     info "  Admin:    admin / ${pw}"
@@ -436,23 +442,28 @@ phase_configure_network() {
         return 0
     fi
 
+    # Debian 12 images use systemd-networkd (not netplan)
+    # The macvlan NIC appears as enp5s0 in Incus VMs
     vm_exec bash -c "
         hostnamectl set-hostname '${VM_NAME}'
 
-        cat > /etc/netplan/50-static.yaml << 'NETPLAN'
+        # Disable any existing DHCP config
-network:
+        rm -f /etc/network/interfaces.d/* 2>/dev/null || true
-  version: 2
+
-  ethernets:
+        # Configure static IP via systemd-networkd
-    enp5s0:
+        mkdir -p /etc/systemd/network
-      addresses: [${AWX_IP}/${AWX_PREFIX}]
+        cat > /etc/systemd/network/10-static.network << 'NETWORKD'
-      routes:
+[Match]
-        - to: default
+Name=enp5s0
-          via: ${AWX_GATEWAY}
+
-      nameservers:
+[Network]
-        addresses: [${AWX_DNS}]
+Address=${AWX_IP}/${AWX_PREFIX}
-NETPLAN
+Gateway=${AWX_GATEWAY}
-        chmod 600 /etc/netplan/50-static.yaml
+DNS=${AWX_DNS}
-        netplan apply
+NETWORKD
+
+        systemctl enable systemd-networkd
+        systemctl restart systemd-networkd
     "
 
     success "Static IP ${AWX_IP}/${AWX_PREFIX} configured"
@@ -474,6 +485,14 @@ phase_install_k3s() {
         return 0
     fi
 
+    # Ensure curl is available (minimal Debian 12 may not have it)
+    info "Installing prerequisites..."
+    vm_exec bash -c "
+        export DEBIAN_FRONTEND=noninteractive
+        apt-get update -qq
+        apt-get install -y -qq curl python3 2>&1 | tail -1
+    "
+
     vm_exec bash -c "
         curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
     "
@@ -572,9 +591,9 @@ phase_verify() {
     admin_pw=$(get_awx_admin_password)
     success "Admin password retrieved"
 
-    # Test API ping
+    # Test API ping (NodePort 30080)
     local ping_result
-    ping_result=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null)
+    ping_result=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null)
     if echo "$ping_result" | python3 -c "import sys,json; json.load(sys.stdin)" &>/dev/null; then
         success "AWX API responding"
     else
@@ -629,11 +648,11 @@ action_status() {
         warn "Pods: ${running}/${total} running"
     fi
 
-    # AWX web
+    # AWX web (NodePort 30080)
     local http_code
-    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
+    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
     if [[ "$http_code" == "200" ]]; then
-        success "AWX API: responding (HTTP ${http_code})"
+        success "AWX API: responding at $(awx_url) (HTTP ${http_code})"
     else
         error "AWX API: not responding (HTTP ${http_code})"
     fi
@@ -649,7 +668,7 @@ action_status() {
 
     # AWX version
     local version
-    version=$(curl -sk "https://${AWX_IP}/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
+    version=$(curl -sk "$(awx_url)/api/v2/ping/" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))" 2>/dev/null || echo "unknown")
     info "AWX version: ${version}"
 
     # Aether connectivity
@@ -707,10 +726,10 @@ action_heal() {
         success "All pods healthy"
     fi
 
-    # 3. AWX web responsiveness
+    # 3. AWX web responsiveness (NodePort 30080)
     info "Checking AWX web..."
     local http_code
-    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "https://${AWX_IP}/api/v2/ping/" 2>/dev/null || echo "000")
+    http_code=$(curl -sk -o /dev/null -w '%{http_code}' "$(awx_url)/api/v2/ping/" 2>/dev/null || echo "000")
     if [[ "$http_code" == "200" ]]; then
         success "AWX web: responding"
     else
@@ -763,98 +782,85 @@ action_configure() {
     awx_auth() {
         local method="$1" path="$2"
         shift 2
-        curl -sk -X "$method" "https://${AWX_IP}${path}" \
+        curl -sk -X "$method" "$(awx_url)${path}" \
             --user "admin:${admin_pw}" \
             -H "Content-Type: application/json" \
             "$@"
     }
 
-    # 1. Create SCM credential for private Git repo
+    # 1. Push Incus client cert/key to AWX project directory
-    step "Creating SCM credential"
+    #    Playbooks use Incus REST API (not SSH) — cert is required.
-    local ssh_key=""
+    #    AWX EE mounts project dir at /runner/project/ during job execution.
-    local ssh_key_path="${HOME}/.ssh/id_ed25519"
+    step "Pushing Incus client certificate to AWX"
-    [[ ! -f "$ssh_key_path" ]] && ssh_key_path="${HOME}/.ssh/id_rsa"
+    local incus_cert="${HOME}/.config/incus/client.crt"
-    if [[ -f "$ssh_key_path" ]]; then
+    local incus_key="${HOME}/.config/incus/client.key"
-        ssh_key=$(cat "$ssh_key_path")
+    if [[ -f "$incus_cert" && -f "$incus_key" ]]; then
-        info "Using SSH key: ${ssh_key_path}"
+        local task_pod
-    else
+        task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
-        warn "No SSH key found at ~/.ssh/id_ed25519 or ~/.ssh/id_rsa"
+            -l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
-        warn "SCM credential will be created without a key — update manually in AWX"
+        if [[ -n "$task_pod" ]]; then
-    fi
+            local project_dir="/var/lib/awx/projects/incus-contrib"
-
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "$project_dir"
-    local scm_cred_id
+            # Push cert via base64 to avoid file push permission issues
-    scm_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-contrib-scm" 2>/dev/null \
+            local cert_b64 key_b64
-        | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
+            cert_b64=$(base64 -w0 "$incus_cert")
-
+            key_b64=$(base64 -w0 "$incus_key")
-    if [[ -z "$scm_cred_id" ]]; then
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
-        if [[ -n "$ssh_key" ]]; then
+                bash -c "echo '${cert_b64}' | base64 -d > ${project_dir}/incus-client.crt"
-            scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
-                -d "$(python3 -c "
+                bash -c "echo '${key_b64}' | base64 -d > ${project_dir}/incus-client.key"
-import json, sys
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
-print(json.dumps({
+                chmod 600 "${project_dir}/incus-client.key"
-    'name': 'incus-contrib-scm',
+            success "Incus client cert/key pushed to AWX task pod"
-    'organization': 1,
-    'credential_type': 2,
-    'inputs': {'ssh_key_data': sys.stdin.read()}
-}))" <<< "$ssh_key")" 2>/dev/null \
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
         else
-            scm_cred_id=$(awx_auth POST "/api/v2/credentials/" \
+            warn "Cannot find AWX task pod — push cert manually later"
-                -d '{"name":"incus-contrib-scm","organization":1,"credential_type":2,"inputs":{}}' 2>/dev/null \
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
         fi
-        [[ -n "$scm_cred_id" ]] && success "SCM credential created (ID: ${scm_cred_id})" || warn "Failed to create SCM credential"
     else
-        success "SCM credential exists (ID: ${scm_cred_id})"
+        warn "Incus client cert not found at ${incus_cert}"
+        warn "Playbooks need the cert to talk to Incus API — push it manually"
     fi
 
-    # 2. Create project
+    # 2. Create manual project (local_path)
+    #    AWX EE containers cannot reach the private git repo, so we use a
+    #    manual project with files pushed directly to the task pod.
     step "Creating project"
     local project_id
     project_id=$(awx_auth GET "/api/v2/projects/?name=incus-contrib" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
     if [[ -z "$project_id" ]]; then
-        local project_payload
+        project_id=$(awx_auth POST "/api/v2/projects/" \
-        project_payload=$(python3 -c "
+            -d '{"name":"incus-contrib","organization":1,"scm_type":"","local_path":"incus-contrib"}' 2>/dev/null \
-import json
-d = {
-    'name': 'incus-contrib',
-    'organization': 1,
-    'scm_type': 'git',
-    'scm_url': '${GIT_REPO}',
-    'scm_branch': '${GIT_BRANCH}',
-    'scm_update_on_launch': True
-}
-print(json.dumps(d))")
-        [[ -n "$scm_cred_id" ]] && project_payload=$(echo "$project_payload" | python3 -c "
-import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(json.dumps(d))")
-
-        project_id=$(awx_auth POST "/api/v2/projects/" -d "$project_payload" 2>/dev/null \
             | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
-        [[ -n "$project_id" ]] && success "Project created (ID: ${project_id})" || warn "Failed to create project"
+        [[ -n "$project_id" ]] && success "Project created (ID: ${project_id}, local_path)" || warn "Failed to create project"
     else
         success "Project exists (ID: ${project_id})"
     fi
 
-    # Wait for initial project sync
+    # Push playbooks to AWX task pod project directory
     if [[ -n "$project_id" ]]; then
-        info "Waiting for project sync..."
+        local task_pod
-        local sync_elapsed=0
+        task_pod=$(vm_exec kubectl -n "$AWX_NAMESPACE" get pods \
-        while [[ $sync_elapsed -lt 120 ]]; do
+            -l app.kubernetes.io/component=awx-task -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
-            local status
+        if [[ -n "$task_pod" ]]; then
-            status=$(awx_auth GET "/api/v2/projects/${project_id}/" 2>/dev/null \
+            local project_dir="/var/lib/awx/projects/incus-contrib"
-                | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || true)
+            vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- mkdir -p "${project_dir}/playbooks"
-            if [[ "$status" == "successful" ]]; then
+            for playbook in post-deploy.yml decommission.yml; do
-                success "Project synced"
+                local pb_path="${SCRIPT_DIR}/../ansible/playbooks/${playbook}"
-                break
+                if [[ -f "$pb_path" ]]; then
-            elif [[ "$status" == "failed" || "$status" == "error" ]]; then
+                    local pb_b64
-                warn "Project sync failed — check SCM credential and repo URL"
+                    pb_b64=$(base64 -w0 "$pb_path")
-                break
+                    vm_exec kubectl -n "$AWX_NAMESPACE" exec "$task_pod" -- \
-            fi
+                        bash -c "echo '${pb_b64}' | base64 -d > ${project_dir}/playbooks/${playbook}"
-            sleep 5
+                    detail "Pushed ${playbook}"
-            sync_elapsed=$((sync_elapsed + 5))
+                else
-        done
+                    warn "Playbook not found: ${pb_path}"
+                fi
+            done
+            success "Playbooks pushed to AWX project directory"
+        else
+            warn "Cannot find AWX task pod — push playbooks manually"
+        fi
     fi
 
     # 3. Create inventory
@@ -872,26 +878,16 @@ import sys,json; d=json.load(sys.stdin); d['credential']=${scm_cred_id}; print(j
         success "Inventory exists (ID: ${inventory_id})"
     fi
 
-    # 4. Create machine credential (SSH to instances)
+    # 4. Create machine credential (required by AWX templates, though
+    #    playbooks use Incus API not SSH)
     step "Creating machine credential"
     local machine_cred_id
     machine_cred_id=$(awx_auth GET "/api/v2/credentials/?name=incus-instances" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
     if [[ -z "$machine_cred_id" ]]; then
-        local machine_cred_payload='{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}'
+        machine_cred_id=$(awx_auth POST "/api/v2/credentials/" \
-        if [[ -n "$ssh_key" ]]; then
+            -d '{"name":"incus-instances","organization":1,"credential_type":1,"inputs":{"username":"root"}}' 2>/dev/null \
-            machine_cred_payload=$(python3 -c "
-import json, sys
-key = sys.stdin.read()
-print(json.dumps({
-    'name': 'incus-instances',
-    'organization': 1,
-    'credential_type': 1,
-    'inputs': {'username': 'root', 'ssh_key_data': key}
-}))" <<< "$ssh_key")
-        fi
-        machine_cred_id=$(awx_auth POST "/api/v2/credentials/" -d "$machine_cred_payload" 2>/dev/null \
             | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
         [[ -n "$machine_cred_id" ]] && success "Machine credential created (ID: ${machine_cred_id})" || warn "Failed to create machine credential"
     else
@@ -971,7 +967,7 @@ action_join_aether() {
     admin_pw=$(get_awx_admin_password)
 
     local pat_response
-    pat_response=$(curl -sk -X POST "https://${AWX_IP}/api/v2/users/1/personal_tokens/" \
+    pat_response=$(curl -sk -X POST "$(awx_url)/api/v2/users/1/personal_tokens/" \
         --user "admin:${admin_pw}" \
         -H "Content-Type: application/json" \
         -d '{"description": "Aether integration", "scope": "write"}' 2>/dev/null)
@@ -988,10 +984,10 @@ action_join_aether() {
 
     # 2. Get template IDs
     local post_deploy_id decommission_id
-    post_deploy_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=post-deploy" \
+    post_deploy_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=post-deploy" \
         --user "admin:${admin_pw}" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
-    decommission_id=$(curl -sk "https://${AWX_IP}/api/v2/job_templates/?name=decommission" \
+    decommission_id=$(curl -sk "$(awx_url)/api/v2/job_templates/?name=decommission" \
         --user "admin:${admin_pw}" 2>/dev/null \
         | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['results'][0]['id'] if r.get('count',0)>0 else '')" 2>/dev/null || true)
 
@@ -1005,7 +1001,7 @@ action_join_aether() {
     info "2. Navigate to Settings → Ansible Automation (/awx-endpoints)"
     info "3. Add Endpoint:"
     info "     Name:       lab-awx"
-    info "     URL:        https://${AWX_IP}"
+    info "     URL:        $(awx_url)"
     info "     Token:      ${awx_token}"
     info "     Verify SSL: unchecked"
     info "4. Configure cluster AWX binding:"
@@ -1023,7 +1019,7 @@ action_join_aether() {
         detail "  -H 'Referer: ${AETHER_URL}/' \\"
         detail "  -X POST ${AETHER_URL}/api/awx/endpoints \\"
         detail "  -H 'Content-Type: application/json' \\"
-        detail "  -d '{\"name\":\"lab-awx\",\"url\":\"https://${AWX_IP}\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
+        detail "  -d '{\"name\":\"lab-awx\",\"url\":\"http://${AWX_IP}:30080\",\"token\":\"${awx_token}\",\"verify_ssl\":false}'"
         detail ""
         detail "# Bind cluster (replace ENDPOINT_ID):"
         detail "curl -sSk -b cookies.txt -H 'X-CSRF-Token: \$CSRF' \\"