incus-contrib/notes/awx-guide.md

AWX — Ansible Automation for Aether Lifecycle Hooks

AWX (the open-source upstream of Ansible Tower) provides a web UI and REST API for running Ansible playbooks. Aether integrates with AWX to run post-deploy and decommission playbooks as lifecycle hooks on every instance create/delete.

This guide covers deploying AWX on the Incus cluster, connecting it to Aether, writing lifecycle playbooks, and troubleshooting.

Architecture

  User deploys instance via Aether UI
          |
          v
  Aether creates instance on Incus cluster
          |
          v
  Aether calls AWX API: POST /api/v2/job_templates/{id}/launch/
    with extra_vars: ffsdn_instance_name, ffsdn_instance_ip, ...
          |
          v
  AWX runs post-deploy.yml playbook
    1. Pushes setup script to instance (Incus file API)
    2. Executes script (Incus exec API)
    3. Verifies /etc/deploy-info was written
    4. Logs to deployment ledger
          |
          v
  Aether marks deployment complete (or rolls back on failure)

Decommission works in reverse — Aether triggers the decommission template before deleting the instance. Decommission failures do NOT block deletion.

Separation of concerns

Layer	Tool	Responsibility
Infrastructure	Aether + Incus	Instance creation, networking, resources, lifecycle
Configuration	AWX + Ansible	OS config, packages, services, integrations
Automation glue	Aether AWX binding	Triggers playbooks at create/delete

What Aether passes to AWX (validated)

Every AWX job launched by Aether receives these ffsdn_ prefixed extra vars:

Variable	Description	Example
ffsdn_instance_name	Instance name	awx-e2e-test
ffsdn_instance_ip	IP address assigned by Aether	10.207.217.5
ffsdn_cluster_id	Numeric cluster ID	52
ffsdn_cluster_name	Cluster display name	oc-lab-cluster
ffsdn_deployed_by	Aether username who triggered deploy	admin
ffsdn_image_os	Image OS	Debian
ffsdn_image_release	Image release	bookworm
ffsdn_image_alias	Image alias (may be empty)	""

Important: Aether does NOT pass vm_name, vm_ip, environment, owner, or cost_center. The original plan assumed these names, but real testing revealed Aether uses the ffsdn_ prefix exclusively.

AWX API endpoints Aether uses

Endpoint	Purpose
GET /api/v2/ping/	Health check
GET /api/v2/job_templates/{id}/	Validate template exists
POST /api/v2/job_templates/{id}/launch/	Trigger job with extra_vars
GET /api/v2/jobs/{id}/	Poll job status

Lifecycle behavior

Hook	Trigger	On failure
Post-deploy	After instance creation	Aether auto-rollbacks (deletes instance)
Decommission	Before instance deletion	Failure does NOT block deletion

Aether polls GET /api/v2/jobs/{id}/ until the job reaches a terminal state (successful, failed, error) or the timeout expires. Post-deploy rollback means the instance is deleted — the user sees "Ansible job N finished with status: failed. Instance has been deleted."

Lab deployment

Resource requirements

Resource	Recommended	Minimum
vCPU	4	2
RAM	8 GiB	4 GiB
Disk	40 GiB	20 GiB

K3s uses ~600 MiB, AWX pods ~3 GiB, PostgreSQL ~400 MiB. The web pod was originally set to 1 GiB limit but OOMKilled; 2 GiB is the tested minimum.

Lab details

Setting	Value
VM name	awx
Location	oc-node-02
IP	192.168.102.161/22 (VLAN 69)
Gateway	192.168.100.1
DNS	192.168.100.1
AWX API port	30080 (K3s NodePort)
OS	Debian 12
K8s	K3s (single-node)
AWX	24.6.1
AWX Operator	2.19.1

Note: AWX is exposed via K3s NodePort on port 30080, NOT via Traefik ingress. The ingress returns 404 when accessed by IP (requires hostname). NodePort works reliably with IP-based access.

Manual deployment (tested)

Step 1: Create VM

# Launch Debian 12 VM on the cluster
incus launch images:debian/12 oc-node-02:awx --vm \
    --target oc-node-02 \
    -c limits.cpu=4 -c limits.memory=8GiB \
    -d root,size=40GiB

# Replace default NIC with macvlan for direct VLAN access
# IMPORTANT: use 'config device add' not 'profile device remove'
# Profile device removal fails on cluster members
incus config device add oc-node-02:awx eth0 nic \
    nictype=macvlan parent=mgmt

Bug found: incus profile device remove fails on cluster members with "Profile device eth0 not found" because profiles are cluster-wide. Use incus config device add instead, which creates an instance-level override that takes priority over the profile.

Step 2: Configure static IP

Debian 12 uses systemd-networkd, not netplan. The original plan assumed netplan but that's Ubuntu-only.

incus exec oc-node-02:awx -- bash -c "
    hostnamectl set-hostname awx

    # Debian 12 uses systemd-networkd
    cat > /etc/systemd/network/10-static.network << 'NETCFG'
[Match]
Name=enp5s0

[Network]
Address=192.168.102.161/22
Gateway=192.168.100.1
DNS=192.168.100.1
NETCFG

    systemctl restart systemd-networkd
"

After IP change, wait a few seconds for the new IP to take effect.

Step 3: Install prerequisites and K3s

# Debian 12 minimal doesn't have curl or git
incus exec oc-node-02:awx -- bash -c "
    apt-get update && apt-get install -y curl git

    curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
    kubectl wait --for=condition=Ready node --all --timeout=120s
"

Step 4: Deploy AWX Operator

incus exec oc-node-02:awx -- bash -c "
    kubectl create namespace awx

    mkdir -p /opt/awx/operator
    cat > /opt/awx/operator/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.19.1
namespace: awx
EOF
    kubectl apply -k /opt/awx/operator/

    # Wait for operator (~2-3 min)
    kubectl -n awx wait --for=condition=Available \
        deployment/awx-operator-controller-manager --timeout=300s
"

Step 5: Deploy AWX instance

The AWX custom resource is defined in incusos/awx-manifests/base/awx.yaml. Key settings discovered during testing:

• PVC: must use ReadWriteOnce (not ReadWriteMany) for K3s local-path
• Web pod memory: 2 GiB limit minimum (1 GiB causes OOMKill)
• Service type: NodePort on port 30080 (ingress returns 404 for IP access)

# Push manifests to VM and apply
incus exec oc-node-02:awx -- mkdir -p /opt/awx/base

# Push awx.yaml and kustomization.yaml (from incusos/awx-manifests/base/)
# Then apply:
incus exec oc-node-02:awx -- kubectl apply -k /opt/awx/base/

# Wait 5-10 min for all pods to start
incus exec oc-node-02:awx -- kubectl -n awx get pods -w

Expected final pod state:

awx-operator-controller-manager-xxx   2/2     Running
awx-postgres-15-0                     1/1     Running
awx-task-xxx                          4/4     Running
awx-web-xxx                           3/3     Running

Step 6: Verify

# Get admin password
ADMIN_PW=$(incus exec oc-node-02:awx -- kubectl -n awx get secret \
    awx-admin-password -o jsonpath='{.data.password}' | base64 -d)
echo "Admin password: $ADMIN_PW"

# Test API via NodePort
curl -sk http://192.168.102.161:30080/api/v2/ping/

AWX web UI: http://192.168.102.161:30080/ — login with admin / password above.

AWX configuration (tested)

Project (manual, no Git dependency)

AWX Git-based projects require SSH key configuration and network access to the Git server. For simplicity, use a manual project — playbook files are pushed directly to the AWX task pod:

# Create manual project via API
AWX_TOKEN="..."  # from PAT creation below
curl -sk -X POST http://192.168.102.161:30080/api/v2/projects/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "incus-contrib",
        "organization": 1,
        "scm_type": "",
        "local_path": "incus-contrib"
    }'

Then push playbooks to the task pod:

AWX_POD=$(incus exec oc-node-02:awx -- kubectl -n awx get pods \
    -l app.kubernetes.io/name=awx-task -o jsonpath='{.items[0].metadata.name}')

# Create project directory structure
incus exec oc-node-02:awx -- kubectl -n awx exec $AWX_POD -- \
    mkdir -p /var/lib/awx/projects/incus-contrib/playbooks

# Push playbooks (via base64 to handle stdin issues)
B64=$(base64 -w0 ansible/playbooks/post-deploy.yml)
incus exec oc-node-02:awx -- bash -c "echo '$B64' | base64 -d > /root/post-deploy.yml"
incus exec oc-node-02:awx -- kubectl -n awx cp /root/post-deploy.yml \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/playbooks/post-deploy.yml

Important: also push the Incus client certificate and key to the project directory — the playbooks use these for Incus API calls:

incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.crt \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.crt
incus exec oc-node-02:awx -- kubectl -n awx cp /root/incus-client.key \
    $AWX_POD:/var/lib/awx/projects/incus-contrib/incus-client.key

Inventory

The playbooks use localhost exclusively (no SSH, Incus API only). The inventory is a placeholder required by AWX:

curl -sk -X POST http://192.168.102.161:30080/api/v2/inventories/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"name": "incus-instances", "organization": 1}'

Credentials

A machine credential (SSH key) is created but only used as a fallback. The primary playbook approach uses the Incus REST API with client certificates, not SSH.

# Generate SSH key on AWX VM
incus exec oc-node-02:awx -- ssh-keygen -t ed25519 -f /root/.ssh/awx_key -N ""

# Create machine credential in AWX
PRIV_KEY=$(incus exec oc-node-02:awx -- cat /root/.ssh/awx_key)
curl -sk -X POST http://192.168.102.161:30080/api/v2/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
        \"name\": \"incus-instances\",
        \"organization\": 1,
        \"credential_type\": 1,
        \"inputs\": {\"username\": \"root\", \"ssh_key_data\": $(echo "$PRIV_KEY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}
    }"

Job templates

Template	Playbook	Template ID
post-deploy	playbooks/post-deploy.yml	10
decommission	playbooks/decommission.yml	11

Both must have ask_variables_on_launch: true:

curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "post-deploy",
        "organization": 1,
        "project": 6,
        "playbook": "playbooks/post-deploy.yml",
        "inventory": 2,
        "ask_variables_on_launch": true
    }'

Attach credentials to templates (required even if playbook doesn't use SSH):

curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/credentials/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"id": 4}'

Personal Access Token

AWX_TOKEN=$(curl -sk -X POST http://192.168.102.161:30080/api/v2/users/1/personal_tokens/ \
    --user "admin:${ADMIN_PW}" \
    -H "Content-Type: application/json" \
    -d '{"description": "Aether integration", "scope": "write"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

Aether integration

Register AWX endpoint

Via Aether API (requires session cookies + CSRF token):

curl -sSk -b /tmp/aether-cookies.txt \
    -H "X-CSRF-Token: $CSRF" \
    -H "Referer: https://192.168.102.160:8443/" \
    -X POST https://192.168.102.160:8443/api/awx/endpoints \
    -H "Content-Type: application/json" \
    -d '{
        "name": "lab-awx",
        "url": "http://192.168.102.161:30080",
        "token": "'$AWX_TOKEN'",
        "verify_ssl": false
    }'

Tested result: endpoint ID 2 created, AWX health shows "healthy" at /api/health/awx:

{"awx_healthy":true,"awx_version":"24.6.1","awx_status":"ok"}

Bind cluster to AWX

Known bug: PUT /api/clusters/{id}/awx-config returns {"error":"Invalid cluster ID"} for all valid cluster IDs. This Aether API endpoint appears non-functional in v6.4.317.

Workaround: direct PostgreSQL update on the Aether database:

incus exec oc-node-01:aether -- bash -c "
    docker exec aether-postgres psql -U aether -d incusovnsdnc -c \"
        UPDATE incus_ovn_clusters SET
            awx_endpoint_id = 2,
            awx_post_deploy_template_id = 10,
            awx_decommission_template_id = 11,
            awx_job_timeout_seconds = 600
        WHERE cluster_id = 52;
    \"
"

Verify integration

# Check Aether's AWX health endpoint
curl -sSk -b /tmp/aether-cookies.txt \
    https://192.168.102.160:8443/api/health/awx

# Deploy a test instance via Aether → AWX job triggers automatically
# Job history visible at: /deploy page → "Your Recent Ansible Automation Jobs"

Writing playbooks

Key design decision: Incus API, not SSH

The playbooks use the Incus REST API (file push + exec) instead of SSH for all instance configuration. This is necessary because:

1. Bridge network isolation: containers on incusbr0 (10.207.217.0/24) are not routable from AWX (192.168.102.161). The bridge is NAT'd and IncusOS nodes don't forward inbound traffic to the bridge.
2. No SSH dependency: fresh Debian 12 containers don't have python3 installed. With SSH, a raw bootstrap step is needed first.
3. Works with any network: the Incus API is accessible at the cluster node level (192.168.102.140-142:8443), regardless of which overlay or bridge network the container is on.

The trade-off: playbooks require an Incus client certificate in the AWX project directory, and the playbook pattern is less "Ansible-native" (no SSH connection plugins, no gather_facts, no modules running on the target).

File paths in AWX

AWX runs playbooks inside an Execution Environment (EE) container via receptor. During job execution:

Path	Description
/runner/project/	Project root (maps to the manual project directory)
/runner/project/playbooks/	Playbook directory
/runner/project/incus-client.crt	Incus client certificate
/runner/project/incus-client.key	Incus client key
/runner/artifacts/{job_id}/	Job artifacts (logs, SSH keys)

Important: files at /var/lib/awx/projects/ (the task pod) are NOT accessible at the same path during job execution. The EE mounts the project at /runner/project/.

post-deploy.yml pattern (tested, validated)

- name: Post-deploy — configure instance via Incus API
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    # 1. Validate required ffsdn_* variables from Aether
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - ffsdn_instance_name is defined
          - ffsdn_instance_name | length > 0

    # 2. Wait for instance to be Running
    - name: Wait for instance to be running
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/state"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: state
      until: state.json.metadata.status == "Running"
      retries: 12
      delay: 5

    # 3. Push setup script via Incus file API
    - name: Push setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/tmp/setup.sh"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body: "{{ setup_script }}"
        headers:
          Content-Type: "application/octet-stream"
          X-Incus-type: "file"
          X-Incus-mode: "0755"

    # 4. Execute via Incus exec API
    - name: Execute setup script
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        body_format: json
        body:
          command: ["/bin/bash", "/tmp/setup.sh"]
          record-output: true
          interactive: false
          wait-for-websocket: false
        status_code: [202]
      register: exec_result

    # 5. Wait for completion via operation API
    - name: Wait for script to complete
      ansible.builtin.uri:
        url: "{{ incus_api }}{{ exec_result.json.operation }}/wait?timeout=300"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
      register: exec_wait

    # 6. Verify exit code
    - name: Verify exit code
      ansible.builtin.assert:
        that:
          - exec_wait.json.metadata.metadata.return | int == 0

    # 7. Verify result via file API
    - name: Read /etc/deploy-info
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/files?path=/etc/deploy-info"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        return_content: true
      register: deploy_info

Incus API endpoints used by playbooks

Endpoint	Method	Purpose
/1.0/instances/{name}/state	GET	Check if instance is Running
/1.0/instances/{name}/files?path=...	POST	Push files to instance
/1.0/instances/{name}/files?path=...	GET	Read files from instance
/1.0/instances/{name}/exec	POST	Execute commands in instance
/1.0/operations/{uuid}/wait	GET	Wait for async operation
/1.0/instances/{name}	GET	Check if instance exists

Note: exec output (stdout/stderr from record-output: true) is NOT retrievable via the /1.0/instances/{name}/logs/ API. The log filename format (exec_UUID.stdout) is rejected as "not valid". Verify results by reading files written by the script instead.

decommission.yml pattern (tested, validated)

- name: Decommission — graceful shutdown and logging
  hosts: localhost
  gather_facts: false
  connection: local

  vars:
    incus_api: "https://192.168.102.140:8443"
    incus_cert: "/runner/project/incus-client.crt"
    incus_key: "/runner/project/incus-client.key"

  tasks:
    - name: Check if instance still exists
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}"
        client_cert: "{{ incus_cert }}"
        client_key: "{{ incus_key }}"
        validate_certs: false
        status_code: [200, 404]
      register: instance_check

    - name: Stop services (best-effort)
      ansible.builtin.uri:
        url: "{{ incus_api }}/1.0/instances/{{ ffsdn_instance_name }}/exec"
        method: POST
        # ... service stop command
      ignore_errors: true
      when: instance_check.status == 200

    - name: Record in ledger
      ansible.builtin.lineinfile:
        path: /tmp/awx-deploy-ledger.log
        # ... log entry

Bugs discovered during validation

Ansible reserved keyword: environment

Ansible's environment keyword is reserved — it sets process environment variables for tasks. If Aether passed environment: "lab" as an extra var, Ansible would interpret it as the keyword, resolving to [] instead of the string value. The fix was to use deploy_env instead. In practice, Aether uses ffsdn_* prefixes which avoid this entirely.

Recursive template loop with self-referencing vars

# BROKEN — causes infinite recursion if vm_ip is not provided
vars:
  vm_ip: "{{ vm_ip | default('') }}"

When the variable name matches the template reference AND the variable is not provided as an extra var, Ansible enters infinite recursion. The fix: don't self-reference. Use the ffsdn_* variables directly without redeclaring them in vars blocks.

AWX EE runs at /runner/project/, not /var/lib/awx/projects/

Files pushed to the AWX task pod at /var/lib/awx/projects/incus-contrib/ are mounted in the EE container at /runner/project/. All file paths in playbooks must use the /runner/project/ prefix.

Bridge network not routable from AWX VM

Containers on incusbr0 (10.207.217.0/24) are NOT reachable from the AWX VM (192.168.102.161). The bridge is NAT'd outbound only; IncusOS nodes don't forward inbound traffic from the management network to the bridge. Adding a static route on the AWX VM (ip route add 10.207.217.0/24 via 192.168.102.140) doesn't work because the node's nftables rules block non-established connections from external interfaces to the bridge.

Solution: use the Incus REST API instead of SSH. The cluster API (192.168.102.140:8443) is always reachable, and provides file push + exec endpoints that work regardless of instance network topology.

Aether cluster AWX config API bug

PUT /api/clusters/{id}/awx-config returns {"error":"Invalid cluster ID"} for all valid cluster IDs. Tested with different CSRF tokens, session cookies, request formats (JSON fields, POST vs PUT). The endpoint is not documented in Aether's swagger.yaml. Workaround: direct PostgreSQL UPDATE.

AWX web pod OOMKill at 1 GiB

The AWX web pod (nginx + uwsgi) needs at least 2 GiB memory limit. With 1 GiB, it starts successfully but gets OOMKilled under load. Set in the AWX CR: web_resource_requirements.limits.memory: 2Gi.

AWX PVC needs ReadWriteOnce

K3s local-path provisioner creates hostPath volumes which only support ReadWriteOnce. If the AWX CR specifies ReadWriteMany, the PVC stays Pending and PostgreSQL can't start. Set: postgres_storage_class: local-path and ensure ReadWriteOnce access mode.

Troubleshooting

Pod status

# List all AWX pods
incus exec oc-node-02:awx -- kubectl -n awx get pods -o wide

# Describe a failing pod
incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>

# View pod logs
incus exec oc-node-02:awx -- kubectl -n awx logs <pod-name>

Job output

# Get real-time job output (text format)
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/stdout/?format=txt \
    -H "Authorization: Bearer $AWX_TOKEN"

# Check job status
curl -sk http://192.168.102.161:30080/api/v2/jobs/{id}/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); \
        print('status:', d['status'], 'failed:', d['failed'])"

Common issues

AWX pods stuck in Pending: insufficient resources or PVC issue. Check:

incus exec oc-node-02:awx -- kubectl -n awx describe pod <pod-name>
incus exec oc-node-02:awx -- kubectl -n awx get pvc

Job fails with recursive template error: self-referencing var definition. Check playbook vars blocks for patterns like var: "{{ var | default('') }}".

Job fails with "No such file or directory": cert/key not at the correct path. Remember: EE uses /runner/project/, not /var/lib/awx/projects/.

Job fails with "Connection failure": Incus API not reachable from AWX pod, or client cert not trusted by cluster. Test from inside the pod:

incus exec oc-node-02:awx -- kubectl -n awx exec <task-pod> -c awx-ee -- \
    curl -sk --cert /runner/project/incus-client.crt \
    --key /runner/project/incus-client.key \
    https://192.168.102.140:8443/1.0

Aether deploy rolls back immediately: AWX job failed. Check the AWX job output for the specific error. Common causes: template misconfiguration, missing credential attachment, playbook syntax errors.

Useful commands

# AWX version
curl -sk http://192.168.102.161:30080/api/v2/ping/

# List job templates
curl -sk http://192.168.102.161:30080/api/v2/job_templates/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{t[\"id\"]:3d} {t[\"name\"]}') \
        for t in json.load(sys.stdin)['results']]"

# List recent jobs
curl -sk "http://192.168.102.161:30080/api/v2/jobs/?order_by=-id&page_size=5" \
    -H "Authorization: Bearer $AWX_TOKEN" \
    | python3 -c "import sys,json; [print(f'{j[\"id\"]:4d} {j[\"status\"]:12s} {j[\"name\"]}') \
        for j in json.load(sys.stdin)['results']]"

# Manually trigger post-deploy
curl -sk -X POST http://192.168.102.161:30080/api/v2/job_templates/10/launch/ \
    -H "Authorization: Bearer $AWX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"extra_vars": {
        "ffsdn_instance_name": "test-ct",
        "ffsdn_instance_ip": "10.207.217.99",
        "ffsdn_cluster_name": "oc-lab-cluster",
        "ffsdn_deployed_by": "admin",
        "ffsdn_image_os": "Debian",
        "ffsdn_image_release": "bookworm"
    }}'

Rollback

If AWX is broken beyond repair:

incus delete oc-node-02:awx --force
# Then redeploy from Step 1

The AWX VM is a standalone workload with no cluster dependencies. Destroying and redeploying has zero impact on the Incus cluster or other workloads.