IP: 10.10.10.60
+ +``` + +Traffic distribution across multiple backends confirms HA load balancing +is working. OVN uses connection-based hashing — sequential requests from +the same source may hit the same backend. + +### 8.5 Health and Status + +```bash +./deploy-haproxy --status +``` + +Shows HAProxy container health, backend status, VIP connectivity, and +Aether service health stats including per-backend UP/DOWN status and +current session count. + +### 8.6 ACL Notes + +Aether creates per-container ACLs with default-deny rules. The deploy +script configures services to work within this model. If external access +to the VIP doesn't work, check the HAProxy containers' ACLs: + +```bash +incus network acl list oc-node-01: +``` + +Look for ACLs named `52-ffsdn-haproxy-52-01-aether-acl` and similar. The +script's service creation should have added the necessary ingress rules +for backend traffic. -## Section 8: Network Isolation & Security +## Section 9: AWX Lifecycle Automation -### 8.1 Create Isolated Network +Deploy AWX (the open-source Ansible Tower) on a Debian 12 VM with K3s, +configure lifecycle playbooks, and register with Aether for automated +post-deploy and decommission hooks. + +### 9.1 Prerequisites Check + +```bash +./deploy-awx --doctor +``` + +### 9.2 Deploy + +```bash +./deploy-awx --deploy +``` + +The script runs 6 phases: + +1. **Create VM**: Debian 12 with 4 vCPU, 8 GiB RAM, 40 GiB disk on + oc-node-02 with macvlan NIC +2. **Configure network**: static IP 192.168.102.161/22 +3. **Install K3s**: lightweight Kubernetes (`--write-kubeconfig-mode 644`) +4. **Deploy AWX Operator**: from kustomize manifests in `awx-manifests/` +5. **Deploy AWX instance**: web UI on NodePort 30080, task engine, + PostgreSQL +6. **Verify**: API ping, admin password retrieval + +Full deployment takes 15-20 minutes. AWX is accessible via HTTP (not +HTTPS) at port 30080 — Traefik's ingress returns 404 for IP-based access. + +### 9.3 Configure + +Set up the AWX project, inventory, credentials, and job templates: + +```bash +./deploy-awx --configure +``` + +This creates: + +- **Project**: `incus-contrib` (manual/local_path — playbooks are pushed + directly to the AWX task pod since the private Git repo is not reachable + from the Execution Environment container) +- **Inventory**: `incus-instances` +- **Credential**: `incus-instances` (machine type — required by AWX + templates, though the playbooks use Incus API not SSH) +- **Templates**: `post-deploy` and `decommission` + +The Incus client cert/key are pushed to the AWX task pod at +`/var/lib/awx/projects/incus-contrib/`. During job execution, the EE +container mounts this as `/runner/project/` — the playbooks reference +certs at that path. + +### 9.4 Register with Aether + +```bash +./deploy-awx --join-aether +``` + +This creates a Personal Access Token (PAT) in AWX and prints the manual +steps to complete Aether integration. In the Aether web UI: + +1. Navigate to **Settings → Ansible Automation** (`/awx-endpoints`) +2. **Add Endpoint**: + - **Name**: `lab-awx` + - **URL**: `http://192.168.102.161:30080` + - **Token**: the PAT printed by the script + - **Verify SSL**: unchecked +3. **Configure cluster AWX binding** (from the cluster settings): + - **Post-deploy template ID**: printed by the script + - **Decommission template ID**: printed by the script + - **Job timeout**: 600 seconds + +### 9.5 How the Lifecycle Hooks Work + +When Aether creates an instance, it triggers the AWX **post-deploy** +template with `ffsdn_*` variables (instance name, IP, cluster, deployer, +image info). The playbook: + +1. Waits for the instance to reach Running state +2. Pushes a setup script via Incus file API +3. Executes it via Incus exec API (sets hostname, installs base packages, + configures unattended upgrades, writes deployment metadata) +4. Verifies success + +When Aether deletes an instance, it triggers the **decommission** template. +The playbook gracefully stops application services and records the event +in a ledger. + +**Important**: post-deploy failures cause Aether to auto-rollback (delete +the instance). Decommission failures do NOT block deletion. + +### 9.6 Verify + +```bash +# AWX API health +curl -s http://192.168.102.161:30080/api/v2/ping/ | python3 -m json.tool + +# Full status +./deploy-awx --status +``` + +The status output shows VM health, K3s state, pod status, admin password, +AWX version, and Aether connectivity. + + +## Section 10: Observability Stack + +Deploy Prometheus, Grafana, Loki, Promtail, and node-exporters for full +cluster observability. The stack scrapes Incus metrics (with TLS client +certs), HAProxy stats, and host-level metrics. + +### 10.1 Deploy + +```bash +./deploy-observability --deploy +``` + +The script runs 9 phases: + +1. **Monitoring container**: Debian 12, 2 GiB RAM, 20 GiB disk on + oc-node-02 at 10.10.10.70 +2. **Prometheus**: scrapes Incus nodes (`:8443/1.0/metrics` with client + certs), HAProxy stats (`:8404`), and node-exporters (`:9100`) +3. **Loki + Promtail**: log aggregation from systemd journal and + `/var/log` +4. **Grafana**: dashboards with Prometheus and Loki datasources +5. **Node-exporters**: 3 Alpine containers (node-exp-01/02/03) with host + filesystem bind-mounts for full host metrics +6. **ACL**: `monitoring-allow` ACL on all observability containers +7. **HAProxy exporter**: prometheus exporter route added to HAProxy + containers, plus monitoring ACL rules for scrape access on port 8404 +8. **Network forward**: 192.168.103.201 → Grafana (`:3000`) and + Prometheus (`:9090`) +9. **Dashboards**: pre-built JSON dashboards uploaded to Grafana + +Full deployment takes ~5 minutes. + +### 10.2 Access Dashboards + +- **Grafana**: `http://192.168.103.201:3000` (admin / admin) +- **Prometheus**: `http://192.168.103.201:9090` + +Pre-loaded dashboards: + +| Dashboard | Description | +|-----------|-------------| +| Incus Cluster Overview | Instances, memory, CPU across all 3 nodes | +| HAProxy Traffic | Request rate, backend health, error rates | +| Host Resources | CPU, memory, disk, network per node | + +### 10.3 Verify + +```bash +./deploy-observability --status +``` + +Shows container health, service status (prometheus, grafana-server, loki, +promtail), Prometheus target scrape results, Grafana API health, Loki +ingestion status, and network forward connectivity. + +For detailed diagnostics: + +```bash +./deploy-observability --doctor +``` + +This checks container networking (gateway, node-exporter, HAProxy +reachability), Prometheus scrape errors, service health with log output +on failure, and disk space. + + +## Section 11: Network Isolation & Security + +### 11.1 Create Isolated Network ```bash incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \ @@ -1011,14 +1306,14 @@ incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \ ipv6.address=none ``` -### 8.2 Launch Isolated Containers +### 11.2 Launch Isolated Containers ```bash incus launch images:debian/12 oc-node-01:iso-app-01 --network net-isolated --target oc-node-01 incus launch images:debian/12 oc-node-01:iso-app-02 --network net-isolated --target oc-node-02 ``` -### 8.3 Verify Network Isolation +### 11.3 Verify Network Isolation Containers on net-isolated can reach each other: @@ -1033,13 +1328,13 @@ same OVN network can reach each other across nodes. But net-prod **cannot** reach net-isolated: ```bash -incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2" +incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2" ``` Actual result: **100% packet loss**. Different OVN networks are fully isolated — separate L2 domains, no routing between them. -### 8.4 Create Network ACL +### 11.4 Create Network ACL Create an ACL that blocks ICMP from a specific source: @@ -1047,11 +1342,11 @@ Create an ACL that blocks ICMP from a specific source: incus network acl create oc-node-01:block-ping incus network acl rule add oc-node-01:block-ping ingress \ action=drop protocol=icmp4 \ - source=10.10.10.0/24 \ - description="Block ICMP from net-prod subnet" + source=10.10.20.0/24 \ + description="Block ICMP within net-isolated" ``` -### 8.5 Apply and Test ACL +### 11.5 Apply and Test ACL Apply the ACL to net-isolated: @@ -1059,8 +1354,7 @@ Apply the ACL to net-isolated: incus network set oc-node-01:net-isolated security.acls=block-ping ``` -Verify ICMP is blocked between net-isolated containers (since they match -the source range — adjust the ACL source for targeted blocking): +Verify ICMP is blocked: ```bash incus exec oc-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2" @@ -1078,7 +1372,7 @@ Verify ICMP works again: incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2" ``` -### 8.6 Network Peering +### 11.6 Network Peering Connect net-prod and net-isolated so containers on both networks can communicate. Peering is bilateral — create a peer on both sides: @@ -1093,12 +1387,12 @@ incus network peer create oc-node-01:net-isolated peer-to-prod net-prod \ --description "Peer to production network" ``` -### 8.7 Verify Peering +### 11.7 Verify Peering Cross-network ping (prod → isolated): ```bash -incus exec oc-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2" +incus exec oc-node-01:nginx-lb-01 -- ping -c 3 "$IP_ISO2" ``` Actual result: 0% packet loss with TTL=62 (64 - 2 router hops), @@ -1107,11 +1401,11 @@ confirming traffic traverses the OVN routers on both sides of the peering. Cross-network ping (isolated → prod): ```bash -IP_WEB1=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1) -incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1" +NLB1_IP=$(incus list oc-node-01:nginx-lb-01 --columns 4 --format csv | cut -d' ' -f1) +incus exec oc-node-01:iso-app-01 -- ping -c 3 "$NLB1_IP" ``` -### 8.8 Remove Peering +### 11.8 Remove Peering ```bash incus network peer delete oc-node-01:net-prod peer-to-isolated @@ -1121,7 +1415,7 @@ incus network peer delete oc-node-01:net-isolated peer-to-prod Verify isolation is restored: ```bash -incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2" +incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2" ``` Expected: 100% packet loss. Networks are isolated again. @@ -1134,125 +1428,42 @@ incus delete oc-node-01:iso-app-02 --force ``` -## Section 9: Load Balancers & Network Forwards +## Section 12: Live Migration -### 9.1 Create OVN Load Balancer +### 12.1 Deploy Test VM -Create a load balancer with a VIP from the UPLINK range: +Deploy a temporary VM on net-prod for migration testing. Live migration +requires VMs (containers use a different migration path). ```bash -incus network load-balancer create oc-node-01:net-prod 192.168.103.200 +incus launch images:debian/12 oc-node-01:migration-test --vm \ + --network net-prod --target oc-node-01 ``` -Add backend servers. **Important**: backends require the instance's **IP -address**, not its name. Get the IPs first: +Wait for boot (~30-60s), then configure for stateful migration: ```bash -WEB1_IP=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1) -WEB2_IP=$(incus list oc-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1) -echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP" +incus stop oc-node-01:migration-test +incus config set oc-node-01:migration-test limits.cpu=0-1 +incus config set oc-node-01:migration-test migration.stateful=true +incus config device override oc-node-01:migration-test root size.state=2GiB +incus start oc-node-01:migration-test ``` -Add backends using IP addresses: +**Critical**: `limits.cpu` must be a **range** (e.g., `0-1`), not an +integer. Without the range, QEMU sets `maxcpus` based on the host's CPU +count, which varies across nodes and breaks migration with +`Missing section footer for ICH9LPC`. + +**Important**: use `device override` (not `device set`) because the `root` +device comes from the default profile. + +### 12.2 Create Heartbeat Service + +Create a simple counter to verify state continuity across migration: ```bash -incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \ - web-01 "$WEB1_IP" 80 -incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \ - web-02 "$WEB2_IP" 80 -``` - -Add a port mapping: - -```bash -incus network load-balancer port add oc-node-01:net-prod 192.168.103.200 \ - tcp 80 web-01,web-02 -``` - -### 9.2 Test Load Balancer - -From your dev machine (must be on the same VLAN or have routing to -192.168.103.0/24): - -```bash -for i in $(seq 1 6); do - curl -s http://192.168.103.200 -done -``` - -Actual output: - -``` -Server: prod-web-01 -Server: prod-web-01 -Server: prod-web-01 -Server: prod-web-02 -Server: prod-web-02 -Server: prod-web-02 -``` - -OVN uses connection-based hashing (not round-robin). Multiple requests from -the same source will typically hit the same backend. Different source ports -or connections may hit different backends. - -### 9.3 Create Network Forward - -Network forwards expose internal services on LAN IPs. Forward -tcp:8080 → prod-api-01:80. **Like LB backends, forwards require IP addresses**: - -```bash -API_IP=$(incus list oc-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1) - -incus network forward create oc-node-01:net-prod 192.168.103.201 -incus network forward port add oc-node-01:net-prod 192.168.103.201 \ - tcp 8080 "$API_IP" 80 -``` - -### 9.4 Test Network Forward - -```bash -curl -s http://192.168.103.201:8080 -``` - -Actual output: `API: prod-api-01` - -### 9.5 DNS Resolution - -OVN provides per-network DNS. Containers can resolve each other by -hostname: - -```bash -incus exec oc-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus" -``` - -Actual output: `10.10.10.3` — OVN DNS resolves instance names within -each network. - - -## Section 10: Live Migration - -### 10.1 Verify Migration Readiness - -Check that VMs have the required configuration: - -```bash -for vm in prod-db-01 prod-app-01; do - echo "=== $vm ===" - incus config get oc-node-01:$vm limits.cpu - incus config get oc-node-01:$vm migration.stateful - incus config device get oc-node-01:$vm root size.state -done -``` - -Expected: `0-1`, `true`, `2GiB` for each VM. - -### 10.2 Create Heartbeat Service - -Create a simple counter in prod-db-01 to verify state continuity across -migration: - -```bash -incus exec oc-node-01:prod-db-01 -- bash -c ' +incus exec oc-node-01:migration-test -- bash -c ' mkdir -p /tmp/heartbeat nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \ > /dev/null 2>&1 & @@ -1263,135 +1474,102 @@ incus exec oc-node-01:prod-db-01 -- bash -c ' Read the counter: ```bash -incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` Note the value. After migration, the counter should continue from where it left off (live migration preserves running state). -### 10.3 Live Migration Round-Trip +### 12.3 Live Migration Round-Trip Check current location: ```bash -incus list oc-node-01:prod-db-01 --columns nL --format csv +incus list oc-node-01:migration-test --columns nL --format csv ``` **Migrate node-01 → node-02:** ```bash -time incus move oc-node-01:prod-db-01 --target oc-node-02 +time incus move oc-node-01:migration-test --target oc-node-02 ``` -Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect: +Actual result: ~7s (~140 MB/s). Wait for the VM agent to reconnect: ```bash sleep 4 -incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` -Counter went from 9 → 25. The heartbeat process was never interrupted — -it continued counting during migration. +The heartbeat counter continues incrementing — the process was never +interrupted during migration. **Migrate node-02 → node-03:** ```bash -time incus move oc-node-01:prod-db-01 --target oc-node-03 +time incus move oc-node-01:migration-test --target oc-node-03 sleep 4 -incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` -Actual result: 7.379s. Counter went to 41. - **Migrate node-03 → node-01 (back to origin):** ```bash -time incus move oc-node-01:prod-db-01 --target oc-node-01 +time incus move oc-node-01:migration-test --target oc-node-01 sleep 4 -incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` -Actual result: 6.896s. Counter went to 56. - Verify the VM is back on node-01: ```bash -incus list oc-node-01:prod-db-01 --columns nL --format csv +incus list oc-node-01:migration-test --columns nL --format csv ``` -### 10.4 Active I/O During Migration - -Start a continuous write inside the VM: - -```bash -incus exec oc-node-01:prod-db-01 -- bash -c ' - dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 & - echo "Write started, PID: $!" -' -``` - -Migrate while I/O is active: - -```bash -time incus move oc-node-01:prod-db-01 --target oc-node-02 -sleep 4 -``` - -Verify the file exists and is intact: - -```bash -incus exec oc-node-01:prod-db-01 -- ls -la /tmp/testfile -incus exec oc-node-01:prod-db-01 -- md5sum /tmp/testfile -``` - -Move back: - -```bash -incus move oc-node-01:prod-db-01 --target oc-node-01 -sleep 4 -``` - -### 10.5 Stateful Stop/Restore +### 12.4 Stateful Stop/Restore Stateful stop saves VM memory to disk. On start, the VM resumes exactly where it was: ```bash # Note the heartbeat counter -incus exec oc-node-01:prod-app-01 -- bash -c ' - mkdir -p /tmp/heartbeat - echo 42 > /tmp/heartbeat/counter - cat /tmp/heartbeat/counter -' +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` Stateful stop: ```bash -incus stop oc-node-01:prod-app-01 --stateful +incus stop oc-node-01:migration-test --stateful ``` Start (resumes from saved state): ```bash -incus start oc-node-01:prod-app-01 +incus start oc-node-01:migration-test sleep 4 -incus exec oc-node-01:prod-app-01 -- cat /tmp/heartbeat/counter +incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter ``` -Expected: `42` — the file (and entire VM state) is preserved. +The counter resumes from the saved value — the entire VM state is +preserved across stateful stop/start cycles. **If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the saved state: ```bash -incus start oc-node-01:prod-app-01 --stateless +incus start oc-node-01:migration-test --stateless +``` + +### 12.5 Clean Up Test VM + +```bash +incus delete oc-node-01:migration-test --force ``` -## Section 11: Cluster Lifecycle +## Section 13: Cluster Lifecycle -### 11.1 Evacuation & Restore +### 13.1 Evacuation & Restore Evacuate node-02. All workloads are moved to other nodes: @@ -1405,16 +1583,8 @@ Check workload distribution — nothing on node-02: incus list oc-node-01: --columns nstL --format table ``` -Actual behavior: VMs with `migration.stateful=true` are live-migrated -(prod-app-01 migrated to oc-node-03). Containers are stopped and moved -(prod-web-02 stopped, moved to oc-node-03, then started). The `--force` -flag skips confirmation prompts. - -**Note**: if VMs lack the `limits.cpu` range fix, use `--action stop` -instead to avoid migration failures: -```bash -incus cluster evacuate oc-node-01:oc-node-02 --force --action stop -``` +Actual behavior: containers are stopped, moved to other nodes, then +restarted. The `--force` flag skips confirmation prompts. Verify node-02 shows EVACUATED: @@ -1437,7 +1607,7 @@ incus cluster list oc-node-01: All nodes should show ONLINE. -### 11.2 Node Failure Simulation +### 13.2 Node Failure Simulation A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat detects the failure in ~40 seconds. After the node is restarted: @@ -1465,7 +1635,7 @@ you can tolerate temporary network disruption): # → oc-node-02 shows ONLINE ``` -### 11.3 Node Replacement +### 13.3 Node Replacement Full procedure: evacuate a node, remove it from the cluster, destroy the VM, deploy a fresh node, and join it back. This tests the complete lifecycle. @@ -1555,7 +1725,7 @@ incus cluster list oc-node-01: All 3 nodes should be ONLINE with `ovn-chassis` role. -### 11.4 Cluster Rebalancing +### 13.4 Cluster Rebalancing Enable automatic workload rebalancing. When a new node joins (or workloads are unevenly distributed), Incus redistributes VMs: @@ -1584,7 +1754,7 @@ incus config unset oc-node-01: cluster.rebalance.threshold ``` -## Section 12: OC Dashboard +## Section 14: OC Dashboard **Important**: Switch to the OC remote first. The OC CLI does not support `remote:` suffix syntax: @@ -1634,30 +1804,45 @@ OC has full visibility and management of the cluster — including update delivery, server inventory, and cluster formation. -## Section 13: Cleanup +## Section 15: Cleanup -### Delete All Workloads +### Tear Down Services (Reverse Order) + +Tear down deployed services in reverse order. Each script includes an +interactive confirmation prompt: ```bash -# Delete containers -for c in prod-web-01 prod-web-02 prod-api-01; do - incus delete oc-node-01:$c --force -done +cd incusos -# Delete VMs -for vm in prod-db-01 prod-app-01; do - incus delete oc-node-01:$vm --force -done +# 1. Observability stack +./deploy-observability --cleanup + +# 2. AWX VM +./deploy-awx --cleanup + +# 3. HAProxy infrastructure + backends +./deploy-haproxy --cleanup +``` + +### Delete Aether VM + +```bash +incus stop oc-node-01:aether --force +incus delete oc-node-01:aether --force +incus image delete oc-node-01:aether-golden-image-v6 2>/dev/null || true +``` + +### Remove Isolated Network (if created) + +```bash +incus network delete oc-node-01:net-isolated 2>/dev/null || true +incus network acl delete oc-node-01:block-ping 2>/dev/null || true ``` ### Remove OVN Networks ```bash -# Delete OVN networks incus network delete oc-node-01:net-prod -incus network delete oc-node-01:net-isolated 2>/dev/null || true - -# Delete UPLINK incus network delete oc-node-01:UPLINK ``` @@ -1703,29 +1888,62 @@ Restart later with: ``` -## Section 14: Verification Checklist +## Section 16: Verification Checklist | # | Check | Command | Expected | |---|-------|---------|----------| | 1 | All VMs running | `incusos-proxmox --status examples/lab-oc-nodes.yaml` | 4 VMs running, port 8443 open | -| 2 | Scrub schedule healthy | `incus query oc-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` | -| 3 | OC accessible | `operations-center remote switch oc-lab && operations-center admin os show` | Shows version, uptime | +| 2 | Scrub schedule | `incus query oc-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` | +| 3 | OC accessible | `operations-center admin os show` | Shows version, uptime | | 4 | Cluster formed | `incus cluster list oc-node-01:` | 3 nodes ONLINE | -| 5 | Storage pool | `incus storage list oc-node-01:` | `local` pool on all members | -| 6 | Bridge isolation | Ping cross-node on incusbr0 | 100% loss (expected) | -| 7 | OVN connectivity | Ping cross-node on net-prod | 0% loss | -| 8 | Internet via OVN | `ping 1.1.1.1` from OVN container | 0% loss | -| 9 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) | -| 10 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 | -| 11 | Load balancer | `curl http://192.168.103.200` | Backend response | -| 12 | Network forward | `curl http://192.168.103.201:8080` | API response | -| 13 | DNS resolution | `dig prod-web-02.incus` from container | Resolves to 10.10.10.x | -| 14 | VM live migration | `incus move` VM between nodes | State preserved | -| 15 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned | -| 16 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved | +| 5 | OVN connectivity | Ping cross-node on net-prod | 0% loss | +| 6 | Aether web UI | `https://192.168.102.160:8443` | Login page loads | +| 7 | Aether API | `GET /api/clusters` with JWT | Returns cluster data | +| 8 | HAProxy VIP | `curl http://192.168.103.200` | Backend response | +| 9 | HAProxy HA | `./deploy-haproxy --status` | Both containers running | +| 10 | AWX API | `curl http://192.168.102.161:30080/api/v2/ping/` | JSON with version | +| 11 | AWX templates | `./deploy-awx --status` | Pods running, templates configured | +| 12 | Grafana | `http://192.168.103.201:3000` | Dashboard loads | +| 13 | Prometheus targets | `./deploy-observability --status` | All targets UP | +| 14 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) | +| 15 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 | +| 16 | VM live migration | `incus move` VM between nodes | State preserved | +| 17 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned | +| 18 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved | +| 19 | DNS resolution | `dig hostname.incus` from container | Resolves to 10.10.10.x | -## Section 15: Quick Reference +## Section 17: Quick Reference + +### Service Endpoints + +| Service | URL | Auth | +|---------|-----|------| +| Aether | `https://192.168.102.160:8443` | admin / (env file) | +| AWX | `http://192.168.102.161:30080` | admin / (from K8s secret) | +| Grafana | `http://192.168.103.201:3000` | admin / admin | +| Prometheus | `http://192.168.103.201:9090` | none | +| HAProxy VIP | `http://192.168.103.200` | none | +| OC Web UI | `https://192.168.102.120:8443/ui/` | client certificate | + +### Deploy Script Commands + +| Script | Action | Description | +|--------|--------|-------------| +| `deploy-haproxy --doctor` | Prerequisites | Check tools, Aether, cluster | +| `deploy-haproxy --deploy` | Full deploy | Backends + image + infra + service | +| `deploy-haproxy --status` | Health check | Container + VIP + backend status | +| `deploy-haproxy --cleanup` | Tear down | Remove all HAProxy resources | +| `deploy-awx --deploy` | Full deploy | VM + K3s + AWX operator + instance | +| `deploy-awx --configure` | Configure | Project + inventory + templates | +| `deploy-awx --join-aether` | Register | Create PAT, print Aether steps | +| `deploy-awx --status` | Health check | VM + K3s + pods + API | +| `deploy-awx --heal` | Fix issues | Restart pods, sync project | +| `deploy-awx --cleanup` | Tear down | Destroy AWX VM | +| `deploy-observability --deploy` | Full deploy | All 9 phases | +| `deploy-observability --status` | Health check | Services + targets + forward | +| `deploy-observability --doctor` | Diagnose | Network + scrape + disk | +| `deploy-observability --cleanup` | Tear down | Remove all observability | ### Cluster Command Syntax @@ -1800,8 +2018,12 @@ incus start REMOTE:VM | OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration | | `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore | | "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network | -| "Invalid target address" on LB backend | Backend needs IP, not instance name | Use instance IP address (e.g., `10.10.10.2`) | | OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first | | Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement | | "Device from profile(s) cannot be modified" | `root` device comes from default profile | Use `incus config device override` instead of `device set` | | `incus launch` hangs or times out | Image download to new node takes 2-3 min | Run launches one at a time, not chained with `&&` | +| Aether CSRF error on API call | Session auth endpoints need CSRF token | Use Playwright or `aether-browser` helper | +| AWX pods OOMKilled | Web pod needs 2 GiB minimum | Check `awx-manifests/base/awx.yaml` resource limits | +| HAProxy 405 on health check | HTTP mode sends OPTIONS requests | Use TCP mode for nginx backends | +| Prometheus scrape failing on Incus | Client certs missing or wrong perms | Check `/etc/prometheus/certs/` in monitoring container | +| Grafana not reachable via forward | Network forward not created | `./deploy-observability --doctor` to diagnose |