diff --git a/.gitignore b/.gitignore
index 4851a3d..8801b88 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,8 +26,9 @@ Thumbs.db
 # Go binaries (flasher-tool)
 flasher-tool
 
-# Proxmox connection config (contains credentials)
-proxmox.yaml
+# Proxmox connection configs (contain secrets, per-target and default)
+incusos/proxmox.yaml
+incusos/targets/*/proxmox.yaml
 
 # Per-target env files
 env-*
diff --git a/hetzner/hetzner-lab-guide.md b/hetzner/hetzner-lab-guide.md
new file mode 100644
index 0000000..11e073a
--- /dev/null
+++ b/hetzner/hetzner-lab-guide.md
@@ -0,0 +1,907 @@
+# Hetzner IncusOS Lab — End-to-End Guide
+
+Full deployment guide for a production-quality IncusOS lab on a Hetzner
+dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
+networking, HAProxy load balancing, AWX automation, and observability.
+
+**Estimated time**: 3-4 hours for a clean run.
+
+---
+
+## Overview
+
+```
+Hetzner dedicated (256 GB RAM, 32+ cores)
+└─ Proxmox VE 9
+   ├─ hz-oc       (VMID 910) — Operations Center 0.4.x
+   ├─ hz-node-01  (VMID 911) — Incus cluster init + OVN control plane
+   ├─ hz-node-02  (VMID 912) — Incus cluster member
+   └─ hz-node-03  (VMID 913) — Incus cluster member
+      └─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
+         ├─ ffsdn-haproxy-52-01/02  — HAProxy HA pair
+         ├─ nginx-lb-01/02/03       — Test backends
+         ├─ monitoring              — Prometheus + Grafana
+         ├─ node-exp-01/02/03       — Node exporters
+         └─ awx                     — AWX Operator on K3s
+```
+
+**Network topology:**
+- vmbr0: Public interface (SSH + WireGuard only)
+- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
+- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
+- OVN overlay: 10.10.10.0/24 (containers on net-prod)
+- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
+
+---
+
+## Prerequisites
+
+- Hetzner dedicated server with Proxmox VE 9 installed and configured
+  (see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
+- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
+- This repository cloned locally
+- `env` file populated (see below)
+- `incusos/targets/hetzner/proxmox.yaml` filled in
+- Incus client installed on workstation
+
+### env file
+
+```bash
+# incusos-contrib/env
+export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
+export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
+export AETHER_ADMIN_PASSWORD="admin1234"
+```
+
+### proxmox.yaml
+
+```bash
+cp incusos/targets/hetzner/proxmox.yaml.example \
+   incusos/targets/hetzner/proxmox.yaml
+# Fill in: host, node name, token_id, storage
+```
+
+---
+
+## 1. Deploy IncusOS VMs
+
+```bash
+source env
+
+incusos/incusos-proxmox \
+    --proxmox incusos/targets/hetzner/proxmox.yaml \
+    incusos/targets/hetzner/lab-production.yaml
+```
+
+**What gets deployed:**
+
+| VMID | Name      | IP            | RAM   | Cores | Disk |
+|------|-----------|---------------|-------|-------|------|
+| 910  | hz-oc     | 10.10.0.110   | 8 GB  | 4     | 100G |
+| 911  | hz-node-01| 10.10.0.111   | 80 GB | 16    | 150G |
+| 912  | hz-node-02| 10.10.0.112   | 80 GB | 16    | 100G |
+| 913  | hz-node-03| 10.10.0.113   | 80 GB | 16    | 100G |
+
+**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
+interrupt. Monitor progress:
+
+```bash
+# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
+source env
+incusos/helpers/proxmox-screenshot 910   # hz-oc
+incusos/helpers/proxmox-screenshot 911   # hz-node-01
+```
+
+Wait until all VMs are reachable:
+
+```bash
+for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
+    echo -n "$ip: "
+    ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
+done
+```
+
+---
+
+## 2. Add Incus remotes
+
+Add the cluster remote pointing at hz-node-01 (the init node):
+
+```bash
+incus remote add hz-cluster https://10.10.0.111:8443 \
+    --accept-certificate --auth-type tls
+```
+
+> **Note**: The `incus remote add` command prompts for a trust token.
+> Obtain the token from OC after cluster formation (section 3), or
+> if using a pre-formed cluster, generate one with:
+> `incus config trust add --name workstation`
+
+---
+
+## 3. Cluster formation via Operations Center
+
+Open the Operations Center UI:
+
+```
+https://10.10.0.110:8443
+Username: admin
+Password: admin   (first login, then set a new password)
+```
+
+### 3.1 Add Incus nodes
+
+In OC: **Servers → Add Server** for each node:
+
+| Field   | hz-node-01       | hz-node-02       | hz-node-03       |
+|---------|------------------|------------------|------------------|
+| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
+| Trust   | via token        | via token        | via token        |
+
+Each node's trust token is displayed on the IncusOS console on first boot.
+
+### 3.2 Form the cluster
+
+In OC: **Clusters → Create Cluster**
+
+- Name: `hz-cluster`
+- Init node: `hz-node-01`
+- Members: `hz-node-02`, `hz-node-03`
+- App seed: `{}` (empty JSON for manual post-formation setup)
+
+OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
+
+Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
+HAProxy and AWX configuration.
+
+### 3.3 Verify cluster
+
+```bash
+incus cluster list hz-cluster:
+# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
+```
+
+---
+
+## 4. OVN networking
+
+### 4.1 Create the OVN uplink network
+
+The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
+`eth0` or `ens18`. This is specific to IncusOS.
+
+```bash
+# Create UPLINK on each node
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-01 \
+    config parent=mgmt
+
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-02 \
+    config parent=mgmt
+
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-03 \
+    config parent=mgmt
+
+# Finalize UPLINK on cluster level
+incus network create hz-cluster:UPLINK \
+    config ipv4.address=none \
+    config ipv6.address=none
+```
+
+### 4.2 Deploy OVN central
+
+OVN central runs as a privileged container on hz-node-01:
+
+```bash
+incus launch images:debian/12 hz-cluster:ovn-central \
+    --vm=false \
+    --target hz-node-01 \
+    -c security.privileged=true \
+    -d root,size=20GiB
+
+# Wait for agent
+sleep 15
+
+# Install ovn-central
+incus exec hz-cluster:ovn-central -- bash -c "
+    apt-get update -qq
+    apt-get install -y -qq ovn-central
+    systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
+    systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
+"
+```
+
+### 4.3 Expose OVN NB/SB ports to cluster
+
+Create proxy devices so cluster nodes can reach OVN:
+
+```bash
+# Get ovn-central IP
+OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
+
+incus config device add hz-cluster:ovn-central ovn-nb proxy \
+    listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
+    bind=host
+incus config device add hz-cluster:ovn-central ovn-sb proxy \
+    listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
+    bind=host
+```
+
+### 4.4 Create OVN overlay network
+
+```bash
+# Use hz-node-01 IP as OVN NB/SB server
+NODE01_IP="10.10.0.111"
+
+incus network create hz-cluster:net-prod \
+    --type=ovn \
+    config network=UPLINK \
+    config ipv4.address=10.10.10.1/24 \
+    config ipv4.nat=true \
+    config ipv6.address=none \
+    config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
+    config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
+```
+
+Wait for the network to reach **Created** state:
+
+```bash
+incus network info hz-cluster:net-prod
+```
+
+### 4.5 Configure external IP range (OVN VIPs)
+
+```bash
+# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
+# These are allocated on the Proxmox host's vmbr1 bridge
+# No additional configuration needed — OVN will ARP for these IPs
+```
+
+Verify:
+
+```bash
+incus network list hz-cluster:
+# net-prod should show type: ovn, state: Created
+```
+
+---
+
+## 5. Deploy Aether
+
+### 5.1 Deploy the Aether VM
+
+Use the macvlan method to give Aether a direct vmbr1 IP:
+
+```bash
+incus launch images:ubuntu/22.04 hz-cluster:aether \
+    --vm \
+    --target hz-node-01 \
+    -d root,size=40GiB \
+    -c limits.cpu=4 \
+    -c limits.memory=8GiB \
+    -d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
+```
+
+> **Alternative**: use Aether golden image if already built on the cluster.
+> Check `incus image list hz-cluster: | grep aether`.
+
+Wait for the VM, then SSH in:
+
+```bash
+ssh ubuntu@10.10.0.120
+```
+
+Follow the [Aether installation guide](../notes/aether-guide.md) for the
+full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
+
+### 5.2 Configure Aether cluster
+
+In Aether UI (`https://10.10.0.120:8443`):
+
+1. **Settings → Clusters → Add Cluster**
+   - Name: hz-cluster
+   - Endpoint: `https://10.10.0.111:8443`
+   - Authenticate (generates a trust token, add it to the cluster)
+
+2. **Settings → Operations Centers → Add OC**
+   - URL: `https://10.10.0.110:8443`
+
+3. Verify hz-cluster instances sync in Aether dashboard.
+
+Aether default credentials: `admin` / `admin` (change on first login).
+
+For Aether password, set in `env`:
+
+```bash
+export AETHER_ADMIN_PASSWORD="admin1234"
+```
+
+---
+
+## 6. HAProxy load balancing
+
+### 6.1 Create config file
+
+Save as `incusos/targets/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
+
+```yaml
+haproxy:
+  cluster_remote: hz-cluster
+  cluster_id: 52            # from OC cluster listing
+  ovn_network: net-prod
+  vip: 10.10.0.200
+  haproxy_01_ip: 10.10.10.50
+  haproxy_02_ip: 10.10.10.51
+  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
+  service_name: web-test
+  image_version: "1.0.0"
+  aether_url: https://10.10.0.120:8443
+```
+
+> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
+> It's the numeric ID assigned when the cluster was created.
+
+### 6.2 Build HAProxy image
+
+Before deploying infrastructure, build and push the HAProxy image:
+
+In Aether UI → **HAProxy → Images → Build**:
+- Version: `1.0.0`
+- Cluster: `hz-cluster`
+- Click **Build** and wait 5-10 minutes
+
+After build, push to hz-cluster:
+- Click **Push** → select `hz-cluster`
+- Click **Set Current**
+
+Alternatively, use the script (requires the UI steps above for first build):
+
+```bash
+source env
+incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
+```
+
+### 6.3 Full deploy
+
+```bash
+source env
+incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
+```
+
+This:
+1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
+2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
+3. Creates the `web-test` service (HTTP, round-robin, health checks)
+
+On subsequent runs (if backends already exist):
+
+```bash
+incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
+    --skip-backends --skip-image
+```
+
+### 6.4 Verify
+
+```bash
+curl http://10.10.0.200/
+# Should show "Backend: nginx-lb-0x"
+
+# Test distribution
+for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
+```
+
+---
+
+## 7. AWX automation
+
+### 7.1 Create config file
+
+Save as `incusos/targets/hetzner/awx.yaml`:
+
+```yaml
+awx:
+  vm_name: awx
+  target_remote: hz-cluster
+  target_node: hz-node-03    # specific node within cluster
+  ip: 10.10.0.122/24
+  gateway: 10.10.0.1
+  dns: 10.10.0.1
+  cpu: 4
+  memory: 8GiB
+  disk: 40GiB
+  aether_url: https://10.10.0.120:8443
+  aether_cluster_id: 52
+```
+
+> **target_node vs target_remote**: `target_remote` is the Incus remote
+> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
+> member to deploy on (e.g. `hz-node-03`). Both are required for
+> cluster deployments.
+
+### 7.2 Deploy
+
+```bash
+source env
+incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy
+```
+
+This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
+AWX will be at `http://10.10.0.122:30080`.
+
+On partial failure (e.g. network issue mid-deploy), resume:
+
+```bash
+incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy --resume
+# Skips VM creation and K3s install phases, jumps to AWX Operator phase
+```
+
+If you get `permission denied` on temp files from a previous aborted run:
+
+```bash
+incus exec hz-cluster:awx -- rm -f \
+    /tmp/awx-operator-kustomization.yaml \
+    /tmp/awx-kustomization.yaml \
+    /tmp/awx-instance.yaml
+```
+
+### 7.3 Configure AWX
+
+> **Wait for the awx-task pod before running --configure.** The AWX web
+> deployment becomes ready first (~3 min), but `--configure` needs the
+> task pod. Check it:
+> ```bash
+> incus exec hz-cluster:awx -- kubectl -n awx wait \
+>     --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
+>     --timeout=300s
+> ```
+
+```bash
+source env
+incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --configure
+```
+
+This creates:
+- Project: `incus-contrib` (linked to your git repo)
+- Inventory: `incus-lab`
+- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
+
+Get AWX admin password:
+
+```bash
+incusos/deploy-awx -c /tmp/hz-awx.yaml --status
+```
+
+### 7.4 Join Aether
+
+Generate an AWX API token for Aether:
+
+```bash
+source env
+incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --join-aether
+# Prints the PAT token and template IDs
+```
+
+In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
+- Name: `lab-awx`
+- URL: `http://10.10.0.122:30080`
+- Token: (paste token from `--join-aether` output)
+- Verify SSL: unchecked (HTTP endpoint)
+
+Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
+
+### 7.5 Bind AWX to cluster
+
+**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
+endpoint returns `400 "Invalid cluster ID"` due to a bug.
+
+**Workaround** (direct DB update):
+
+```bash
+# First confirm the AWX endpoint ID (assigned by the database):
+incus exec hz-cluster:aether -- bash -c \
+    "su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
+
+# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
+incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
+    UPDATE incus_ovn_clusters
+    SET awx_endpoint_id=2,
+        awx_post_deploy_template_id=9,
+        awx_decommission_template_id=10,
+        awx_job_timeout_seconds=600
+    WHERE cluster_id=52;
+\\\"\""
+```
+
+> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
+> `postgres` system user. The endpoint ID is auto-assigned — always check
+> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
+
+> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
+> to see if the endpoint has been fixed. If it returns a valid schema,
+> use the API instead of the DB workaround.
+
+Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
+the AWX endpoint, post-deploy, and decommission templates.
+
+---
+
+## 8. Observability stack
+
+### 8.1 Deploy
+
+```bash
+source env
+incusos/deploy-observability \
+    --remote hz-cluster \
+    --monitoring-target hz-node-01 \
+    --monitoring-ip 10.10.10.70 \
+    --forward-vip 10.10.0.201 \
+    --deploy
+```
+
+> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
+> Operations Center (hz-oc) runs OC, not standard Incus — its
+> `/1.0/instances` endpoint returns 404 and will fail the node
+> discovery phase.
+
+> **Grafana install may fail if net-prod containers lack internet.**
+> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
+> after container launch but can become unreliable. If phase 4 fails:
+>
+> ```bash
+> # Download Grafana locally and push it in
+> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
+> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
+> incus exec hz-cluster:monitoring -- bash -c "
+>     export DEBIAN_FRONTEND=noninteractive
+>     dpkg -i /tmp/grafana.deb; apt-get -f install -y
+>     mkdir -p /etc/grafana/provisioning/datasources
+>     cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
+> apiVersion: 1
+> datasources:
+>   - name: Prometheus
+>     type: prometheus
+>     uid: prometheus
+>     access: proxy
+>     url: http://localhost:9090
+>     isDefault: true
+>   - name: Loki
+>     type: loki
+>     uid: loki
+>     access: proxy
+>     url: http://localhost:3100
+> EOF
+>     systemctl daemon-reload
+>     systemctl enable grafana-server
+>     systemctl start grafana-server
+> "
+> ```
+>
+> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
+> network forward, and dashboards).
+
+> **Node-exporter containers are skipped by `--resume`.** If internet
+> was unavailable during phase 5, deploy them manually:
+>
+> ```bash
+> # Download node_exporter binary locally
+> curl -sL -o /tmp/ne.tar.gz \
+>     https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
+> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
+>
+> for i in 1 2 3; do
+>   name="node-exp-0${i}"
+>   ip="10.10.10.7${i}"
+>   node="hz-node-0${i}"
+>
+>   incus launch images:alpine/3.20 hz-cluster:${name} \
+>     -n net-prod --target ${node} \
+>     -c limits.memory=128MiB -c security.privileged=true
+>   incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
+>   sleep 5
+>
+>   for dev in proc sys root; do
+>     src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
+>     incus config device add hz-cluster:${name} host-${dev} disk \
+>       source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
+>     incus config device add hz-cluster:${name} host-${dev} disk \
+>       source=/ path=/host/root readonly=true 2>/dev/null || true
+>   done
+>
+>   incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
+>     hz-cluster:${name}/usr/bin/node_exporter
+>
+>   incus exec hz-cluster:${name} -- sh -c "
+>     chmod +x /usr/bin/node_exporter
+>     cat > /etc/init.d/node-exporter << 'INIT'
+> #!/sbin/openrc-run
+> description=\"Node Exporter\"
+> command=/usr/bin/node_exporter
+> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
+> command_background=true
+> pidfile=/run/node-exporter.pid
+> INIT
+>     chmod +x /etc/init.d/node-exporter
+>     rc-update add node-exporter default
+>     rc-service node-exporter start
+>
+>     cat > /etc/network/interfaces << NETCFG
+> auto lo
+> iface lo inet loopback
+> auto eth0
+> iface eth0 inet static
+>     address ${ip}/24
+>     gateway 10.10.10.1
+> NETCFG
+>     rc-service networking restart
+>   "
+>
+>   incus config device set hz-cluster:${name} eth0 \
+>     security.acls=monitoring-allow \
+>     security.acls.default.ingress.action=allow \
+>     security.acls.default.egress.action=allow
+> done
+> ```
+
+On partial failure (phases 6-9 only), resume:
+
+```bash
+source env
+incusos/deploy-observability \
+    --remote hz-cluster \
+    --monitoring-target hz-node-01 \
+    --monitoring-ip 10.10.10.70 \
+    --forward-vip 10.10.0.201 \
+    --resume --deploy
+```
+
+After deploying, fix the HAProxy prometheus exporter (default stats page
+doesn't expose `/metrics`):
+
+```bash
+for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
+  incus exec hz-cluster:${ct} -- bash -c "
+    sed -i 's/    stats enable/    http-request use-service prometheus-exporter if { path \/metrics }\n    stats enable/' /etc/haproxy/haproxy.cfg
+    systemctl reload haproxy
+  "
+done
+```
+
+### 8.2 Access
+
+| Service    | URL                          | Credentials |
+|------------|------------------------------|-------------|
+| Grafana    | http://10.10.0.201:3000      | admin/admin |
+| Prometheus | http://10.10.0.201:9090      | —           |
+
+9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
+storage, networking, capacity planning, and log explorer.
+
+Verify all Prometheus targets are up:
+
+```bash
+curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
+import sys,json
+for t in json.load(sys.stdin)['data']['activeTargets']:
+    print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
+"
+# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
+```
+
+---
+
+## 9. WireGuard access (macOS)
+
+If you use the macOS WireGuard app from the App Store:
+
+- The app uses Network Extension (visible as `utun5` in `ifconfig`)
+- `wg show` always shows empty — this is normal with the App Store app
+- The app allows only **one active tunnel at a time**
+
+For a second simultaneous tunnel:
+
+```bash
+# Install wg-quick for macOS
+brew install wireguard-tools
+
+# Start the second tunnel alongside the App Store one
+sudo wg-quick up ~/.config/wireguard/hetzner.conf
+```
+
+If the server added a new WireGuard peer (new workstation/client):
+
+```bash
+# On the Hetzner Proxmox host (via the existing tunnel or public IP)
+ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
+ssh hetzner-lab "wg-quick save wg0"
+```
+
+---
+
+## 10. Incus remote management
+
+### Add remotes
+
+```bash
+# OC (Operations Center — not standard Incus)
+# Note: Must be added manually — incus remote add loops prompting for token
+# OC does not support the standard trust token flow.
+# Edit ~/.config/incus/config.yml directly:
+```
+
+Add to `~/.config/incus/config.yml`:
+
+```yaml
+remotes:
+  oc-server:
+    addr: https://10.10.0.110:8443
+    auth_type: tls
+    protocol: incus
+    public: false
+  hz-cluster:
+    addr: https://10.10.0.111:8443
+    auth_type: tls
+    protocol: incus
+    public: false
+```
+
+Copy the OC certificate (accept it with curl first):
+
+```bash
+# Accept and save OC cert
+curl -sk https://10.10.0.110:8443 > /dev/null
+# The cert ends up in your browser cache — use openssl to fetch it
+openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
+    | openssl x509 > ~/.config/incus/servercerts/oc-server.crt
+```
+
+> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
+> fail. OC does not expose the standard `/1.0/instances` endpoint.
+
+### After redeploy (remote cert changed)
+
+```bash
+# Remove old remote and re-add
+incus remote remove hz-cluster
+incus remote add hz-cluster https://10.10.0.111:8443 \
+    --accept-certificate --auth-type tls
+```
+
+---
+
+## 11. Quick reference
+
+### Access URLs
+
+| Service         | URL                              | Credentials         |
+|-----------------|----------------------------------|---------------------|
+| Operations Center | https://10.10.0.110:8443       | admin / (set on first login) |
+| Aether          | https://10.10.0.120:8443         | admin / admin1234   |
+| AWX             | http://10.10.0.122:30080         | admin / (from deploy-awx --status) |
+| Grafana         | http://10.10.0.201:3000          | admin / admin       |
+| Prometheus      | http://10.10.0.201:9090          | —                   |
+| HAProxy VIP     | http://10.10.0.200/              | —                   |
+
+### IP allocation
+
+| Range                 | Purpose                      |
+|-----------------------|------------------------------|
+| 10.10.0.1             | Proxmox host (vmbr1 gateway) |
+| 10.10.0.110-113       | IncusOS VMs (OC + nodes)     |
+| 10.10.0.120           | Aether VM (macvlan)          |
+| 10.10.0.122           | AWX VM (macvlan)             |
+| 10.10.0.200+          | OVN external IPs (VIPs)      |
+| 10.10.10.0/24         | OVN overlay (containers)     |
+| 10.10.10.50-51        | HAProxy containers           |
+| 10.10.10.60-62        | nginx test backends          |
+| 10.10.10.70           | monitoring container         |
+| 10.10.10.71-73        | node-exporter containers     |
+| 10.10.99.0/24         | WireGuard tunnel             |
+| 10.10.99.1            | WireGuard server             |
+| 10.10.99.2            | Your workstation             |
+
+### Cluster IDs
+
+- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
+  your OC UI if different)
+
+### Useful incus commands
+
+```bash
+# List all instances in cluster
+incus list hz-cluster:
+
+# List cluster members
+incus cluster list hz-cluster:
+
+# List networks
+incus network list hz-cluster:
+
+# Exec into a container
+incus exec hz-cluster:monitoring -- bash
+
+# Start/stop a container
+incus start hz-cluster:monitoring
+incus stop hz-cluster:monitoring --force
+
+# Check container network
+incus exec hz-cluster:nginx-lb-01 -- ip a
+```
+
+### Teardown
+
+```bash
+# Force destroy all VMs (no nice cleanup needed for test lab)
+source env
+for vmid in 910 911 912 913; do
+    incusos/helpers/proxmox-api POST \
+        "/nodes/aether-lab/qemu/${vmid}/status/stop" \
+        '{"forceStop":1}' 2>/dev/null || true
+    sleep 3
+    incusos/helpers/proxmox-api DELETE \
+        "/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
+done
+
+# Remove stale remotes
+incus remote remove hz-cluster 2>/dev/null || true
+incus remote remove oc-server 2>/dev/null || true
+```
+
+---
+
+## Known issues and workarounds
+
+### OC remote add loops (incus remote add oc-server fails)
+
+`incus remote add` enters a trust token prompt loop for OC because OC
+does not implement the standard Incus trust token flow.
+
+**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
+and save the TLS certificate separately.
+
+### IncusOS management NIC name is `mgmt`, not `eth0`
+
+All IncusOS instances use `mgmt` as the management NIC name. Use
+`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
+
+### Aether `PUT /api/clusters/{id}/awx-config` returns 400
+
+Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
+
+### deploy-observability: don't use --oc-server with OC
+
+The `--oc-server` flag tries to list instances via the standard Incus
+API (`/1.0/instances`), which OC does not implement. Omit the flag.
+
+### OVN SNAT internet access from net-prod containers is unreliable
+
+Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
+reach the internet. This works briefly after container launch (via Proxmox
+NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
+stale. Connections return `Connection refused` from Fastly CDN.
+
+**Workaround**: Download packages locally and push them in with
+`incus file push` instead of relying on apt/apk inside net-prod containers.
+
+Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
+Proxmox masquerade and are unaffected.
+
+### deploy-awx temp file permission errors on resume
+
+Temp files created as root inside the AWX VM persist between runs.
+Clean up before resuming:
+
+```bash
+incus exec hz-cluster:awx -- rm -f \
+    /tmp/awx-operator-kustomization.yaml \
+    /tmp/awx-kustomization.yaml \
+    /tmp/awx-instance.yaml
+```
+
+### aether_url in config files must use key-specific sed
+
+Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
+A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
+already fixed in all scripts in this repo.
diff --git a/hetzner/proxmox-setup b/hetzner/proxmox-setup
index c4ac4c7..0cf0f8f 100755
--- a/hetzner/proxmox-setup
+++ b/hetzner/proxmox-setup
@@ -34,7 +34,6 @@ WG_PORT="51820"
 SKIP_REPOS=false
 SKIP_WIREGUARD=false
 SKIP_FIREWALL=false
-SKIP_STORAGE=false
 
 # Derived (computed after argument parsing)
 SUBNET_BASE=""          # e.g. 10.10.0
@@ -72,12 +71,7 @@ DETECT_VMBR1_IP=""              # e.g. "10.10.0.1/24" or empty
 DETECT_IP_FORWARD=""            # "1" or "0"
 DETECT_NAT_ACTIVE=""            # "yes" or empty
 
-# Storage
-STATE_STORAGE="done"            # done | available | needed
 DETECT_ZPOOLS=""
-DETECT_PVE_STORAGE=""
-DETECT_AVAILABLE_DISKS=""
-DETECT_AVAILABLE_DISK_COUNT="0"
 
 # WireGuard
 STATE_WIREGUARD="needed"        # done | needed
@@ -91,11 +85,16 @@ STATE_FIREWALL="needed"         # done | needed
 DETECT_NFT_RULES=""             # "yes" if nftables rules loaded with our pattern
 DETECT_NFT_FILE=""              # "yes" if config file exists
 
+# DNS (dnsmasq on vmbr1 -- required for IncusOS boot)
+STATE_DNSMASQ="needed"          # done | needed
+DETECT_DNSMASQ_ACTIVE=""        # "yes" if dnsmasq is installed and listening on vmbr1
+
 # API token
 STATE_API="needed"              # done | partial | needed
 DETECT_API_ROLE=""              # "yes" or empty
 DETECT_API_POOL=""              # "yes" or empty
 DETECT_API_TOKEN=""             # "yes" or empty
+DETECT_API_ACLS=""              # "yes" if /vms ACL is set
 
 # ---------------------------------------------------------------------------
 # Color and formatting
@@ -206,7 +205,6 @@ ${BOLD}DESCRIPTION${RESET}
 
     - No-subscription repositories and system update
     - Private bridge (vmbr1) with NAT masquerading
-    - ZFS storage pool on extra disks
     - WireGuard tunnel for secure remote access
     - nftables firewall lockdown
     - API token for incusos-proxmox automation
@@ -220,7 +218,6 @@ ${BOLD}OPTIONS${RESET}
     ${BOLD}--wg-subnet CIDR${RESET}      WireGuard subnet (default: ${WG_SUBNET})
     ${BOLD}--wg-port PORT${RESET}        WireGuard listen port (default: ${WG_PORT})
     ${BOLD}--skip-repos${RESET}          Skip repository changes
-    ${BOLD}--skip-storage${RESET}        Skip storage pool creation
     ${BOLD}--skip-wireguard${RESET}      Skip WireGuard setup
     ${BOLD}--skip-firewall${RESET}       Skip firewall lockdown
     ${BOLD}-n, --dry-run${RESET}         Preview actions without executing
@@ -282,10 +279,6 @@ parse_args() {
                 SKIP_REPOS=true
                 shift
                 ;;
-            --skip-storage)
-                SKIP_STORAGE=true
-                shift
-                ;;
             --skip-wireguard)
                 SKIP_WIREGUARD=true
                 shift
@@ -389,8 +382,6 @@ echo '---BLOCK_DEVICES---'
 lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
 echo '---ZPOOLS---'
 zpool list -H 2>/dev/null || true
-echo '---PVE_STORAGE---'
-pvesm status 2>/dev/null || true
 echo '---ENTERPRISE_REPOS---'
 # Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats
 enterprise_active='no'
@@ -466,6 +457,16 @@ echo '---NFT_FILE---'
 test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
 echo '---NFT_RULES---'
 nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
+echo '---DNSMASQ---'
+# dnsmasq installed and listening on port 53 on vmbr1?
+dnsmasq_ok='no'
+if command -v dnsmasq >/dev/null 2>&1 && systemctl is-active dnsmasq >/dev/null 2>&1; then
+    # Check it's listening on the bridge IP
+    if ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq; then
+        dnsmasq_ok='yes'
+    fi
+fi
+echo "$dnsmasq_ok"
 echo '---API_ROLE---'
 pveum role list --output-format json 2>/dev/null \
     | python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \
@@ -478,30 +479,10 @@ echo '---API_TOKEN---'
 pveum user token list automation@pve --output-format json 2>/dev/null \
     | python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \
     || echo 'no'
-echo '---AVAILABLE_DISKS---'
-# Build list of whole-disk names that are part of ZFS pools.
-# zpool status shows partition names (e.g. nvme0n1p3), so strip the
-# partition suffix to get the parent disk name.
-zfs_disks=$(zpool status 2>/dev/null \
-    | grep -oP '(nvme\S+|sd[a-z]+\S*|vd[a-z]+\S*)' \
-    | sed -E 's/p?[0-9]+$//' \
-    | sort -u || true)
-lsblk -dno NAME,SIZE,TYPE | while read name size type; do
-    [ "$type" != 'disk' ] && continue
-    # Skip disks used by ZFS pools
-    echo "$zfs_disks" | grep -qw "$name" && continue
-    # Skip disks with active mount points
-    lsblk -no MOUNTPOINT "/dev/$name" 2>/dev/null | grep -q '/' && continue
-    # Check for existing partition table / signatures
-    partcount=$(lsblk -no NAME "/dev/$name" 2>/dev/null | wc -l)
-    if [ "$partcount" -gt 1 ]; then
-        # Has partitions -- detect type
-        fstype=$(lsblk -no FSTYPE "/dev/$name" 2>/dev/null | grep -v '^$' | head -1)
-        echo "$name $size partitioned:${fstype:-unknown}"
-    else
-        echo "$name $size clean"
-    fi
-done
+echo '---API_ACLS---'
+pveum acl list --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; acls=json.load(sys.stdin); paths={a.get("path") for a in acls if "automation@pve" in a.get("ugid","")}; ok=("/vms" in paths and "/sdn" in paths); print("yes" if ok else "no")' 2>/dev/null \
+    || echo 'no'
 echo '---END---'
 DETECT
 )
@@ -545,7 +526,6 @@ _parse_section() {
         PUBLIC_IP)      DETECT_PUBLIC_IP="$content" ;;
         BLOCK_DEVICES)  DETECT_BLOCK_DEVICES="$content" ;;
         ZPOOLS)         DETECT_ZPOOLS="$content" ;;
-        PVE_STORAGE)    DETECT_PVE_STORAGE="$content" ;;
         ENTERPRISE_REPOS)
             if [[ "$content" == "yes" ]]; then
                 DETECT_ENTERPRISE_ACTIVE="yes"
@@ -577,6 +557,9 @@ _parse_section() {
             ;;
         NFT_FILE)       DETECT_NFT_FILE="$content" ;;
         NFT_RULES)      DETECT_NFT_RULES="$content" ;;
+        DNSMASQ)
+            if [[ "$content" == "yes" ]]; then DETECT_DNSMASQ_ACTIVE="yes"; fi
+            ;;
         API_ROLE)
             if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi
             ;;
@@ -586,13 +569,8 @@ _parse_section() {
         API_TOKEN)
             if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi
             ;;
-        AVAILABLE_DISKS)
-            DETECT_AVAILABLE_DISKS="$content"
-            if [[ -n "$content" ]]; then
-                DETECT_AVAILABLE_DISK_COUNT=$(echo "$content" | wc -l)
-            else
-                DETECT_AVAILABLE_DISK_COUNT="0"
-            fi
+        API_ACLS)
+            if [[ "$content" == "yes" ]]; then DETECT_API_ACLS="yes"; fi
             ;;
     esac
 }
@@ -625,21 +603,6 @@ _compute_states() {
         STATE_BRIDGE="needed"
     fi
 
-    # Storage: check if any ZFS pool is registered as PVE storage
-    local zfs_in_pve
-    zfs_in_pve=$(echo "$DETECT_PVE_STORAGE" | grep -c "zfspool" || true)
-    if [[ "$zfs_in_pve" -gt 0 ]]; then
-        if [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
-            STATE_STORAGE="available"  # has pool but also unused disks
-        else
-            STATE_STORAGE="done"
-        fi
-    elif [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
-        STATE_STORAGE="needed"
-    else
-        STATE_STORAGE="done"  # no pool, but no disks available either
-    fi
-
     # WireGuard
     if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
         STATE_WIREGUARD="done"
@@ -655,13 +618,20 @@ _compute_states() {
     fi
 
     # API token
-    if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]]; then
+    if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]] && [[ "$DETECT_API_ACLS" == "yes" ]]; then
         STATE_API="done"
     elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
         STATE_API="partial"
     else
         STATE_API="needed"
     fi
+
+    # dnsmasq (DNS for VMs on vmbr1 -- required for IncusOS boot)
+    if [[ "$DETECT_DNSMASQ_ACTIVE" == "yes" ]]; then
+        STATE_DNSMASQ="done"
+    else
+        STATE_DNSMASQ="needed"
+    fi
 }
 
 # ---------------------------------------------------------------------------
@@ -710,12 +680,10 @@ show_status() {
 
     local effective_repos="$STATE_REPOS"
     local effective_update="$STATE_UPDATE"
-    local effective_storage="$STATE_STORAGE"
     local effective_wg="$STATE_WIREGUARD"
     local effective_fw="$STATE_FIREWALL"
 
     if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi
-    if [[ "$SKIP_STORAGE" == true ]]; then effective_storage="skipped"; fi
     if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi
     if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi
 
@@ -748,14 +716,6 @@ show_status() {
         needed)  echo "  vmbr1 not yet created" ;;
     esac
 
-    printf "  %-20s %b" "Storage" "$(status_icon "$effective_storage")"
-    case "$effective_storage" in
-        done)      echo "  system pool registered, no extra disks" ;;
-        available) echo "  system pool registered, ${DETECT_AVAILABLE_DISK_COUNT} extra disk(s) available" ;;
-        needed)    echo "  ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s), no pool created" ;;
-        skipped)   echo "" ;;
-    esac
-
     printf "  %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
     case "$effective_wg" in
         done)    echo "  wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
@@ -774,9 +734,9 @@ show_status() {
         done)    echo "  nftables rules active" ;;
         needed)
             if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
-                echo "  config file exists but rules not loaded"
+                echo "  config written, apply manually (see firewall step)"
             else
-                echo "  no rules configured"
+                echo "  config not yet written"
             fi
             ;;
         skipped) echo "" ;;
@@ -795,6 +755,12 @@ show_status() {
         needed)  echo "  not configured" ;;
     esac
 
+    printf "  %-20s %b" "DNS (dnsmasq)" "$(status_icon "$STATE_DNSMASQ")"
+    case "$STATE_DNSMASQ" in
+        done)   echo "  dnsmasq active, forwarding DNS on vmbr1" ;;
+        needed) echo "  not installed (required for IncusOS boot)" ;;
+    esac
+
     echo ""
 
     # Count pending items
@@ -802,10 +768,10 @@ show_status() {
     if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi
     if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi
     if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi
-    if [[ "$effective_storage" == "needed" ]]; then pending=$((pending + 1)); fi
     if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi
     if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi
     if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi
+    if [[ "$STATE_DNSMASQ" == "needed" ]]; then pending=$((pending + 1)); fi
 
     if [[ "$pending" -eq 0 ]]; then
         success "All configured -- nothing to do"
@@ -1001,126 +967,46 @@ iface vmbr1 inet static
 }
 
 # ---------------------------------------------------------------------------
-# Apply: storage
+# Apply: dnsmasq (DNS for VMs on vmbr1)
 # ---------------------------------------------------------------------------
 
-apply_storage() {
-    if [[ "$SKIP_STORAGE" == true ]] || [[ "$STATE_STORAGE" == "done" ]]; then
+apply_dnsmasq() {
+    if [[ "$STATE_DNSMASQ" == "done" ]]; then
         return 0
     fi
 
-    step "ZFS storage pool"
+    step "DNS (dnsmasq on vmbr1)"
 
-    if [[ "$STATE_STORAGE" == "available" ]]; then
-        info "ZFS pool already registered with Proxmox"
-        info "${DETECT_AVAILABLE_DISK_COUNT} additional unused disk(s) available:"
-        echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
-            if [[ -n "$line" ]]; then echo "          /dev/$line"; fi
-        done
-        if ! confirm "Create an additional storage pool?"; then
-            return 0
-        fi
-    fi
+    info "Will: install dnsmasq, configure it to forward DNS queries from VMs on vmbr1"
+    info "  Interface: vmbr1 (${SUBNET_GW})"
+    info "  Upstream:  1.1.1.1, 8.8.8.8"
+    info "  Required for IncusOS boot (downloads updates from images.linuxcontainers.org)"
 
-    if [[ "$STATE_STORAGE" == "needed" ]]; then
-        # Check if a zpool named local-zfs already exists but isn't in PVE
-        local existing_pool
-        existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true)
-        if [[ -n "$existing_pool" ]]; then
-            info "ZFS pool 'local-zfs' exists but not registered with Proxmox"
-            if confirm "Register local-zfs with Proxmox?"; then
-                if ! dry_run_guard "pvesm add zfspool local-zfs"; then
-                    remote "pvesm add zfspool local-zfs -pool local-zfs"
-                    remote "pvesm set local-zfs -content images,rootdir"
-                    success "local-zfs registered"
-                fi
-            fi
-            return 0
-        fi
-    fi
-
-    if [[ "$DETECT_AVAILABLE_DISK_COUNT" -eq 0 ]]; then
-        info "No unused disks found"
-        return 0
-    fi
-
-    info "Available disks:"
-    local has_partitioned=false
-    echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
-        if [[ -n "$line" ]]; then
-            local dname dsize dstate
-            dname=$(echo "$line" | awk '{print $1}')
-            dsize=$(echo "$line" | awk '{print $2}')
-            dstate=$(echo "$line" | awk '{print $3}')
-            if [[ "$dstate" == partitioned:* ]]; then
-                local fsinfo="${dstate#partitioned:}"
-                echo -e "          /dev/${dname}  ${dsize}  ${YELLOW}has existing partitions (${fsinfo})${RESET}"
-            else
-                echo "          /dev/${dname}  ${dsize}  clean"
-            fi
-        fi
-    done
-
-    # Check for partitioned disks that need wiping
-    local partitioned_disks
-    partitioned_disks=$(echo "$DETECT_AVAILABLE_DISKS" | awk '/partitioned:/{print "/dev/"$1}' | tr '\n' ' ')
-    if [[ -n "${partitioned_disks// /}" ]]; then
-        has_partitioned=true
-        warn "Disk(s) with existing partitions must be wiped before use in a ZFS pool"
-        warn "Affected: ${partitioned_disks}"
-    fi
-
-    local disk_count="$DETECT_AVAILABLE_DISK_COUNT"
-    local disk_names
-    disk_names=$(echo "$DETECT_AVAILABLE_DISKS" | awk 'NF{print "/dev/"$1}' | tr '\n' ' ')
-
-    local pool_type
-    if [[ "$disk_count" -eq 1 ]]; then
-        pool_type="single"
-        info "Recommendation: single disk (no redundancy)"
-    elif [[ "$disk_count" -eq 2 ]]; then
-        pool_type="mirror"
-        info "Recommendation: mirror (2 disks, full redundancy)"
-    else
-        pool_type="raidz1"
-        info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)"
-    fi
-
-    local pool_name
-    pool_name=$(prompt_value "Pool name" "vm-storage")
-
-    if [[ "$has_partitioned" == true ]]; then
-        warn "This will wipe partition tables on: ${partitioned_disks}"
-    fi
-
-    if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then
+    if ! confirm "Install and configure dnsmasq?"; then
         info "Skipped"
         return 0
     fi
 
-    if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then
-        return 0
-    fi
+    dry_run_guard
 
-    # Wipe partition tables on disks that have them
-    for disk in $partitioned_disks; do
-        info "Wiping partition table on ${disk}..."
-        remote "wipefs -a ${disk}"
-        remote "sgdisk -Z ${disk} 2>/dev/null || true"
-    done
+    remote "apt-get install -y dnsmasq"
 
-    local zpool_cmd="zpool create -f ${pool_name}"
-    case "$pool_type" in
-        mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;;
-        raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;;
-        single) zpool_cmd="${zpool_cmd} ${disk_names}" ;;
-    esac
+    local dnsmasq_conf
+    dnsmasq_conf="# DNS for VMs on vmbr1 (${SUBNET})
+# Generated by proxmox-setup -- do not edit manually
+interface=vmbr1
+bind-interfaces
+no-resolv
+server=1.1.1.1
+server=8.8.8.8
+domain-needed
+bogus-priv"
 
-    remote "$zpool_cmd"
-    remote "pvesm add zfspool ${pool_name} -pool ${pool_name}"
-    remote "pvesm set ${pool_name} -content images,rootdir"
+    remote_write "/etc/dnsmasq.d/vmbr1.conf" "$dnsmasq_conf"
+    remote "systemctl enable dnsmasq && systemctl restart dnsmasq"
+    remote "ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq && echo 'OK' || echo 'WARN: dnsmasq not listening on port 53'"
 
-    success "ZFS pool '${pool_name}' created and registered"
+    success "dnsmasq configured on vmbr1"
 }
 
 # ---------------------------------------------------------------------------
@@ -1144,6 +1030,21 @@ apply_wireguard() {
         return 0
     fi
 
+    if [[ "$DRY_RUN" == true ]]; then
+        info "(dry-run) Client config will look like:"
+        echo ""
+        echo "  [Interface]"
+        echo "  PrivateKey = <generated on setup>"
+        echo "  Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}"
+        echo ""
+        echo "  [Peer]"
+        echo "  PublicKey = <server public key>"
+        echo "  Endpoint = ${DETECT_PUBLIC_IP:-<server-public-ip>}:${WG_PORT}"
+        echo "  AllowedIPs = 10.10.0.0/16"
+        echo "  PersistentKeepalive = 25"
+        echo ""
+    fi
+
     if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
         return 0
     fi
@@ -1206,6 +1107,22 @@ AllowedIPs = ${WG_CLIENT_IP}/32"
     echo "  PersistentKeepalive = 25"
     echo ""
     warn "Save the client config now -- private key is not stored anywhere else"
+    echo ""
+    info "Verify WireGuard before applying firewall rules"
+    if confirm "Connect your WireGuard client now and verify the tunnel?"; then
+        info "Checking server-side WireGuard status..."
+        local wg_status
+        wg_status=$(remote_output "wg show wg0 2>/dev/null" || true)
+        if echo "$wg_status" | grep -q "latest handshake"; then
+            success "WireGuard is working -- handshake confirmed"
+        elif echo "$wg_status" | grep -q "peer:"; then
+            warn "Peer configured but no handshake yet -- client connected?"
+            info "Try: ping ${WG_SERVER_IP}"
+        else
+            warn "No peers visible -- check wg0 is up on server"
+            remote_output "systemctl status wg-quick@wg0 --no-pager -l" || true
+        fi
+    fi
 }
 
 # ---------------------------------------------------------------------------
@@ -1219,31 +1136,6 @@ apply_firewall() {
 
     step "Firewall lockdown (nftables)"
 
-    if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
-        info "Config file exists but rules not loaded"
-        if confirm "Load existing firewall rules from /etc/nftables-hetzner.conf?"; then
-            if ! dry_run_guard "nft -f /etc/nftables-hetzner.conf"; then
-                remote "nft -f /etc/nftables-hetzner.conf"
-                remote "systemctl enable nftables"
-                success "Firewall rules loaded"
-            fi
-        fi
-        return 0
-    fi
-
-    info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only"
-    info "Web UI (8006) and VM traffic only via WireGuard"
-    warn "Make sure WireGuard is working before applying firewall rules!"
-
-    if ! confirm "Apply firewall rules?"; then
-        info "Skipped"
-        return 0
-    fi
-
-    if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then
-        return 0
-    fi
-
     local nft_rules="#!/usr/sbin/nft -f
 
 flush ruleset
@@ -1295,12 +1187,26 @@ table inet filter {
     }
 }"
 
-    remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
-    remote "nft -f /etc/nftables-hetzner.conf"
-    remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf"
-    remote "systemctl enable nftables"
+    if [[ "$DETECT_NFT_FILE" != "yes" ]]; then
+        if dry_run_guard "Write /etc/nftables-hetzner.conf on ${SSH_HOST}"; then
+            info "Rules that would be written to /etc/nftables-hetzner.conf:"
+            echo "$nft_rules" | sed 's/^/    /'
+            echo ""
+        else
+            remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
+            success "Config written to /etc/nftables-hetzner.conf"
+        fi
+    else
+        info "Config already at /etc/nftables-hetzner.conf (rules not yet loaded)"
+    fi
 
-    success "Firewall applied (SSH + WireGuard only on public interface)"
+    warn "Firewall rules are NOT applied automatically — misfire locks out SSH on Hetzner permanently"
+    info "Once WireGuard is verified, apply manually:"
+    echo ""
+    echo "  ssh ${SSH_HOST} nft -f /etc/nftables-hetzner.conf"
+    echo "  ssh ${SSH_HOST} systemctl enable nftables"
+    echo "  ssh ${SSH_HOST} nft list ruleset    # verify"
+    echo ""
 }
 
 # ---------------------------------------------------------------------------
@@ -1319,6 +1225,7 @@ apply_api_token() {
     [[ "$DETECT_API_ROLE" == "yes" ]]  || actions="create IncusOSDeployer role"
     [[ "$DETECT_API_POOL" == "yes" ]]  || actions="${actions:+$actions, }create IncusLab pool"
     [[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
+    [[ "$DETECT_API_ACLS" == "yes" ]]  || actions="${actions:+$actions, }add missing ACLs (/vms, /sdn, storage)"
 
     info "Will: ${actions}"
 
@@ -1335,9 +1242,8 @@ apply_api_token() {
     if [[ "$DETECT_API_ROLE" != "yes" ]]; then
         remote "pveum role add IncusOSDeployer -privs \
             'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
-             VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
-             VM.Monitor VM.Audit VM.Console \
-             Datastore.AllocateSpace Datastore.Audit \
+             VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Audit \
+             Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
              Pool.Allocate Pool.Audit \
              SDN.Use Sys.Modify'"
         success "Created IncusOSDeployer role"
@@ -1360,8 +1266,22 @@ apply_api_token() {
         local token_secret
         token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
 
-        # Assign ACLs
+        success "API token created"
+        echo ""
+        warn "Token secret (save now -- shown only once):"
+        echo ""
+        echo "  ${token_secret}"
+        echo ""
+    fi
+
+    # Assign ACLs (idempotent -- run whenever any ACL is missing).
+    # PVE 9.x requires /sdn for bridge-backed VMs even with SDN.Use in the role.
+    # /vms and /nodes/<node> are required for VM creation and start via API.
+    if [[ "$DETECT_API_ACLS" != "yes" ]]; then
+        remote "pveum acl modify /vms -user automation@pve -role IncusOSDeployer"
+        remote "pveum acl modify /nodes/${DETECT_HOSTNAME} -user automation@pve -role IncusOSDeployer"
         remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
+        remote "pveum acl modify /sdn -user automation@pve -role IncusOSDeployer"
 
         local storage_name
         storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
@@ -1370,12 +1290,7 @@ apply_api_token() {
         fi
         remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
 
-        success "API token created"
-        echo ""
-        warn "Token secret (save now -- shown only once):"
-        echo ""
-        echo "  ${token_secret}"
-        echo ""
+        success "ACLs configured"
     fi
 }
 
@@ -1406,7 +1321,6 @@ show_summary() {
     echo ""
     echo "  # Hetzner"
     echo "  HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
-    echo "  HETZNER_PROXMOX_ROOT_PASSWORD=<your root password>"
 
     echo ""
     echo -e "${BOLD}Next steps:${RESET}"
@@ -1457,7 +1371,7 @@ main() {
     apply_repos
     apply_update
     apply_bridge
-    apply_storage
+    apply_dnsmasq
     apply_wireguard
     apply_firewall
     apply_api_token
diff --git a/incusos/deploy-awx b/incusos/deploy-awx
index 316d888..f5224c0 100755
--- a/incusos/deploy-awx
+++ b/incusos/deploy-awx
@@ -22,6 +22,7 @@ readonly AWX_NAMESPACE="awx"
 
 VM_NAME="awx"
 TARGET_REMOTE="oc-node-02"
+TARGET_NODE=""   # Node within cluster for --target; defaults to TARGET_REMOTE if empty
 AWX_IP="192.168.102.161"
 AWX_PREFIX="22"
 AWX_GATEWAY="192.168.100.1"
@@ -39,6 +40,7 @@ AETHER_CLUSTER_ID="52"
 DRY_RUN=false
 VERBOSE=false
 QUIET=false
+RESUME=false
 CONFIG_FILE=""
 ACTION=""
 
@@ -163,8 +165,10 @@ parse_config() {
     local val
     val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && VM_NAME="$val"
-    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && TARGET_REMOTE="$val"
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && TARGET_NODE="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     if [[ -n "$val" ]]; then
         AWX_IP="${val%%/*}"
@@ -184,7 +188,7 @@ parse_config() {
     [[ -n "$val" ]] && GIT_REPO="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && GIT_BRANCH="$val"
-    val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_URL="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
@@ -375,8 +379,10 @@ action_deploy() {
     echo
 
     if vm_exists; then
-        if vm_running; then
-            error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first."
+        if [[ "${RESUME:-false}" == true ]]; then
+            info "AWX VM already exists — resuming from phase 4"
+        elif vm_running; then
+            error "AWX VM already exists and is running. Use --status, --heal, --resume, or --cleanup first."
             return 1
         else
             warn "AWX VM exists but is not running — starting it"
@@ -387,9 +393,11 @@ action_deploy() {
         fi
     fi
 
-    phase_create_vm
-    phase_configure_network
-    phase_install_k3s
+    if [[ "${RESUME:-false}" != true ]]; then
+        phase_create_vm
+        phase_configure_network
+        phase_install_k3s
+    fi
     phase_deploy_awx_operator
     phase_deploy_awx_instance
     phase_verify
@@ -410,12 +418,13 @@ action_deploy() {
 phase_create_vm() {
     step "Phase 1: Create VM"
 
-    if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then
+    local target_node="${TARGET_NODE:-$TARGET_REMOTE}"
+    if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${target_node} ..."; then
         return 0
     fi
 
     incus launch images:debian/12 "$(vm_ref)" --vm \
-        --target "$TARGET_REMOTE" \
+        --target "$target_node" \
         -c "limits.cpu=${AWX_CPU}" \
         -c "limits.memory=${AWX_MEMORY}" \
         -d "root,size=${AWX_DISK}"
@@ -490,7 +499,7 @@ phase_install_k3s() {
     vm_exec bash -c "
         export DEBIAN_FRONTEND=noninteractive
         apt-get update -qq
-        apt-get install -y -qq curl python3 2>&1 | tail -1
+        apt-get install -y -qq curl python3 git 2>&1 | tail -1
     "
 
     vm_exec bash -c "
@@ -1162,6 +1171,7 @@ parse_args() {
             --cleanup)     ACTION="cleanup";;
             --doctor)      ACTION="doctor";;
             -c|--config)   CONFIG_FILE="$2"; shift;;
+            --resume)      RESUME=true;;
             -n|--dry-run)  DRY_RUN=true;;
             -v|--verbose)  VERBOSE=true;;
             -q|--quiet)    QUIET=true;;
diff --git a/incusos/deploy-haproxy b/incusos/deploy-haproxy
index 916d508..b9b1bff 100755
--- a/incusos/deploy-haproxy
+++ b/incusos/deploy-haproxy
@@ -184,7 +184,7 @@ parse_config() {
     [[ -n "$val" ]] && SERVICE_NAME="$val"
     val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && IMAGE_VERSION="$val"
-    val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_URL="$val"
 }
 
diff --git a/incusos/deploy-observability b/incusos/deploy-observability
index 396893c..89632a2 100755
--- a/incusos/deploy-observability
+++ b/incusos/deploy-observability
@@ -43,6 +43,7 @@ OC_SERVER_IP=""
 DRY_RUN=false
 VERBOSE=false
 QUIET=false
+RESUME=false
 ACTION=""
 MANUAL_NODES=""
 
@@ -101,9 +102,12 @@ ${BOLD}ACTIONS${RESET}
     --doctor          Diagnose common issues
 
 ${BOLD}OPTIONS${RESET}
-    -r, --remote NAME     Incus remote (default: ${CLUSTER_REMOTE})
-    --nodes NODE1,NODE2   Manual node list (skip auto-discovery)
-    --oc-server IP        Monitor oc-server at this IP (proxy device for 9100)
+    -r, --remote NAME         Incus remote (default: ${CLUSTER_REMOTE})
+    --nodes NODE1,NODE2       Manual node list (skip auto-discovery)
+    --monitoring-target NODE  Node to deploy monitoring container on (default: ${MONITORING_TARGET})
+    --monitoring-ip IP        IP for monitoring container (default: ${MONITORING_IP})
+    --forward-vip IP          OVN external IP for Grafana/Prometheus forward (default: ${FORWARD_VIP})
+    --oc-server IP            Monitor oc-server at this IP (proxy device for 9100)
     -n, --dry-run         Preview actions without executing
     -v, --verbose         Show detailed output
     -q, --quiet           Suppress informational output
@@ -298,16 +302,22 @@ action_deploy() {
     echo
 
     if container_exists "monitoring"; then
-        error "Monitoring container already exists"
-        error "Use --status to check health, or --cleanup first."
-        return 1
+        if [[ "$RESUME" == true ]]; then
+            info "Monitoring container already exists — resuming from phase 5b"
+        else
+            error "Monitoring container already exists"
+            error "Use --status to check health, --cleanup first, or --resume to continue."
+            return 1
+        fi
     fi
 
-    phase_deploy_monitoring
-    phase_install_prometheus
-    phase_install_loki
-    phase_install_grafana
-    phase_deploy_node_exporters
+    if [[ "$RESUME" != true ]]; then
+        phase_deploy_monitoring
+        phase_install_prometheus
+        phase_install_loki
+        phase_install_grafana
+        phase_deploy_node_exporters
+    fi
     phase_deploy_oc_server_exporter
     phase_create_acl
     phase_configure_haproxy_exporter
@@ -1706,9 +1716,13 @@ parse_args() {
             --status)      ACTION="status";;
             --cleanup)     ACTION="cleanup";;
             --doctor)      ACTION="doctor";;
-            -r|--remote)   CLUSTER_REMOTE="$2"; shift;;
-            --nodes)       MANUAL_NODES="$2"; shift;;
-            --oc-server)   OC_SERVER_IP="$2"; shift;;
+            -r|--remote)           CLUSTER_REMOTE="$2"; shift;;
+            --nodes)               MANUAL_NODES="$2"; shift;;
+            --monitoring-target)   MONITORING_TARGET="$2"; shift;;
+            --monitoring-ip)       MONITORING_IP="$2"; shift;;
+            --forward-vip)         FORWARD_VIP="$2"; shift;;
+            --oc-server)           OC_SERVER_IP="$2"; shift;;
+            --resume)              RESUME=true;;
             -n|--dry-run)  DRY_RUN=true;;
             -v|--verbose)  VERBOSE=true;;
             -q|--quiet)    QUIET=true;;
diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox
index 0866b75..e3f1322 100755
--- a/incusos/incusos-proxmox
+++ b/incusos/incusos-proxmox
@@ -1405,6 +1405,13 @@ json.dump(result, sys.stdout, indent=2)
     val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
     val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
     val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
+    # api_token_secret in proxmox.yaml overrides the repo-wide env file, useful
+    # for per-target configs where the env file holds a different target's token.
+    val=$(json_get "$px_json" ".api_token_secret" "")
+    if [[ -n "$val" ]]; then
+        export PROXMOX_TOKEN_SECRET="$val"
+        detail "Using api_token_secret from proxmox.yaml"
+    fi
 
     rm -f "$px_json"
 }
@@ -2281,6 +2288,7 @@ build_vm_description() {
 # Check if a VM has our management marker in its description
 pve_vm_is_managed() {
     local vmid="$1"
+    local encoded_marker="${MANAGED_MARKER//:/%3A}"
 
     if [[ "$DRY_RUN" == true ]]; then
         return 1
@@ -2296,7 +2304,8 @@ pve_vm_is_managed() {
             ;;
     esac
 
-    echo "$config_output" | grep -qF "$MANAGED_MARKER"
+    echo "$config_output" | grep -qF "$MANAGED_MARKER" || \
+        echo "$config_output" | grep -qF "$encoded_marker"
 }
 
 # Check if a VM still has install media (ide2) attached.
@@ -2397,15 +2406,15 @@ preflight_checks() {
     if [[ "$DRY_RUN" != true ]]; then
         case "$PVE_METHOD" in
             ssh)
-                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then
+                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_STORAGE"; then
                     error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
-                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
+                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
                     error "Fix: update 'proxmox.storage' in your config file"
                     errors=$((errors + 1))
                 fi
-                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then
+                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_ISO_STORAGE"; then
                     error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
-                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
+                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
                     error "Fix: update 'proxmox.iso_storage' in your config file"
                     errors=$((errors + 1))
                 fi
diff --git a/incusos/targets/hetzner/awx.yaml b/incusos/targets/hetzner/awx.yaml
new file mode 100644
index 0000000..31f692b
--- /dev/null
+++ b/incusos/targets/hetzner/awx.yaml
@@ -0,0 +1,27 @@
+# awx.yaml - AWX deployment config for Hetzner lab
+#
+# Deploys a Debian 12 VM with K3s + AWX Operator on hz-node-03.
+# VM uses macvlan parent=mgmt for direct access on the 10.10.0.0/24 network.
+#
+# Architecture:
+#   awx  (10.10.0.122)  - K3s + AWX Operator + AWX instance
+#   AWX exposed on NodePort 30080 (HTTP)
+#
+# Usage:
+#   source env
+#   ./incusos/deploy-awx \
+#     --config incusos/targets/hetzner/awx.yaml \
+#     --deploy
+
+awx:
+  vm_name: awx
+  target_remote: hz-cluster
+  target_node: hz-node-03
+  ip: 10.10.0.122/24
+  gateway: 10.10.0.1
+  dns: 10.10.0.1
+  cpu: 4
+  memory: 8GiB
+  disk: 40GiB
+  aether_url: https://10.10.0.120:8443
+  aether_cluster_id: 52
diff --git a/incusos/targets/hetzner/haproxy.yaml b/incusos/targets/hetzner/haproxy.yaml
new file mode 100644
index 0000000..f6c6cfd
--- /dev/null
+++ b/incusos/targets/hetzner/haproxy.yaml
@@ -0,0 +1,28 @@
+# haproxy.yaml - HAProxy load balancing config for Hetzner lab
+#
+# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
+# with a VIP on the UPLINK external range (10.10.0.200-210).
+#
+# Architecture:
+#   ffsdn-haproxy-52-01  (10.10.10.50)  - HAProxy instance 1 on net-prod
+#   ffsdn-haproxy-52-02  (10.10.10.51)  - HAProxy instance 2 on net-prod
+#   nginx-lb-01/02/03    (10.10.10.60-62) - nginx test backends on net-prod
+#   VIP:                  10.10.0.200:80  - OVN external IP for external access
+#
+# Usage:
+#   source env
+#   ./incusos/deploy-haproxy \
+#     --config incusos/targets/hetzner/haproxy.yaml \
+#     --deploy
+
+haproxy:
+  cluster_remote: hz-cluster
+  cluster_id: 52
+  ovn_network: net-prod
+  vip: 10.10.0.200
+  haproxy_01_ip: 10.10.10.50
+  haproxy_02_ip: 10.10.10.51
+  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
+  service_name: web-test
+  image_version: "1.0.0"
+  aether_url: https://10.10.0.120:8443
diff --git a/incusos/targets/hetzner/lab-production.yaml b/incusos/targets/hetzner/lab-production.yaml
index 00abcc8..932fa58 100644
--- a/incusos/targets/hetzner/lab-production.yaml
+++ b/incusos/targets/hetzner/lab-production.yaml
@@ -1,6 +1,6 @@
 # lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
 #
-# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering.
+# Deploys 1 Operations Center instance and 3 Incus nodes for OC-managed clustering.
 # Sized for a Hetzner dedicated server with generous resources.
 #
 # Architecture:
@@ -12,7 +12,7 @@
 # Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
 # OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
 #
-# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads)
+# RAM budget: ~248 GiB of 256 GiB (8 GiB OC + 3×80 GiB nodes; ~8 GiB reserved for Proxmox host)
 #
 # Usage:
 #   incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
@@ -26,8 +26,8 @@
 # Connection settings from proxmox.yaml in this directory.
 
 defaults:
-  cores: 8
-  memory: 16384
+  cores: 16
+  memory: 81920            # ~80 GiB per node (3 nodes × 80 = 240 GiB + 8 for OC = 248 GiB total)
   disk: 100
   start_vmid: 910
 
@@ -36,12 +36,12 @@ vms:
     app: operations-center
     apply_defaults: true
     cores: 4
-    memory: 8192
+    memory: 8192            # OC needs very little RAM
     ip: 10.10.0.110/24
 
   - name: hz-node-01
     app: incus
-    apply_defaults: true          # init node: needs storage pool + network
+    apply_defaults: false         # OC manages cluster formation + storage/network setup
     disk: 150                     # extra space for OVN control plane container
     ip: 10.10.0.111/24
 
diff --git a/incusos/targets/hetzner/proxmox.yaml.example b/incusos/targets/hetzner/proxmox.yaml.example
index e2cf20d..6985492 100644
--- a/incusos/targets/hetzner/proxmox.yaml.example
+++ b/incusos/targets/hetzner/proxmox.yaml.example
@@ -14,6 +14,7 @@
 host: 10.10.0.1               # Proxmox host IP (via WireGuard, on vmbr1)
 method: api                    # api recommended (WireGuard may add SSH latency)
 api_token_id: automation@pve!deploy
+api_token_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  # token secret (overrides env PROXMOX_TOKEN_SECRET)
 node: pve                      # Proxmox node name (adjust if different)
 storage: local-zfs             # VM disk storage (ZFS pool)
 iso_storage: local             # ISO/image storage