# Hetzner IncusOS Lab — End-to-End Guide

Full deployment guide for a production-quality IncusOS lab on a Hetzner
dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
networking, HAProxy load balancing, AWX automation, and observability.

**Estimated time**: 3-4 hours for a clean run.

---

## Overview

```
Hetzner dedicated (256 GB RAM, 32+ cores)
└─ Proxmox VE 9
   ├─ hz-oc       (VMID 910) — Operations Center 0.4.x
   ├─ hz-node-01  (VMID 911) — Incus cluster init + OVN control plane
   ├─ hz-node-02  (VMID 912) — Incus cluster member
   └─ hz-node-03  (VMID 913) — Incus cluster member
      └─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
         ├─ ffsdn-haproxy-52-01/02  — HAProxy HA pair
         ├─ nginx-lb-01/02/03       — Test backends
         ├─ monitoring              — Prometheus + Grafana
         ├─ node-exp-01/02/03       — Node exporters
         └─ awx                     — AWX Operator on K3s
```

**Network topology:**
- vmbr0: Public interface (SSH + WireGuard only)
- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
- OVN overlay: 10.10.10.0/24 (containers on net-prod)
- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)

---

## Prerequisites

- Hetzner dedicated server with Proxmox VE 9 installed and configured
  (see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
- This repository cloned locally
- `env` file populated (see below)
- `envs/hetzner/proxmox.yaml` filled in
- Incus client installed on workstation

### env file

```bash
# incusos-contrib/env
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
export AETHER_ADMIN_PASSWORD="admin1234"
```

### proxmox.yaml

```bash
cp envs/hetzner/proxmox.yaml.example \
   envs/hetzner/proxmox.yaml
# Fill in: host, node name, token_id, storage
```

---

## 1. Deploy IncusOS VMs

```bash
source env

bin/incusos-proxmox \
    --proxmox envs/hetzner/proxmox.yaml \
    envs/hetzner/lab-production.yaml
```

**What gets deployed:**

| VMID | Name      | IP            | RAM   | Cores | Disk |
|------|-----------|---------------|-------|-------|------|
| 910  | hz-oc     | 10.10.0.110   | 8 GB  | 4     | 100G |
| 911  | hz-node-01| 10.10.0.111   | 80 GB | 16    | 150G |
| 912  | hz-node-02| 10.10.0.112   | 80 GB | 16    | 100G |
| 913  | hz-node-03| 10.10.0.113   | 80 GB | 16    | 100G |

**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
interrupt. Monitor progress:

```bash
# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
source env
bin/helpers/proxmox-screenshot 910   # hz-oc
bin/helpers/proxmox-screenshot 911   # hz-node-01
```

Wait until all VMs are reachable:

```bash
for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
    echo -n "$ip: "
    ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
done
```

---

## 2. Add Incus remotes

Add the cluster remote pointing at hz-node-01 (the init node):

```bash
incus remote add hz-cluster https://10.10.0.111:8443 \
    --accept-certificate --auth-type tls
```

> **Note**: The `incus remote add` command prompts for a trust token.
> Obtain the token from OC after cluster formation (section 3), or
> if using a pre-formed cluster, generate one with:
> `incus config trust add --name workstation`

---

## 3. Cluster formation via Operations Center

Open the Operations Center UI:

```
https://10.10.0.110:8443
Username: admin
Password: admin   (first login, then set a new password)
```

### 3.1 Add Incus nodes

In OC: **Servers → Add Server** for each node:

| Field   | hz-node-01       | hz-node-02       | hz-node-03       |
|---------|------------------|------------------|------------------|
| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
| Trust   | via token        | via token        | via token        |

Each node's trust token is displayed on the IncusOS console on first boot.

### 3.2 Form the cluster

In OC: **Clusters → Create Cluster**

- Name: `hz-cluster`
- Init node: `hz-node-01`
- Members: `hz-node-02`, `hz-node-03`
- App seed: `{}` (empty JSON for manual post-formation setup)

OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.

Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
HAProxy and AWX configuration.

### 3.3 Verify cluster

```bash
incus cluster list hz-cluster:
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
```

---

## 4. OVN networking

### 4.1 Create the OVN uplink network

The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
`eth0` or `ens18`. This is specific to IncusOS.

```bash
# Create UPLINK on each node
incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-01 \
    config parent=mgmt

incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-02 \
    config parent=mgmt

incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-03 \
    config parent=mgmt

# Finalize UPLINK on cluster level
incus network create hz-cluster:UPLINK \
    config ipv4.address=none \
    config ipv6.address=none
```

### 4.2 Deploy OVN central

OVN central runs as a privileged container on hz-node-01:

```bash
incus launch images:debian/12 hz-cluster:ovn-central \
    --vm=false \
    --target hz-node-01 \
    -c security.privileged=true \
    -d root,size=20GiB

# Wait for agent
sleep 15

# Install ovn-central
incus exec hz-cluster:ovn-central -- bash -c "
    apt-get update -qq
    apt-get install -y -qq ovn-central
    systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
    systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
"
```

### 4.3 Expose OVN NB/SB ports to cluster

Create proxy devices so cluster nodes can reach OVN:

```bash
# Get ovn-central IP
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')

incus config device add hz-cluster:ovn-central ovn-nb proxy \
    listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
    bind=host
incus config device add hz-cluster:ovn-central ovn-sb proxy \
    listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
    bind=host
```

### 4.4 Create OVN overlay network

```bash
# Use hz-node-01 IP as OVN NB/SB server
NODE01_IP="10.10.0.111"

incus network create hz-cluster:net-prod \
    --type=ovn \
    config network=UPLINK \
    config ipv4.address=10.10.10.1/24 \
    config ipv4.nat=true \
    config ipv6.address=none \
    config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
    config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
```

Wait for the network to reach **Created** state:

```bash
incus network info hz-cluster:net-prod
```

### 4.5 Configure external IP range (OVN VIPs)

```bash
# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
# These are allocated on the Proxmox host's vmbr1 bridge
# No additional configuration needed — OVN will ARP for these IPs
```

Verify:

```bash
incus network list hz-cluster:
# net-prod should show type: ovn, state: Created
```

---

## 5. Deploy Aether

### 5.1 Deploy the Aether VM

Use the macvlan method to give Aether a direct vmbr1 IP:

```bash
incus launch images:ubuntu/22.04 hz-cluster:aether \
    --vm \
    --target hz-node-01 \
    -d root,size=40GiB \
    -c limits.cpu=4 \
    -c limits.memory=8GiB \
    -d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
```

> **Alternative**: use Aether golden image if already built on the cluster.
> Check `incus image list hz-cluster: | grep aether`.

Wait for the VM, then SSH in:

```bash
ssh ubuntu@10.10.0.120
```

Follow the [Aether installation guide](../notes/aether-guide.md) for the
full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.

### 5.2 Configure Aether cluster

In Aether UI (`https://10.10.0.120:8443`):

1. **Settings → Clusters → Add Cluster**
   - Name: hz-cluster
   - Endpoint: `https://10.10.0.111:8443`
   - Authenticate (generates a trust token, add it to the cluster)

2. **Settings → Operations Centers → Add OC**
   - URL: `https://10.10.0.110:8443`

3. Verify hz-cluster instances sync in Aether dashboard.

Aether default credentials: `admin` / `admin` (change on first login).

For Aether password, set in `env`:

```bash
export AETHER_ADMIN_PASSWORD="admin1234"
```

---

## 6. HAProxy load balancing

### 6.1 Create config file

Save as `envs/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):

```yaml
haproxy:
  cluster_remote: hz-cluster
  cluster_id: 52            # from OC cluster listing
  ovn_network: net-prod
  vip: 10.10.0.200
  haproxy_01_ip: 10.10.10.50
  haproxy_02_ip: 10.10.10.51
  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
  service_name: web-test
  image_version: "1.0.0"
  aether_url: https://10.10.0.120:8443
```

> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
> It's the numeric ID assigned when the cluster was created.

### 6.2 Build HAProxy image

Before deploying infrastructure, build and push the HAProxy image:

In Aether UI → **HAProxy → Images → Build**:
- Version: `1.0.0`
- Cluster: `hz-cluster`
- Click **Build** and wait 5-10 minutes

After build, push to hz-cluster:
- Click **Push** → select `hz-cluster`
- Click **Set Current**

Alternatively, use the script (requires the UI steps above for first build):

```bash
source env
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
```

### 6.3 Full deploy

```bash
source env
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
```

This:
1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
3. Creates the `web-test` service (HTTP, round-robin, health checks)

On subsequent runs (if backends already exist):

```bash
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
    --skip-backends --skip-image
```

### 6.4 Verify

```bash
curl http://10.10.0.200/
# Should show "Backend: nginx-lb-0x"

# Test distribution
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
```

---

## 7. AWX automation

### 7.1 Create config file

Save as `envs/hetzner/awx.yaml`:

```yaml
awx:
  vm_name: awx
  target_remote: hz-cluster
  target_node: hz-node-03    # specific node within cluster
  ip: 10.10.0.122/24
  gateway: 10.10.0.1
  dns: 10.10.0.1
  cpu: 4
  memory: 8GiB
  disk: 40GiB
  aether_url: https://10.10.0.120:8443
  aether_cluster_id: 52
```

> **target_node vs target_remote**: `target_remote` is the Incus remote
> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
> member to deploy on (e.g. `hz-node-03`). Both are required for
> cluster deployments.

### 7.2 Deploy

```bash
source env
bin/deploy-awx -c envs/hetzner/awx.yaml --deploy
```

This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
AWX will be at `http://10.10.0.122:30080`.

On partial failure (e.g. network issue mid-deploy), resume:

```bash
bin/deploy-awx -c envs/hetzner/awx.yaml --deploy --resume
# Skips VM creation and K3s install phases, jumps to AWX Operator phase
```

If you get `permission denied` on temp files from a previous aborted run:

```bash
incus exec hz-cluster:awx -- rm -f \
    /tmp/awx-operator-kustomization.yaml \
    /tmp/awx-kustomization.yaml \
    /tmp/awx-instance.yaml
```

### 7.3 Configure AWX

> **Wait for the awx-task pod before running --configure.** The AWX web
> deployment becomes ready first (~3 min), but `--configure` needs the
> task pod. Check it:
> ```bash
> incus exec hz-cluster:awx -- kubectl -n awx wait \
>     --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
>     --timeout=300s
> ```

```bash
source env
bin/deploy-awx -c envs/hetzner/awx.yaml --configure
```

This creates:
- Project: `incus-contrib` (linked to your git repo)
- Inventory: `incus-lab`
- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates

Get AWX admin password:

```bash
bin/deploy-awx -c /tmp/hz-awx.yaml --status
```

### 7.4 Join Aether

Generate an AWX API token for Aether:

```bash
source env
bin/deploy-awx -c envs/hetzner/awx.yaml --join-aether
# Prints the PAT token and template IDs
```

In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
- Name: `lab-awx`
- URL: `http://10.10.0.122:30080`
- Token: (paste token from `--join-aether` output)
- Verify SSL: unchecked (HTTP endpoint)

Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.

### 7.5 Bind AWX to cluster

**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
endpoint returns `400 "Invalid cluster ID"` due to a bug.

**Workaround** (direct DB update):

```bash
# First confirm the AWX endpoint ID (assigned by the database):
incus exec hz-cluster:aether -- bash -c \
    "su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""

# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
    UPDATE incus_ovn_clusters
    SET awx_endpoint_id=2,
        awx_post_deploy_template_id=9,
        awx_decommission_template_id=10,
        awx_job_timeout_seconds=600
    WHERE cluster_id=52;
\\\"\""
```

> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
> `postgres` system user. The endpoint ID is auto-assigned — always check
> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.

> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
> to see if the endpoint has been fixed. If it returns a valid schema,
> use the API instead of the DB workaround.

Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
the AWX endpoint, post-deploy, and decommission templates.

---

## 8. Observability stack

### 8.1 Deploy

```bash
source env
bin/deploy-observability \
    --remote hz-cluster \
    --monitoring-target hz-node-01 \
    --monitoring-ip 10.10.10.70 \
    --forward-vip 10.10.0.201 \
    --deploy
```

> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
> Operations Center (hz-oc) runs OC, not standard Incus — its
> `/1.0/instances` endpoint returns 404 and will fail the node
> discovery phase.

> **Grafana install may fail if net-prod containers lack internet.**
> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
> after container launch but can become unreliable. If phase 4 fails:
>
> ```bash
> # Download Grafana locally and push it in
> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
> incus exec hz-cluster:monitoring -- bash -c "
>     export DEBIAN_FRONTEND=noninteractive
>     dpkg -i /tmp/grafana.deb; apt-get -f install -y
>     mkdir -p /etc/grafana/provisioning/datasources
>     cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
> apiVersion: 1
> datasources:
>   - name: Prometheus
>     type: prometheus
>     uid: prometheus
>     access: proxy
>     url: http://localhost:9090
>     isDefault: true
>   - name: Loki
>     type: loki
>     uid: loki
>     access: proxy
>     url: http://localhost:3100
> EOF
>     systemctl daemon-reload
>     systemctl enable grafana-server
>     systemctl start grafana-server
> "
> ```
>
> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
> network forward, and dashboards).

> **Node-exporter containers are skipped by `--resume`.** If internet
> was unavailable during phase 5, deploy them manually:
>
> ```bash
> # Download node_exporter binary locally
> curl -sL -o /tmp/ne.tar.gz \
>     https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
>
> for i in 1 2 3; do
>   name="node-exp-0${i}"
>   ip="10.10.10.7${i}"
>   node="hz-node-0${i}"
>
>   incus launch images:alpine/3.20 hz-cluster:${name} \
>     -n net-prod --target ${node} \
>     -c limits.memory=128MiB -c security.privileged=true
>   incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
>   sleep 5
>
>   for dev in proc sys root; do
>     src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
>     incus config device add hz-cluster:${name} host-${dev} disk \
>       source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
>     incus config device add hz-cluster:${name} host-${dev} disk \
>       source=/ path=/host/root readonly=true 2>/dev/null || true
>   done
>
>   incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
>     hz-cluster:${name}/usr/bin/node_exporter
>
>   incus exec hz-cluster:${name} -- sh -c "
>     chmod +x /usr/bin/node_exporter
>     cat > /etc/init.d/node-exporter << 'INIT'
> #!/sbin/openrc-run
> description=\"Node Exporter\"
> command=/usr/bin/node_exporter
> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
> command_background=true
> pidfile=/run/node-exporter.pid
> INIT
>     chmod +x /etc/init.d/node-exporter
>     rc-update add node-exporter default
>     rc-service node-exporter start
>
>     cat > /etc/network/interfaces << NETCFG
> auto lo
> iface lo inet loopback
> auto eth0
> iface eth0 inet static
>     address ${ip}/24
>     gateway 10.10.10.1
> NETCFG
>     rc-service networking restart
>   "
>
>   incus config device set hz-cluster:${name} eth0 \
>     security.acls=monitoring-allow \
>     security.acls.default.ingress.action=allow \
>     security.acls.default.egress.action=allow
> done
> ```

On partial failure (phases 6-9 only), resume:

```bash
source env
bin/deploy-observability \
    --remote hz-cluster \
    --monitoring-target hz-node-01 \
    --monitoring-ip 10.10.10.70 \
    --forward-vip 10.10.0.201 \
    --resume --deploy
```

After deploying, fix the HAProxy prometheus exporter (default stats page
doesn't expose `/metrics`):

```bash
for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
  incus exec hz-cluster:${ct} -- bash -c "
    sed -i 's/    stats enable/    http-request use-service prometheus-exporter if { path \/metrics }\n    stats enable/' /etc/haproxy/haproxy.cfg
    systemctl reload haproxy
  "
done
```

### 8.2 Access

| Service    | URL                          | Credentials |
|------------|------------------------------|-------------|
| Grafana    | http://10.10.0.201:3000      | admin/admin |
| Prometheus | http://10.10.0.201:9090      | —           |

9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
storage, networking, capacity planning, and log explorer.

Verify all Prometheus targets are up:

```bash
curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
import sys,json
for t in json.load(sys.stdin)['data']['activeTargets']:
    print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
"
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
```

---

## 9. WireGuard access (macOS)

If you use the macOS WireGuard app from the App Store:

- The app uses Network Extension (visible as `utun5` in `ifconfig`)
- `wg show` always shows empty — this is normal with the App Store app
- The app allows only **one active tunnel at a time**

For a second simultaneous tunnel:

```bash
# Install wg-quick for macOS
brew install wireguard-tools

# Start the second tunnel alongside the App Store one
sudo wg-quick up ~/.config/wireguard/hetzner.conf
```

If the server added a new WireGuard peer (new workstation/client):

```bash
# On the Hetzner Proxmox host (via the existing tunnel or public IP)
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
ssh hetzner-lab "wg-quick save wg0"
```

---

## 10. Incus remote management

### Add remotes

```bash
# OC (Operations Center — not standard Incus)
# Note: Must be added manually — incus remote add loops prompting for token
# OC does not support the standard trust token flow.
# Edit ~/.config/incus/config.yml directly:
```

Add to `~/.config/incus/config.yml`:

```yaml
remotes:
  oc-server:
    addr: https://10.10.0.110:8443
    auth_type: tls
    protocol: incus
    public: false
  hz-cluster:
    addr: https://10.10.0.111:8443
    auth_type: tls
    protocol: incus
    public: false
```

Copy the OC certificate (accept it with curl first):

```bash
# Accept and save OC cert
curl -sk https://10.10.0.110:8443 > /dev/null
# The cert ends up in your browser cache — use openssl to fetch it
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
    | openssl x509 > ~/.config/incus/servercerts/oc-server.crt
```

> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
> fail. OC does not expose the standard `/1.0/instances` endpoint.

### After redeploy (remote cert changed)

```bash
# Remove old remote and re-add
incus remote remove hz-cluster
incus remote add hz-cluster https://10.10.0.111:8443 \
    --accept-certificate --auth-type tls
```

---

## 11. Quick reference

### Access URLs

| Service         | URL                              | Credentials         |
|-----------------|----------------------------------|---------------------|
| Operations Center | https://10.10.0.110:8443       | admin / (set on first login) |
| Aether          | https://10.10.0.120:8443         | admin / admin1234   |
| AWX             | http://10.10.0.122:30080         | admin / (from deploy-awx --status) |
| Grafana         | http://10.10.0.201:3000          | admin / admin       |
| Prometheus      | http://10.10.0.201:9090          | —                   |
| HAProxy VIP     | http://10.10.0.200/              | —                   |

### IP allocation

| Range                 | Purpose                      |
|-----------------------|------------------------------|
| 10.10.0.1             | Proxmox host (vmbr1 gateway) |
| 10.10.0.110-113       | IncusOS VMs (OC + nodes)     |
| 10.10.0.120           | Aether VM (macvlan)          |
| 10.10.0.122           | AWX VM (macvlan)             |
| 10.10.0.200+          | OVN external IPs (VIPs)      |
| 10.10.10.0/24         | OVN overlay (containers)     |
| 10.10.10.50-51        | HAProxy containers           |
| 10.10.10.60-62        | nginx test backends          |
| 10.10.10.70           | monitoring container         |
| 10.10.10.71-73        | node-exporter containers     |
| 10.10.99.0/24         | WireGuard tunnel             |
| 10.10.99.1            | WireGuard server             |
| 10.10.99.2            | Your workstation             |

### Cluster IDs

- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
  your OC UI if different)

### Useful incus commands

```bash
# List all instances in cluster
incus list hz-cluster:

# List cluster members
incus cluster list hz-cluster:

# List networks
incus network list hz-cluster:

# Exec into a container
incus exec hz-cluster:monitoring -- bash

# Start/stop a container
incus start hz-cluster:monitoring
incus stop hz-cluster:monitoring --force

# Check container network
incus exec hz-cluster:nginx-lb-01 -- ip a
```

### Teardown

```bash
# Force destroy all VMs (no nice cleanup needed for test lab)
source env
for vmid in 910 911 912 913; do
    bin/helpers/proxmox-api POST \
        "/nodes/aether-lab/qemu/${vmid}/status/stop" \
        '{"forceStop":1}' 2>/dev/null || true
    sleep 3
    bin/helpers/proxmox-api DELETE \
        "/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
done

# Remove stale remotes
incus remote remove hz-cluster 2>/dev/null || true
incus remote remove oc-server 2>/dev/null || true
```

---

## Known issues and workarounds

### OC remote add loops (incus remote add oc-server fails)

`incus remote add` enters a trust token prompt loop for OC because OC
does not implement the standard Incus trust token flow.

**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
and save the TLS certificate separately.

### IncusOS management NIC name is `mgmt`, not `eth0`

All IncusOS instances use `mgmt` as the management NIC name. Use
`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.

### Aether `PUT /api/clusters/{id}/awx-config` returns 400

Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).

### deploy-observability: don't use --oc-server with OC

The `--oc-server` flag tries to list instances via the standard Incus
API (`/1.0/instances`), which OC does not implement. Omit the flag.

### OVN SNAT internet access from net-prod containers is unreliable

Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
reach the internet. This works briefly after container launch (via Proxmox
NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
stale. Connections return `Connection refused` from Fastly CDN.

**Workaround**: Download packages locally and push them in with
`incus file push` instead of relying on apt/apk inside net-prod containers.

Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
Proxmox masquerade and are unaffected.

### deploy-awx temp file permission errors on resume

Temp files created as root inside the AWX VM persist between runs.
Clean up before resuming:

```bash
incus exec hz-cluster:awx -- rm -f \
    /tmp/awx-operator-kustomization.yaml \
    /tmp/awx-kustomization.yaml \
    /tmp/awx-instance.yaml
```

### aether_url in config files must use key-specific sed

Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
already fixed in all scripts in this repo.