26 KiB

Raw Blame History

Hetzner IncusOS Lab — End-to-End Guide

Full deployment guide for a production-quality IncusOS lab on a Hetzner dedicated server: 1 Operations Center + 3-node Incus cluster with OVN networking, HAProxy load balancing, AWX automation, and observability.

Estimated time: 3-4 hours for a clean run.

Overview

Hetzner dedicated (256 GB RAM, 32+ cores)
└─ Proxmox VE 9
   ├─ hz-oc       (VMID 910) — Operations Center 0.4.x
   ├─ hz-node-01  (VMID 911) — Incus cluster init + OVN control plane
   ├─ hz-node-02  (VMID 912) — Incus cluster member
   └─ hz-node-03  (VMID 913) — Incus cluster member
      └─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
         ├─ ffsdn-haproxy-52-01/02  — HAProxy HA pair
         ├─ nginx-lb-01/02/03       — Test backends
         ├─ monitoring              — Prometheus + Grafana
         ├─ node-exp-01/02/03       — Node exporters
         └─ awx                     — AWX Operator on K3s

Network topology:

vmbr0: Public interface (SSH + WireGuard only)
vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
OVN overlay: 10.10.10.0/24 (containers on net-prod)
OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)

Prerequisites

Hetzner dedicated server with Proxmox VE 9 installed and configured (see hetzner-setup.md + proxmox-setup)
WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
This repository cloned locally
env file populated (see below)
incusos/targets/hetzner/proxmox.yaml filled in
Incus client installed on workstation

env file

# incusos-contrib/env
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
export AETHER_ADMIN_PASSWORD="admin1234"

proxmox.yaml

cp incusos/targets/hetzner/proxmox.yaml.example \
   incusos/targets/hetzner/proxmox.yaml
# Fill in: host, node name, token_id, storage

1. Deploy IncusOS VMs

source env

incusos/incusos-proxmox \
    --proxmox incusos/targets/hetzner/proxmox.yaml \
    incusos/targets/hetzner/lab-production.yaml

What gets deployed:

VMID	Name	IP	RAM	Cores	Disk
910	hz-oc	10.10.0.110	8 GB	4	100G
911	hz-node-01	10.10.0.111	80 GB	16	150G
912	hz-node-02	10.10.0.112	80 GB	16	100G
913	hz-node-03	10.10.0.113	80 GB	16	100G

Boot takes 3-5 minutes per VM (sysext download from internet). Do not interrupt. Monitor progress:

# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
source env
incusos/helpers/proxmox-screenshot 910   # hz-oc
incusos/helpers/proxmox-screenshot 911   # hz-node-01

Wait until all VMs are reachable:

for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
    echo -n "$ip: "
    ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
done

2. Add Incus remotes

Add the cluster remote pointing at hz-node-01 (the init node):

incus remote add hz-cluster https://10.10.0.111:8443 \
    --accept-certificate --auth-type tls

Note: The incus remote add command prompts for a trust token. Obtain the token from OC after cluster formation (section 3), or if using a pre-formed cluster, generate one with: incus config trust add --name workstation

3. Cluster formation via Operations Center

Open the Operations Center UI:

https://10.10.0.110:8443
Username: admin
Password: admin   (first login, then set a new password)

3.1 Add Incus nodes

In OC: Servers → Add Server for each node:

Field	hz-node-01	hz-node-02	hz-node-03
Address	10.10.0.111:8443	10.10.0.112:8443	10.10.0.113:8443
Trust	via token	via token	via token

Each node's trust token is displayed on the IncusOS console on first boot.

3.2 Form the cluster

In OC: Clusters → Create Cluster

Name: hz-cluster
Init node: hz-node-01
Members: hz-node-02, hz-node-03
App seed: {} (empty JSON for manual post-formation setup)

OC forms the cluster automatically. Wait for all 3 nodes to show Ready.

Cluster ID is assigned by OC — note it (e.g. 52). You'll need it for HAProxy and AWX configuration.

3.3 Verify cluster

incus cluster list hz-cluster:
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03

4. OVN networking

4.1 Create the OVN uplink network

The UPLINK network uses the IncusOS management NIC (mgmt) — not eth0 or ens18. This is specific to IncusOS.

# Create UPLINK on each node
incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-01 \
    config parent=mgmt

incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-02 \
    config parent=mgmt

incus network create hz-cluster:UPLINK --type=physical \
    --target hz-node-03 \
    config parent=mgmt

# Finalize UPLINK on cluster level
incus network create hz-cluster:UPLINK \
    config ipv4.address=none \
    config ipv6.address=none

4.2 Deploy OVN central

OVN central runs as a privileged container on hz-node-01:

incus launch images:debian/12 hz-cluster:ovn-central \
    --vm=false \
    --target hz-node-01 \
    -c security.privileged=true \
    -d root,size=20GiB

# Wait for agent
sleep 15

# Install ovn-central
incus exec hz-cluster:ovn-central -- bash -c "
    apt-get update -qq
    apt-get install -y -qq ovn-central
    systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
    systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
"

4.3 Expose OVN NB/SB ports to cluster

Create proxy devices so cluster nodes can reach OVN:

# Get ovn-central IP
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')

incus config device add hz-cluster:ovn-central ovn-nb proxy \
    listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
    bind=host
incus config device add hz-cluster:ovn-central ovn-sb proxy \
    listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
    bind=host

4.4 Create OVN overlay network

# Use hz-node-01 IP as OVN NB/SB server
NODE01_IP="10.10.0.111"

incus network create hz-cluster:net-prod \
    --type=ovn \
    config network=UPLINK \
    config ipv4.address=10.10.10.1/24 \
    config ipv4.nat=true \
    config ipv6.address=none \
    config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
    config ovn.southbound_connection="tcp:${NODE01_IP}:6642"

Wait for the network to reach Created state:

incus network info hz-cluster:net-prod

4.5 Configure external IP range (OVN VIPs)

# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
# These are allocated on the Proxmox host's vmbr1 bridge
# No additional configuration needed — OVN will ARP for these IPs

Verify:

incus network list hz-cluster:
# net-prod should show type: ovn, state: Created

5. Deploy Aether

5.1 Deploy the Aether VM

Use the macvlan method to give Aether a direct vmbr1 IP:

incus launch images:ubuntu/22.04 hz-cluster:aether \
    --vm \
    --target hz-node-01 \
    -d root,size=40GiB \
    -c limits.cpu=4 \
    -c limits.memory=8GiB \
    -d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120

Alternative: use Aether golden image if already built on the cluster. Check incus image list hz-cluster: | grep aether.

Wait for the VM, then SSH in:

ssh ubuntu@10.10.0.120

Follow the Aether installation guide for the full Aether setup. Aether is accessible at https://10.10.0.120:8443.

5.2 Configure Aether cluster

In Aether UI (https://10.10.0.120:8443):

Settings → Clusters → Add Cluster
- Name: hz-cluster
- Endpoint: https://10.10.0.111:8443
- Authenticate (generates a trust token, add it to the cluster)
Settings → Operations Centers → Add OC
- URL: https://10.10.0.110:8443
Verify hz-cluster instances sync in Aether dashboard.

Aether default credentials: admin / admin (change on first login).

For Aether password, set in env:

export AETHER_ADMIN_PASSWORD="admin1234"

6. HAProxy load balancing

6.1 Create config file

Save as incusos/targets/hetzner/hz-haproxy.yaml (or /tmp/hz-deploy.yaml):

haproxy:
  cluster_remote: hz-cluster
  cluster_id: 52            # from OC cluster listing
  ovn_network: net-prod
  vip: 10.10.0.200
  haproxy_01_ip: 10.10.10.50
  haproxy_02_ip: 10.10.10.51
  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
  service_name: web-test
  image_version: "1.0.0"
  aether_url: https://10.10.0.120:8443

Cluster ID: Find it via OC API or in Aether UI under Clusters. It's the numeric ID assigned when the cluster was created.

6.2 Build HAProxy image

Before deploying infrastructure, build and push the HAProxy image:

In Aether UI → HAProxy → Images → Build:

Version: 1.0.0
Cluster: hz-cluster
Click Build and wait 5-10 minutes

After build, push to hz-cluster:

Click Push → select hz-cluster
Click Set Current

Alternatively, use the script (requires the UI steps above for first build):

source env
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image

6.3 Full deploy

source env
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy

This:

Launches 3 nginx backend containers (nginx-lb-01/02/03) on net-prod
Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
Creates the web-test service (HTTP, round-robin, health checks)

On subsequent runs (if backends already exist):

incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
    --skip-backends --skip-image

6.4 Verify

curl http://10.10.0.200/
# Should show "Backend: nginx-lb-0x"

# Test distribution
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done

7. AWX automation

7.1 Create config file

Save as incusos/targets/hetzner/awx.yaml:

awx:
  vm_name: awx
  target_remote: hz-cluster
  target_node: hz-node-03    # specific node within cluster
  ip: 10.10.0.122/24
  gateway: 10.10.0.1
  dns: 10.10.0.1
  cpu: 4
  memory: 8GiB
  disk: 40GiB
  aether_url: https://10.10.0.120:8443
  aether_cluster_id: 52

target_node vs target_remote: target_remote is the Incus remote name (e.g. hz-cluster), while target_node is the specific cluster member to deploy on (e.g. hz-node-03). Both are required for cluster deployments.

7.2 Deploy

source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy

This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes. AWX will be at http://10.10.0.122:30080.

On partial failure (e.g. network issue mid-deploy), resume:

incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy --resume
# Skips VM creation and K3s install phases, jumps to AWX Operator phase

If you get permission denied on temp files from a previous aborted run:

incus exec hz-cluster:awx -- rm -f \
    /tmp/awx-operator-kustomization.yaml \
    /tmp/awx-kustomization.yaml \
    /tmp/awx-instance.yaml

7.3 Configure AWX

Wait for the awx-task pod before running --configure. The AWX web deployment becomes ready first (~3 min), but --configure needs the task pod. Check it:
incus exec hz-cluster:awx -- kubectl -n awx wait \
    --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
    --timeout=300s

source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --configure

This creates:

Project: incus-contrib (linked to your git repo)
Inventory: incus-lab
Templates: post-deploy (ID 9), decommission (ID 10), and lifecycle templates

Get AWX admin password:

incusos/deploy-awx -c /tmp/hz-awx.yaml --status

7.4 Join Aether

Generate an AWX API token for Aether:

source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --join-aether
# Prints the PAT token and template IDs

In Aether UI → Ansible Automation → + Add Endpoint:

Name: lab-awx
URL: http://10.10.0.122:30080
Token: (paste token from --join-aether output)
Verify SSL: unchecked (HTTP endpoint)

Click Test Connection (should show Connected! AWX version: 24.x.x) then Save Endpoint.

7.5 Bind AWX to cluster

Note: In Aether ≤ v6.4.202, the PUT /api/clusters/{id}/awx-config endpoint returns 400 "Invalid cluster ID" due to a bug.

Workaround (direct DB update):

# First confirm the AWX endpoint ID (assigned by the database):
incus exec hz-cluster:aether -- bash -c \
    "su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""

# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
    UPDATE incus_ovn_clusters
    SET awx_endpoint_id=2,
        awx_post_deploy_template_id=9,
        awx_decommission_template_id=10,
        awx_job_timeout_seconds=600
    WHERE cluster_id=52;
\\\"\""

Note: The database is PostgreSQL (incusovnsdnc), accessed as the postgres system user. The endpoint ID is auto-assigned — always check it before applying the fix. Template IDs 9 and 10 are set by --configure.

Check future Aether versions: Run curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config to see if the endpoint has been fixed. If it returns a valid schema, use the API instead of the DB workaround.

Verify in Aether UI: Clusters → hz-cluster → Settings should show the AWX endpoint, post-deploy, and decommission templates.

8. Observability stack

8.1 Deploy

source env
incusos/deploy-observability \
    --remote hz-cluster \
    --monitoring-target hz-node-01 \
    --monitoring-ip 10.10.10.70 \
    --forward-vip 10.10.0.201 \
    --deploy

Note: Do NOT pass --oc-server for the Hetzner setup. The Operations Center (hz-oc) runs OC, not standard Incus — its /1.0/instances endpoint returns 404 and will fail the node discovery phase.

Grafana install may fail if net-prod containers lack internet. OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly after container launch but can become unreliable. If phase 4 fails:

# Download Grafana locally and push it in
curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
incus exec hz-cluster:monitoring -- bash -c "
    export DEBIAN_FRONTEND=noninteractive
    dpkg -i /tmp/grafana.deb; apt-get -f install -y
    mkdir -p /etc/grafana/provisioning/datasources
    cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://localhost:3100
EOF
    systemctl daemon-reload
    systemctl enable grafana-server
    systemctl start grafana-server
"

Then continue with --resume --deploy (skips phases 1-5, runs ACLs, network forward, and dashboards).

Node-exporter containers are skipped by --resume. If internet was unavailable during phase 5, deploy them manually:

# Download node_exporter binary locally
curl -sL -o /tmp/ne.tar.gz \
    https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter

for i in 1 2 3; do
  name="node-exp-0${i}"
  ip="10.10.10.7${i}"
  node="hz-node-0${i}"

  incus launch images:alpine/3.20 hz-cluster:${name} \
    -n net-prod --target ${node} \
    -c limits.memory=128MiB -c security.privileged=true
  incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
  sleep 5

  for dev in proc sys root; do
    src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
    incus config device add hz-cluster:${name} host-${dev} disk \
      source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
    incus config device add hz-cluster:${name} host-${dev} disk \
      source=/ path=/host/root readonly=true 2>/dev/null || true
  done

  incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
    hz-cluster:${name}/usr/bin/node_exporter

  incus exec hz-cluster:${name} -- sh -c "
    chmod +x /usr/bin/node_exporter
    cat > /etc/init.d/node-exporter << 'INIT'
#!/sbin/openrc-run
description=\"Node Exporter\"
command=/usr/bin/node_exporter
command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
command_background=true
pidfile=/run/node-exporter.pid
INIT
    chmod +x /etc/init.d/node-exporter
    rc-update add node-exporter default
    rc-service node-exporter start

    cat > /etc/network/interfaces << NETCFG
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
    address ${ip}/24
    gateway 10.10.10.1
NETCFG
    rc-service networking restart
  "

  incus config device set hz-cluster:${name} eth0 \
    security.acls=monitoring-allow \
    security.acls.default.ingress.action=allow \
    security.acls.default.egress.action=allow
done

On partial failure (phases 6-9 only), resume:

source env
incusos/deploy-observability \
    --remote hz-cluster \
    --monitoring-target hz-node-01 \
    --monitoring-ip 10.10.10.70 \
    --forward-vip 10.10.0.201 \
    --resume --deploy

After deploying, fix the HAProxy prometheus exporter (default stats page doesn't expose /metrics):

for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
  incus exec hz-cluster:${ct} -- bash -c "
    sed -i 's/    stats enable/    http-request use-service prometheus-exporter if { path \/metrics }\n    stats enable/' /etc/haproxy/haproxy.cfg
    systemctl reload haproxy
  "
done

8.2 Access

Service	URL	Credentials
Grafana	http://10.10.0.201:3000	admin/admin
Prometheus	http://10.10.0.201:9090	—

9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic, storage, networking, capacity planning, and log explorer.

Verify all Prometheus targets are up:

curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
import sys,json
for t in json.load(sys.stdin)['data']['activeTargets']:
    print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
"
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'

9. WireGuard access (macOS)

If you use the macOS WireGuard app from the App Store:

The app uses Network Extension (visible as utun5 in ifconfig)
wg show always shows empty — this is normal with the App Store app
The app allows only one active tunnel at a time

For a second simultaneous tunnel:

# Install wg-quick for macOS
brew install wireguard-tools

# Start the second tunnel alongside the App Store one
sudo wg-quick up ~/.config/wireguard/hetzner.conf

If the server added a new WireGuard peer (new workstation/client):

# On the Hetzner Proxmox host (via the existing tunnel or public IP)
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
ssh hetzner-lab "wg-quick save wg0"

10. Incus remote management

Add remotes

# OC (Operations Center — not standard Incus)
# Note: Must be added manually — incus remote add loops prompting for token
# OC does not support the standard trust token flow.
# Edit ~/.config/incus/config.yml directly:

Add to ~/.config/incus/config.yml:

remotes:
  oc-server:
    addr: https://10.10.0.110:8443
    auth_type: tls
    protocol: incus
    public: false
  hz-cluster:
    addr: https://10.10.0.111:8443
    auth_type: tls
    protocol: incus
    public: false

Copy the OC certificate (accept it with curl first):

# Accept and save OC cert
curl -sk https://10.10.0.110:8443 > /dev/null
# The cert ends up in your browser cache — use openssl to fetch it
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
    | openssl x509 > ~/.config/incus/servercerts/oc-server.crt

Warning: oc-server is OC, not Incus. incus list oc-server: will fail. OC does not expose the standard /1.0/instances endpoint.

After redeploy (remote cert changed)

# Remove old remote and re-add
incus remote remove hz-cluster
incus remote add hz-cluster https://10.10.0.111:8443 \
    --accept-certificate --auth-type tls

11. Quick reference

Access URLs

Service	URL	Credentials
Operations Center	https://10.10.0.110:8443	admin / (set on first login)
Aether	https://10.10.0.120:8443	admin / admin1234
AWX	http://10.10.0.122:30080	admin / (from deploy-awx --status)
Grafana	http://10.10.0.201:3000	admin / admin
Prometheus	http://10.10.0.201:9090	—
HAProxy VIP	http://10.10.0.200/	—

IP allocation

Range	Purpose
10.10.0.1	Proxmox host (vmbr1 gateway)
10.10.0.110-113	IncusOS VMs (OC + nodes)
10.10.0.120	Aether VM (macvlan)
10.10.0.122	AWX VM (macvlan)
10.10.0.200+	OVN external IPs (VIPs)
10.10.10.0/24	OVN overlay (containers)
10.10.10.50-51	HAProxy containers
10.10.10.60-62	nginx test backends
10.10.10.70	monitoring container
10.10.10.71-73	node-exporter containers
10.10.99.0/24	WireGuard tunnel
10.10.99.1	WireGuard server
10.10.99.2	Your workstation

Cluster IDs

hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check your OC UI if different)

Useful incus commands

# List all instances in cluster
incus list hz-cluster:

# List cluster members
incus cluster list hz-cluster:

# List networks
incus network list hz-cluster:

# Exec into a container
incus exec hz-cluster:monitoring -- bash

# Start/stop a container
incus start hz-cluster:monitoring
incus stop hz-cluster:monitoring --force

# Check container network
incus exec hz-cluster:nginx-lb-01 -- ip a

Teardown

# Force destroy all VMs (no nice cleanup needed for test lab)
source env
for vmid in 910 911 912 913; do
    incusos/helpers/proxmox-api POST \
        "/nodes/aether-lab/qemu/${vmid}/status/stop" \
        '{"forceStop":1}' 2>/dev/null || true
    sleep 3
    incusos/helpers/proxmox-api DELETE \
        "/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
done

# Remove stale remotes
incus remote remove hz-cluster 2>/dev/null || true
incus remote remove oc-server 2>/dev/null || true

Known issues and workarounds

OC remote add loops (incus remote add oc-server fails)

incus remote add enters a trust token prompt loop for OC because OC does not implement the standard Incus trust token flow.

Workaround: Add the remote manually to ~/.config/incus/config.yml and save the TLS certificate separately.

IncusOS management NIC name is `mgmt`, not `eth0`

All IncusOS instances use mgmt as the management NIC name. Use parent=mgmt in UPLINK network config, not parent=eth0 or parent=ens18.

Aether `PUT /api/clusters/{id}/awx-config` returns 400

Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).

deploy-observability: don't use --oc-server with OC

The --oc-server flag tries to list instances via the standard Incus API (/1.0/instances), which OC does not implement. Omit the flag.

OVN SNAT internet access from net-prod containers is unreliable

Containers on net-prod (10.10.10.0/24) use SNAT via 10.10.0.200 to reach the internet. This works briefly after container launch (via Proxmox NAT masquerade) but can become unreliable as ARP for the VIP IP becomes stale. Connections return Connection refused from Fastly CDN.

Workaround: Download packages locally and push them in with incus file push instead of relying on apt/apk inside net-prod containers.

Containers on macvlan mgmt (like AWX, Aether) have reliable internet via Proxmox masquerade and are unaffected.

deploy-awx temp file permission errors on resume

Temp files created as root inside the AWX VM persist between runs. Clean up before resuming:

incus exec hz-cluster:awx -- rm -f \
    /tmp/awx-operator-kustomization.yaml \
    /tmp/awx-kustomization.yaml \
    /tmp/awx-instance.yaml

aether_url in config files must use key-specific sed

Config parsing uses sed 's/^ *aether_url: *//' (key-specific match). A greedy sed 's/.*: *//' would strip the :8443 port. This is already fixed in all scripts in this repo.

26 KiB Raw Blame History