908 lines
26 KiB
Markdown
908 lines
26 KiB
Markdown
# Hetzner IncusOS Lab — End-to-End Guide
|
|
|
|
Full deployment guide for a production-quality IncusOS lab on a Hetzner
|
|
dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
|
|
networking, HAProxy load balancing, AWX automation, and observability.
|
|
|
|
**Estimated time**: 3-4 hours for a clean run.
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
```
|
|
Hetzner dedicated (256 GB RAM, 32+ cores)
|
|
└─ Proxmox VE 9
|
|
├─ hz-oc (VMID 910) — Operations Center 0.4.x
|
|
├─ hz-node-01 (VMID 911) — Incus cluster init + OVN control plane
|
|
├─ hz-node-02 (VMID 912) — Incus cluster member
|
|
└─ hz-node-03 (VMID 913) — Incus cluster member
|
|
└─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
|
|
├─ ffsdn-haproxy-52-01/02 — HAProxy HA pair
|
|
├─ nginx-lb-01/02/03 — Test backends
|
|
├─ monitoring — Prometheus + Grafana
|
|
├─ node-exp-01/02/03 — Node exporters
|
|
└─ awx — AWX Operator on K3s
|
|
```
|
|
|
|
**Network topology:**
|
|
- vmbr0: Public interface (SSH + WireGuard only)
|
|
- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
|
|
- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
|
|
- OVN overlay: 10.10.10.0/24 (containers on net-prod)
|
|
- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
- Hetzner dedicated server with Proxmox VE 9 installed and configured
|
|
(see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
|
|
- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
|
|
- This repository cloned locally
|
|
- `env` file populated (see below)
|
|
- `incusos/targets/hetzner/proxmox.yaml` filled in
|
|
- Incus client installed on workstation
|
|
|
|
### env file
|
|
|
|
```bash
|
|
# incusos-contrib/env
|
|
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
|
|
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
|
|
export AETHER_ADMIN_PASSWORD="admin1234"
|
|
```
|
|
|
|
### proxmox.yaml
|
|
|
|
```bash
|
|
cp incusos/targets/hetzner/proxmox.yaml.example \
|
|
incusos/targets/hetzner/proxmox.yaml
|
|
# Fill in: host, node name, token_id, storage
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Deploy IncusOS VMs
|
|
|
|
```bash
|
|
source env
|
|
|
|
incusos/incusos-proxmox \
|
|
--proxmox incusos/targets/hetzner/proxmox.yaml \
|
|
incusos/targets/hetzner/lab-production.yaml
|
|
```
|
|
|
|
**What gets deployed:**
|
|
|
|
| VMID | Name | IP | RAM | Cores | Disk |
|
|
|------|-----------|---------------|-------|-------|------|
|
|
| 910 | hz-oc | 10.10.0.110 | 8 GB | 4 | 100G |
|
|
| 911 | hz-node-01| 10.10.0.111 | 80 GB | 16 | 150G |
|
|
| 912 | hz-node-02| 10.10.0.112 | 80 GB | 16 | 100G |
|
|
| 913 | hz-node-03| 10.10.0.113 | 80 GB | 16 | 100G |
|
|
|
|
**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
|
|
interrupt. Monitor progress:
|
|
|
|
```bash
|
|
# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
|
|
source env
|
|
incusos/helpers/proxmox-screenshot 910 # hz-oc
|
|
incusos/helpers/proxmox-screenshot 911 # hz-node-01
|
|
```
|
|
|
|
Wait until all VMs are reachable:
|
|
|
|
```bash
|
|
for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
|
|
echo -n "$ip: "
|
|
ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
|
|
done
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Add Incus remotes
|
|
|
|
Add the cluster remote pointing at hz-node-01 (the init node):
|
|
|
|
```bash
|
|
incus remote add hz-cluster https://10.10.0.111:8443 \
|
|
--accept-certificate --auth-type tls
|
|
```
|
|
|
|
> **Note**: The `incus remote add` command prompts for a trust token.
|
|
> Obtain the token from OC after cluster formation (section 3), or
|
|
> if using a pre-formed cluster, generate one with:
|
|
> `incus config trust add --name workstation`
|
|
|
|
---
|
|
|
|
## 3. Cluster formation via Operations Center
|
|
|
|
Open the Operations Center UI:
|
|
|
|
```
|
|
https://10.10.0.110:8443
|
|
Username: admin
|
|
Password: admin (first login, then set a new password)
|
|
```
|
|
|
|
### 3.1 Add Incus nodes
|
|
|
|
In OC: **Servers → Add Server** for each node:
|
|
|
|
| Field | hz-node-01 | hz-node-02 | hz-node-03 |
|
|
|---------|------------------|------------------|------------------|
|
|
| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
|
|
| Trust | via token | via token | via token |
|
|
|
|
Each node's trust token is displayed on the IncusOS console on first boot.
|
|
|
|
### 3.2 Form the cluster
|
|
|
|
In OC: **Clusters → Create Cluster**
|
|
|
|
- Name: `hz-cluster`
|
|
- Init node: `hz-node-01`
|
|
- Members: `hz-node-02`, `hz-node-03`
|
|
- App seed: `{}` (empty JSON for manual post-formation setup)
|
|
|
|
OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
|
|
|
|
Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
|
|
HAProxy and AWX configuration.
|
|
|
|
### 3.3 Verify cluster
|
|
|
|
```bash
|
|
incus cluster list hz-cluster:
|
|
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
|
|
```
|
|
|
|
---
|
|
|
|
## 4. OVN networking
|
|
|
|
### 4.1 Create the OVN uplink network
|
|
|
|
The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
|
|
`eth0` or `ens18`. This is specific to IncusOS.
|
|
|
|
```bash
|
|
# Create UPLINK on each node
|
|
incus network create hz-cluster:UPLINK --type=physical \
|
|
--target hz-node-01 \
|
|
config parent=mgmt
|
|
|
|
incus network create hz-cluster:UPLINK --type=physical \
|
|
--target hz-node-02 \
|
|
config parent=mgmt
|
|
|
|
incus network create hz-cluster:UPLINK --type=physical \
|
|
--target hz-node-03 \
|
|
config parent=mgmt
|
|
|
|
# Finalize UPLINK on cluster level
|
|
incus network create hz-cluster:UPLINK \
|
|
config ipv4.address=none \
|
|
config ipv6.address=none
|
|
```
|
|
|
|
### 4.2 Deploy OVN central
|
|
|
|
OVN central runs as a privileged container on hz-node-01:
|
|
|
|
```bash
|
|
incus launch images:debian/12 hz-cluster:ovn-central \
|
|
--vm=false \
|
|
--target hz-node-01 \
|
|
-c security.privileged=true \
|
|
-d root,size=20GiB
|
|
|
|
# Wait for agent
|
|
sleep 15
|
|
|
|
# Install ovn-central
|
|
incus exec hz-cluster:ovn-central -- bash -c "
|
|
apt-get update -qq
|
|
apt-get install -y -qq ovn-central
|
|
systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
|
systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
|
"
|
|
```
|
|
|
|
### 4.3 Expose OVN NB/SB ports to cluster
|
|
|
|
Create proxy devices so cluster nodes can reach OVN:
|
|
|
|
```bash
|
|
# Get ovn-central IP
|
|
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
|
|
|
|
incus config device add hz-cluster:ovn-central ovn-nb proxy \
|
|
listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
|
|
bind=host
|
|
incus config device add hz-cluster:ovn-central ovn-sb proxy \
|
|
listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
|
|
bind=host
|
|
```
|
|
|
|
### 4.4 Create OVN overlay network
|
|
|
|
```bash
|
|
# Use hz-node-01 IP as OVN NB/SB server
|
|
NODE01_IP="10.10.0.111"
|
|
|
|
incus network create hz-cluster:net-prod \
|
|
--type=ovn \
|
|
config network=UPLINK \
|
|
config ipv4.address=10.10.10.1/24 \
|
|
config ipv4.nat=true \
|
|
config ipv6.address=none \
|
|
config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
|
|
config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
|
|
```
|
|
|
|
Wait for the network to reach **Created** state:
|
|
|
|
```bash
|
|
incus network info hz-cluster:net-prod
|
|
```
|
|
|
|
### 4.5 Configure external IP range (OVN VIPs)
|
|
|
|
```bash
|
|
# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
|
|
# These are allocated on the Proxmox host's vmbr1 bridge
|
|
# No additional configuration needed — OVN will ARP for these IPs
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
incus network list hz-cluster:
|
|
# net-prod should show type: ovn, state: Created
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Deploy Aether
|
|
|
|
### 5.1 Deploy the Aether VM
|
|
|
|
Use the macvlan method to give Aether a direct vmbr1 IP:
|
|
|
|
```bash
|
|
incus launch images:ubuntu/22.04 hz-cluster:aether \
|
|
--vm \
|
|
--target hz-node-01 \
|
|
-d root,size=40GiB \
|
|
-c limits.cpu=4 \
|
|
-c limits.memory=8GiB \
|
|
-d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
|
|
```
|
|
|
|
> **Alternative**: use Aether golden image if already built on the cluster.
|
|
> Check `incus image list hz-cluster: | grep aether`.
|
|
|
|
Wait for the VM, then SSH in:
|
|
|
|
```bash
|
|
ssh ubuntu@10.10.0.120
|
|
```
|
|
|
|
Follow the [Aether installation guide](../notes/aether-guide.md) for the
|
|
full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
|
|
|
|
### 5.2 Configure Aether cluster
|
|
|
|
In Aether UI (`https://10.10.0.120:8443`):
|
|
|
|
1. **Settings → Clusters → Add Cluster**
|
|
- Name: hz-cluster
|
|
- Endpoint: `https://10.10.0.111:8443`
|
|
- Authenticate (generates a trust token, add it to the cluster)
|
|
|
|
2. **Settings → Operations Centers → Add OC**
|
|
- URL: `https://10.10.0.110:8443`
|
|
|
|
3. Verify hz-cluster instances sync in Aether dashboard.
|
|
|
|
Aether default credentials: `admin` / `admin` (change on first login).
|
|
|
|
For Aether password, set in `env`:
|
|
|
|
```bash
|
|
export AETHER_ADMIN_PASSWORD="admin1234"
|
|
```
|
|
|
|
---
|
|
|
|
## 6. HAProxy load balancing
|
|
|
|
### 6.1 Create config file
|
|
|
|
Save as `incusos/targets/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
|
|
|
|
```yaml
|
|
haproxy:
|
|
cluster_remote: hz-cluster
|
|
cluster_id: 52 # from OC cluster listing
|
|
ovn_network: net-prod
|
|
vip: 10.10.0.200
|
|
haproxy_01_ip: 10.10.10.50
|
|
haproxy_02_ip: 10.10.10.51
|
|
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
|
service_name: web-test
|
|
image_version: "1.0.0"
|
|
aether_url: https://10.10.0.120:8443
|
|
```
|
|
|
|
> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
|
|
> It's the numeric ID assigned when the cluster was created.
|
|
|
|
### 6.2 Build HAProxy image
|
|
|
|
Before deploying infrastructure, build and push the HAProxy image:
|
|
|
|
In Aether UI → **HAProxy → Images → Build**:
|
|
- Version: `1.0.0`
|
|
- Cluster: `hz-cluster`
|
|
- Click **Build** and wait 5-10 minutes
|
|
|
|
After build, push to hz-cluster:
|
|
- Click **Push** → select `hz-cluster`
|
|
- Click **Set Current**
|
|
|
|
Alternatively, use the script (requires the UI steps above for first build):
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
|
|
```
|
|
|
|
### 6.3 Full deploy
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
|
|
```
|
|
|
|
This:
|
|
1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
|
|
2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
|
|
3. Creates the `web-test` service (HTTP, round-robin, health checks)
|
|
|
|
On subsequent runs (if backends already exist):
|
|
|
|
```bash
|
|
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
|
|
--skip-backends --skip-image
|
|
```
|
|
|
|
### 6.4 Verify
|
|
|
|
```bash
|
|
curl http://10.10.0.200/
|
|
# Should show "Backend: nginx-lb-0x"
|
|
|
|
# Test distribution
|
|
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
|
|
```
|
|
|
|
---
|
|
|
|
## 7. AWX automation
|
|
|
|
### 7.1 Create config file
|
|
|
|
Save as `incusos/targets/hetzner/awx.yaml`:
|
|
|
|
```yaml
|
|
awx:
|
|
vm_name: awx
|
|
target_remote: hz-cluster
|
|
target_node: hz-node-03 # specific node within cluster
|
|
ip: 10.10.0.122/24
|
|
gateway: 10.10.0.1
|
|
dns: 10.10.0.1
|
|
cpu: 4
|
|
memory: 8GiB
|
|
disk: 40GiB
|
|
aether_url: https://10.10.0.120:8443
|
|
aether_cluster_id: 52
|
|
```
|
|
|
|
> **target_node vs target_remote**: `target_remote` is the Incus remote
|
|
> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
|
|
> member to deploy on (e.g. `hz-node-03`). Both are required for
|
|
> cluster deployments.
|
|
|
|
### 7.2 Deploy
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy
|
|
```
|
|
|
|
This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
|
|
AWX will be at `http://10.10.0.122:30080`.
|
|
|
|
On partial failure (e.g. network issue mid-deploy), resume:
|
|
|
|
```bash
|
|
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy --resume
|
|
# Skips VM creation and K3s install phases, jumps to AWX Operator phase
|
|
```
|
|
|
|
If you get `permission denied` on temp files from a previous aborted run:
|
|
|
|
```bash
|
|
incus exec hz-cluster:awx -- rm -f \
|
|
/tmp/awx-operator-kustomization.yaml \
|
|
/tmp/awx-kustomization.yaml \
|
|
/tmp/awx-instance.yaml
|
|
```
|
|
|
|
### 7.3 Configure AWX
|
|
|
|
> **Wait for the awx-task pod before running --configure.** The AWX web
|
|
> deployment becomes ready first (~3 min), but `--configure` needs the
|
|
> task pod. Check it:
|
|
> ```bash
|
|
> incus exec hz-cluster:awx -- kubectl -n awx wait \
|
|
> --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
|
|
> --timeout=300s
|
|
> ```
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --configure
|
|
```
|
|
|
|
This creates:
|
|
- Project: `incus-contrib` (linked to your git repo)
|
|
- Inventory: `incus-lab`
|
|
- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
|
|
|
|
Get AWX admin password:
|
|
|
|
```bash
|
|
incusos/deploy-awx -c /tmp/hz-awx.yaml --status
|
|
```
|
|
|
|
### 7.4 Join Aether
|
|
|
|
Generate an AWX API token for Aether:
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --join-aether
|
|
# Prints the PAT token and template IDs
|
|
```
|
|
|
|
In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
|
|
- Name: `lab-awx`
|
|
- URL: `http://10.10.0.122:30080`
|
|
- Token: (paste token from `--join-aether` output)
|
|
- Verify SSL: unchecked (HTTP endpoint)
|
|
|
|
Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
|
|
|
|
### 7.5 Bind AWX to cluster
|
|
|
|
**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
|
|
endpoint returns `400 "Invalid cluster ID"` due to a bug.
|
|
|
|
**Workaround** (direct DB update):
|
|
|
|
```bash
|
|
# First confirm the AWX endpoint ID (assigned by the database):
|
|
incus exec hz-cluster:aether -- bash -c \
|
|
"su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
|
|
|
|
# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
|
|
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
|
|
UPDATE incus_ovn_clusters
|
|
SET awx_endpoint_id=2,
|
|
awx_post_deploy_template_id=9,
|
|
awx_decommission_template_id=10,
|
|
awx_job_timeout_seconds=600
|
|
WHERE cluster_id=52;
|
|
\\\"\""
|
|
```
|
|
|
|
> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
|
|
> `postgres` system user. The endpoint ID is auto-assigned — always check
|
|
> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
|
|
|
|
> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
|
|
> to see if the endpoint has been fixed. If it returns a valid schema,
|
|
> use the API instead of the DB workaround.
|
|
|
|
Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
|
|
the AWX endpoint, post-deploy, and decommission templates.
|
|
|
|
---
|
|
|
|
## 8. Observability stack
|
|
|
|
### 8.1 Deploy
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-observability \
|
|
--remote hz-cluster \
|
|
--monitoring-target hz-node-01 \
|
|
--monitoring-ip 10.10.10.70 \
|
|
--forward-vip 10.10.0.201 \
|
|
--deploy
|
|
```
|
|
|
|
> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
|
|
> Operations Center (hz-oc) runs OC, not standard Incus — its
|
|
> `/1.0/instances` endpoint returns 404 and will fail the node
|
|
> discovery phase.
|
|
|
|
> **Grafana install may fail if net-prod containers lack internet.**
|
|
> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
|
|
> after container launch but can become unreliable. If phase 4 fails:
|
|
>
|
|
> ```bash
|
|
> # Download Grafana locally and push it in
|
|
> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
|
|
> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
|
|
> incus exec hz-cluster:monitoring -- bash -c "
|
|
> export DEBIAN_FRONTEND=noninteractive
|
|
> dpkg -i /tmp/grafana.deb; apt-get -f install -y
|
|
> mkdir -p /etc/grafana/provisioning/datasources
|
|
> cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
|
|
> apiVersion: 1
|
|
> datasources:
|
|
> - name: Prometheus
|
|
> type: prometheus
|
|
> uid: prometheus
|
|
> access: proxy
|
|
> url: http://localhost:9090
|
|
> isDefault: true
|
|
> - name: Loki
|
|
> type: loki
|
|
> uid: loki
|
|
> access: proxy
|
|
> url: http://localhost:3100
|
|
> EOF
|
|
> systemctl daemon-reload
|
|
> systemctl enable grafana-server
|
|
> systemctl start grafana-server
|
|
> "
|
|
> ```
|
|
>
|
|
> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
|
|
> network forward, and dashboards).
|
|
|
|
> **Node-exporter containers are skipped by `--resume`.** If internet
|
|
> was unavailable during phase 5, deploy them manually:
|
|
>
|
|
> ```bash
|
|
> # Download node_exporter binary locally
|
|
> curl -sL -o /tmp/ne.tar.gz \
|
|
> https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
|
|
> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
|
|
>
|
|
> for i in 1 2 3; do
|
|
> name="node-exp-0${i}"
|
|
> ip="10.10.10.7${i}"
|
|
> node="hz-node-0${i}"
|
|
>
|
|
> incus launch images:alpine/3.20 hz-cluster:${name} \
|
|
> -n net-prod --target ${node} \
|
|
> -c limits.memory=128MiB -c security.privileged=true
|
|
> incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
|
|
> sleep 5
|
|
>
|
|
> for dev in proc sys root; do
|
|
> src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
|
|
> incus config device add hz-cluster:${name} host-${dev} disk \
|
|
> source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
|
|
> incus config device add hz-cluster:${name} host-${dev} disk \
|
|
> source=/ path=/host/root readonly=true 2>/dev/null || true
|
|
> done
|
|
>
|
|
> incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
|
|
> hz-cluster:${name}/usr/bin/node_exporter
|
|
>
|
|
> incus exec hz-cluster:${name} -- sh -c "
|
|
> chmod +x /usr/bin/node_exporter
|
|
> cat > /etc/init.d/node-exporter << 'INIT'
|
|
> #!/sbin/openrc-run
|
|
> description=\"Node Exporter\"
|
|
> command=/usr/bin/node_exporter
|
|
> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
|
|
> command_background=true
|
|
> pidfile=/run/node-exporter.pid
|
|
> INIT
|
|
> chmod +x /etc/init.d/node-exporter
|
|
> rc-update add node-exporter default
|
|
> rc-service node-exporter start
|
|
>
|
|
> cat > /etc/network/interfaces << NETCFG
|
|
> auto lo
|
|
> iface lo inet loopback
|
|
> auto eth0
|
|
> iface eth0 inet static
|
|
> address ${ip}/24
|
|
> gateway 10.10.10.1
|
|
> NETCFG
|
|
> rc-service networking restart
|
|
> "
|
|
>
|
|
> incus config device set hz-cluster:${name} eth0 \
|
|
> security.acls=monitoring-allow \
|
|
> security.acls.default.ingress.action=allow \
|
|
> security.acls.default.egress.action=allow
|
|
> done
|
|
> ```
|
|
|
|
On partial failure (phases 6-9 only), resume:
|
|
|
|
```bash
|
|
source env
|
|
incusos/deploy-observability \
|
|
--remote hz-cluster \
|
|
--monitoring-target hz-node-01 \
|
|
--monitoring-ip 10.10.10.70 \
|
|
--forward-vip 10.10.0.201 \
|
|
--resume --deploy
|
|
```
|
|
|
|
After deploying, fix the HAProxy prometheus exporter (default stats page
|
|
doesn't expose `/metrics`):
|
|
|
|
```bash
|
|
for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
|
|
incus exec hz-cluster:${ct} -- bash -c "
|
|
sed -i 's/ stats enable/ http-request use-service prometheus-exporter if { path \/metrics }\n stats enable/' /etc/haproxy/haproxy.cfg
|
|
systemctl reload haproxy
|
|
"
|
|
done
|
|
```
|
|
|
|
### 8.2 Access
|
|
|
|
| Service | URL | Credentials |
|
|
|------------|------------------------------|-------------|
|
|
| Grafana | http://10.10.0.201:3000 | admin/admin |
|
|
| Prometheus | http://10.10.0.201:9090 | — |
|
|
|
|
9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
|
|
storage, networking, capacity planning, and log explorer.
|
|
|
|
Verify all Prometheus targets are up:
|
|
|
|
```bash
|
|
curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
|
|
import sys,json
|
|
for t in json.load(sys.stdin)['data']['activeTargets']:
|
|
print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
|
|
"
|
|
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
|
|
```
|
|
|
|
---
|
|
|
|
## 9. WireGuard access (macOS)
|
|
|
|
If you use the macOS WireGuard app from the App Store:
|
|
|
|
- The app uses Network Extension (visible as `utun5` in `ifconfig`)
|
|
- `wg show` always shows empty — this is normal with the App Store app
|
|
- The app allows only **one active tunnel at a time**
|
|
|
|
For a second simultaneous tunnel:
|
|
|
|
```bash
|
|
# Install wg-quick for macOS
|
|
brew install wireguard-tools
|
|
|
|
# Start the second tunnel alongside the App Store one
|
|
sudo wg-quick up ~/.config/wireguard/hetzner.conf
|
|
```
|
|
|
|
If the server added a new WireGuard peer (new workstation/client):
|
|
|
|
```bash
|
|
# On the Hetzner Proxmox host (via the existing tunnel or public IP)
|
|
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
|
|
ssh hetzner-lab "wg-quick save wg0"
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Incus remote management
|
|
|
|
### Add remotes
|
|
|
|
```bash
|
|
# OC (Operations Center — not standard Incus)
|
|
# Note: Must be added manually — incus remote add loops prompting for token
|
|
# OC does not support the standard trust token flow.
|
|
# Edit ~/.config/incus/config.yml directly:
|
|
```
|
|
|
|
Add to `~/.config/incus/config.yml`:
|
|
|
|
```yaml
|
|
remotes:
|
|
oc-server:
|
|
addr: https://10.10.0.110:8443
|
|
auth_type: tls
|
|
protocol: incus
|
|
public: false
|
|
hz-cluster:
|
|
addr: https://10.10.0.111:8443
|
|
auth_type: tls
|
|
protocol: incus
|
|
public: false
|
|
```
|
|
|
|
Copy the OC certificate (accept it with curl first):
|
|
|
|
```bash
|
|
# Accept and save OC cert
|
|
curl -sk https://10.10.0.110:8443 > /dev/null
|
|
# The cert ends up in your browser cache — use openssl to fetch it
|
|
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
|
|
| openssl x509 > ~/.config/incus/servercerts/oc-server.crt
|
|
```
|
|
|
|
> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
|
|
> fail. OC does not expose the standard `/1.0/instances` endpoint.
|
|
|
|
### After redeploy (remote cert changed)
|
|
|
|
```bash
|
|
# Remove old remote and re-add
|
|
incus remote remove hz-cluster
|
|
incus remote add hz-cluster https://10.10.0.111:8443 \
|
|
--accept-certificate --auth-type tls
|
|
```
|
|
|
|
---
|
|
|
|
## 11. Quick reference
|
|
|
|
### Access URLs
|
|
|
|
| Service | URL | Credentials |
|
|
|-----------------|----------------------------------|---------------------|
|
|
| Operations Center | https://10.10.0.110:8443 | admin / (set on first login) |
|
|
| Aether | https://10.10.0.120:8443 | admin / admin1234 |
|
|
| AWX | http://10.10.0.122:30080 | admin / (from deploy-awx --status) |
|
|
| Grafana | http://10.10.0.201:3000 | admin / admin |
|
|
| Prometheus | http://10.10.0.201:9090 | — |
|
|
| HAProxy VIP | http://10.10.0.200/ | — |
|
|
|
|
### IP allocation
|
|
|
|
| Range | Purpose |
|
|
|-----------------------|------------------------------|
|
|
| 10.10.0.1 | Proxmox host (vmbr1 gateway) |
|
|
| 10.10.0.110-113 | IncusOS VMs (OC + nodes) |
|
|
| 10.10.0.120 | Aether VM (macvlan) |
|
|
| 10.10.0.122 | AWX VM (macvlan) |
|
|
| 10.10.0.200+ | OVN external IPs (VIPs) |
|
|
| 10.10.10.0/24 | OVN overlay (containers) |
|
|
| 10.10.10.50-51 | HAProxy containers |
|
|
| 10.10.10.60-62 | nginx test backends |
|
|
| 10.10.10.70 | monitoring container |
|
|
| 10.10.10.71-73 | node-exporter containers |
|
|
| 10.10.99.0/24 | WireGuard tunnel |
|
|
| 10.10.99.1 | WireGuard server |
|
|
| 10.10.99.2 | Your workstation |
|
|
|
|
### Cluster IDs
|
|
|
|
- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
|
|
your OC UI if different)
|
|
|
|
### Useful incus commands
|
|
|
|
```bash
|
|
# List all instances in cluster
|
|
incus list hz-cluster:
|
|
|
|
# List cluster members
|
|
incus cluster list hz-cluster:
|
|
|
|
# List networks
|
|
incus network list hz-cluster:
|
|
|
|
# Exec into a container
|
|
incus exec hz-cluster:monitoring -- bash
|
|
|
|
# Start/stop a container
|
|
incus start hz-cluster:monitoring
|
|
incus stop hz-cluster:monitoring --force
|
|
|
|
# Check container network
|
|
incus exec hz-cluster:nginx-lb-01 -- ip a
|
|
```
|
|
|
|
### Teardown
|
|
|
|
```bash
|
|
# Force destroy all VMs (no nice cleanup needed for test lab)
|
|
source env
|
|
for vmid in 910 911 912 913; do
|
|
incusos/helpers/proxmox-api POST \
|
|
"/nodes/aether-lab/qemu/${vmid}/status/stop" \
|
|
'{"forceStop":1}' 2>/dev/null || true
|
|
sleep 3
|
|
incusos/helpers/proxmox-api DELETE \
|
|
"/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
|
|
done
|
|
|
|
# Remove stale remotes
|
|
incus remote remove hz-cluster 2>/dev/null || true
|
|
incus remote remove oc-server 2>/dev/null || true
|
|
```
|
|
|
|
---
|
|
|
|
## Known issues and workarounds
|
|
|
|
### OC remote add loops (incus remote add oc-server fails)
|
|
|
|
`incus remote add` enters a trust token prompt loop for OC because OC
|
|
does not implement the standard Incus trust token flow.
|
|
|
|
**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
|
|
and save the TLS certificate separately.
|
|
|
|
### IncusOS management NIC name is `mgmt`, not `eth0`
|
|
|
|
All IncusOS instances use `mgmt` as the management NIC name. Use
|
|
`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
|
|
|
|
### Aether `PUT /api/clusters/{id}/awx-config` returns 400
|
|
|
|
Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
|
|
|
|
### deploy-observability: don't use --oc-server with OC
|
|
|
|
The `--oc-server` flag tries to list instances via the standard Incus
|
|
API (`/1.0/instances`), which OC does not implement. Omit the flag.
|
|
|
|
### OVN SNAT internet access from net-prod containers is unreliable
|
|
|
|
Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
|
|
reach the internet. This works briefly after container launch (via Proxmox
|
|
NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
|
|
stale. Connections return `Connection refused` from Fastly CDN.
|
|
|
|
**Workaround**: Download packages locally and push them in with
|
|
`incus file push` instead of relying on apt/apk inside net-prod containers.
|
|
|
|
Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
|
|
Proxmox masquerade and are unaffected.
|
|
|
|
### deploy-awx temp file permission errors on resume
|
|
|
|
Temp files created as root inside the AWX VM persist between runs.
|
|
Clean up before resuming:
|
|
|
|
```bash
|
|
incus exec hz-cluster:awx -- rm -f \
|
|
/tmp/awx-operator-kustomization.yaml \
|
|
/tmp/awx-kustomization.yaml \
|
|
/tmp/awx-instance.yaml
|
|
```
|
|
|
|
### aether_url in config files must use key-specific sed
|
|
|
|
Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
|
|
A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
|
|
already fixed in all scripts in this repo.
|