maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add Hetzner lab guide + target configs for AWX and HAProxy 

Validates full end-to-end deployment: OC + 3-node Incus cluster,
OVN overlay, HAProxy HA pair, AWX on K3s, Prometheus/Grafana stack.

New files:
- hetzner/hetzner-lab-guide.md — complete end-to-end deployment guide
- incusos/targets/hetzner/awx.yaml — AWX deploy config (hz-node-03, 10.10.0.122)
- incusos/targets/hetzner/haproxy.yaml — HAProxy deploy config (VIP 10.10.0.200)

Script fixes discovered during Hetzner deployment:
- deploy-awx: parse target_remote correctly (was reading target_node key);
  add RESUME flag support; fix target_node passthrough to incus launch;
  fix aether_url sed to not strip :8443 port
- deploy-observability: fix --remote flag (was --cluster-remote in docs),
  fix --resume to clearly skip phases 1-5 including node-exporters
- hetzner/proxmox-setup: state detection improvements from full run
- incusos-proxmox, deploy-haproxy: minor fixes from Hetzner testing

Guide covers known Aether v6.4.202 workarounds:
- HAProxy ACL bug (OVN LB DNAT-only, source IP not VIP)
- AWX cluster binding API bug (DB workaround documented)
- OVN SNAT unreliable for net-prod containers (offline install procedure)
- node-exporter manual deploy when --resume skips phase 5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-03-03 16:03:08 +01:00
parent 68daf96e61
commit 594467ddf6
11 changed files with 1171 additions and 260 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@ -26,8 +26,9 @@ Thumbs.db
# Go binaries (flasher-tool) # Go binaries (flasher-tool)
flasher-tool flasher-tool
# Proxmox connection config (contains credentials) # Proxmox connection configs (contain secrets, per-target and default)
proxmox.yaml incusos/proxmox.yaml
incusos/targets/*/proxmox.yaml
# Per-target env files # Per-target env files
env-* env-*

907
hetzner/hetzner-lab-guide.md Normal file
View File

@ -0,0 +1,907 @@
# Hetzner IncusOS Lab — End-to-End Guide
Full deployment guide for a production-quality IncusOS lab on a Hetzner
dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
networking, HAProxy load balancing, AWX automation, and observability.
**Estimated time**: 3-4 hours for a clean run.
---
## Overview
```
Hetzner dedicated (256 GB RAM, 32+ cores)
└─ Proxmox VE 9
├─ hz-oc (VMID 910) — Operations Center 0.4.x
├─ hz-node-01 (VMID 911) — Incus cluster init + OVN control plane
├─ hz-node-02 (VMID 912) — Incus cluster member
└─ hz-node-03 (VMID 913) — Incus cluster member
└─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
├─ ffsdn-haproxy-52-01/02 — HAProxy HA pair
├─ nginx-lb-01/02/03 — Test backends
├─ monitoring — Prometheus + Grafana
├─ node-exp-01/02/03 — Node exporters
└─ awx — AWX Operator on K3s
```
**Network topology:**
- vmbr0: Public interface (SSH + WireGuard only)
- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
- OVN overlay: 10.10.10.0/24 (containers on net-prod)
- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
---
## Prerequisites
- Hetzner dedicated server with Proxmox VE 9 installed and configured
(see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
- This repository cloned locally
- `env` file populated (see below)
- `incusos/targets/hetzner/proxmox.yaml` filled in
- Incus client installed on workstation
### env file
```bash
# incusos-contrib/env
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
export AETHER_ADMIN_PASSWORD="admin1234"
```
### proxmox.yaml
```bash
cp incusos/targets/hetzner/proxmox.yaml.example \
incusos/targets/hetzner/proxmox.yaml
# Fill in: host, node name, token_id, storage
```
---
## 1. Deploy IncusOS VMs
```bash
source env
incusos/incusos-proxmox \
--proxmox incusos/targets/hetzner/proxmox.yaml \
incusos/targets/hetzner/lab-production.yaml
```
**What gets deployed:**
| VMID | Name | IP | RAM | Cores | Disk |
|------|-----------|---------------|-------|-------|------|
| 910 | hz-oc | 10.10.0.110 | 8 GB | 4 | 100G |
| 911 | hz-node-01| 10.10.0.111 | 80 GB | 16 | 150G |
| 912 | hz-node-02| 10.10.0.112 | 80 GB | 16 | 100G |
| 913 | hz-node-03| 10.10.0.113 | 80 GB | 16 | 100G |
**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
interrupt. Monitor progress:
```bash
# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
source env
incusos/helpers/proxmox-screenshot 910 # hz-oc
incusos/helpers/proxmox-screenshot 911 # hz-node-01
```
Wait until all VMs are reachable:
```bash
for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
echo -n "$ip: "
ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
done
```
---
## 2. Add Incus remotes
Add the cluster remote pointing at hz-node-01 (the init node):
```bash
incus remote add hz-cluster https://10.10.0.111:8443 \
--accept-certificate --auth-type tls
```
> **Note**: The `incus remote add` command prompts for a trust token.
> Obtain the token from OC after cluster formation (section 3), or
> if using a pre-formed cluster, generate one with:
> `incus config trust add --name workstation`
---
## 3. Cluster formation via Operations Center
Open the Operations Center UI:
```
https://10.10.0.110:8443
Username: admin
Password: admin (first login, then set a new password)
```
### 3.1 Add Incus nodes
In OC: **Servers → Add Server** for each node:
| Field | hz-node-01 | hz-node-02 | hz-node-03 |
|---------|------------------|------------------|------------------|
| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
| Trust | via token | via token | via token |
Each node's trust token is displayed on the IncusOS console on first boot.
### 3.2 Form the cluster
In OC: **Clusters → Create Cluster**
- Name: `hz-cluster`
- Init node: `hz-node-01`
- Members: `hz-node-02`, `hz-node-03`
- App seed: `{}` (empty JSON for manual post-formation setup)
OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
HAProxy and AWX configuration.
### 3.3 Verify cluster
```bash
incus cluster list hz-cluster:
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
```
---
## 4. OVN networking
### 4.1 Create the OVN uplink network
The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
`eth0` or `ens18`. This is specific to IncusOS.
```bash
# Create UPLINK on each node
incus network create hz-cluster:UPLINK --type=physical \
--target hz-node-01 \
config parent=mgmt
incus network create hz-cluster:UPLINK --type=physical \
--target hz-node-02 \
config parent=mgmt
incus network create hz-cluster:UPLINK --type=physical \
--target hz-node-03 \
config parent=mgmt
# Finalize UPLINK on cluster level
incus network create hz-cluster:UPLINK \
config ipv4.address=none \
config ipv6.address=none
```
### 4.2 Deploy OVN central
OVN central runs as a privileged container on hz-node-01:
```bash
incus launch images:debian/12 hz-cluster:ovn-central \
--vm=false \
--target hz-node-01 \
-c security.privileged=true \
-d root,size=20GiB
# Wait for agent
sleep 15
# Install ovn-central
incus exec hz-cluster:ovn-central -- bash -c "
apt-get update -qq
apt-get install -y -qq ovn-central
systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
"
```
### 4.3 Expose OVN NB/SB ports to cluster
Create proxy devices so cluster nodes can reach OVN:
```bash
# Get ovn-central IP
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
incus config device add hz-cluster:ovn-central ovn-nb proxy \
listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
bind=host
incus config device add hz-cluster:ovn-central ovn-sb proxy \
listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
bind=host
```
### 4.4 Create OVN overlay network
```bash
# Use hz-node-01 IP as OVN NB/SB server
NODE01_IP="10.10.0.111"
incus network create hz-cluster:net-prod \
--type=ovn \
config network=UPLINK \
config ipv4.address=10.10.10.1/24 \
config ipv4.nat=true \
config ipv6.address=none \
config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
```
Wait for the network to reach **Created** state:
```bash
incus network info hz-cluster:net-prod
```
### 4.5 Configure external IP range (OVN VIPs)
```bash
# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
# These are allocated on the Proxmox host's vmbr1 bridge
# No additional configuration needed — OVN will ARP for these IPs
```
Verify:
```bash
incus network list hz-cluster:
# net-prod should show type: ovn, state: Created
```
---
## 5. Deploy Aether
### 5.1 Deploy the Aether VM
Use the macvlan method to give Aether a direct vmbr1 IP:
```bash
incus launch images:ubuntu/22.04 hz-cluster:aether \
--vm \
--target hz-node-01 \
-d root,size=40GiB \
-c limits.cpu=4 \
-c limits.memory=8GiB \
-d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
```
> **Alternative**: use Aether golden image if already built on the cluster.
> Check `incus image list hz-cluster: | grep aether`.
Wait for the VM, then SSH in:
```bash
ssh ubuntu@10.10.0.120
```
Follow the [Aether installation guide](../notes/aether-guide.md) for the
full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
### 5.2 Configure Aether cluster
In Aether UI (`https://10.10.0.120:8443`):
1. **Settings → Clusters → Add Cluster**
- Name: hz-cluster
- Endpoint: `https://10.10.0.111:8443`
- Authenticate (generates a trust token, add it to the cluster)
2. **Settings → Operations Centers → Add OC**
- URL: `https://10.10.0.110:8443`
3. Verify hz-cluster instances sync in Aether dashboard.
Aether default credentials: `admin` / `admin` (change on first login).
For Aether password, set in `env`:
```bash
export AETHER_ADMIN_PASSWORD="admin1234"
```
---
## 6. HAProxy load balancing
### 6.1 Create config file
Save as `incusos/targets/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
```yaml
haproxy:
cluster_remote: hz-cluster
cluster_id: 52 # from OC cluster listing
ovn_network: net-prod
vip: 10.10.0.200
haproxy_01_ip: 10.10.10.50
haproxy_02_ip: 10.10.10.51
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
service_name: web-test
image_version: "1.0.0"
aether_url: https://10.10.0.120:8443
```
> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
> It's the numeric ID assigned when the cluster was created.
### 6.2 Build HAProxy image
Before deploying infrastructure, build and push the HAProxy image:
In Aether UI → **HAProxy → Images → Build**:
- Version: `1.0.0`
- Cluster: `hz-cluster`
- Click **Build** and wait 5-10 minutes
After build, push to hz-cluster:
- Click **Push** → select `hz-cluster`
- Click **Set Current**
Alternatively, use the script (requires the UI steps above for first build):
```bash
source env
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
```
### 6.3 Full deploy
```bash
source env
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
```
This:
1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
3. Creates the `web-test` service (HTTP, round-robin, health checks)
On subsequent runs (if backends already exist):
```bash
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
--skip-backends --skip-image
```
### 6.4 Verify
```bash
curl http://10.10.0.200/
# Should show "Backend: nginx-lb-0x"
# Test distribution
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
```
---
## 7. AWX automation
### 7.1 Create config file
Save as `incusos/targets/hetzner/awx.yaml`:
```yaml
awx:
vm_name: awx
target_remote: hz-cluster
target_node: hz-node-03 # specific node within cluster
ip: 10.10.0.122/24
gateway: 10.10.0.1
dns: 10.10.0.1
cpu: 4
memory: 8GiB
disk: 40GiB
aether_url: https://10.10.0.120:8443
aether_cluster_id: 52
```
> **target_node vs target_remote**: `target_remote` is the Incus remote
> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
> member to deploy on (e.g. `hz-node-03`). Both are required for
> cluster deployments.
### 7.2 Deploy
```bash
source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy
```
This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
AWX will be at `http://10.10.0.122:30080`.
On partial failure (e.g. network issue mid-deploy), resume:
```bash
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy --resume
# Skips VM creation and K3s install phases, jumps to AWX Operator phase
```
If you get `permission denied` on temp files from a previous aborted run:
```bash
incus exec hz-cluster:awx -- rm -f \
/tmp/awx-operator-kustomization.yaml \
/tmp/awx-kustomization.yaml \
/tmp/awx-instance.yaml
```
### 7.3 Configure AWX
> **Wait for the awx-task pod before running --configure.** The AWX web
> deployment becomes ready first (~3 min), but `--configure` needs the
> task pod. Check it:
> ```bash
> incus exec hz-cluster:awx -- kubectl -n awx wait \
> --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
> --timeout=300s
> ```
```bash
source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --configure
```
This creates:
- Project: `incus-contrib` (linked to your git repo)
- Inventory: `incus-lab`
- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
Get AWX admin password:
```bash
incusos/deploy-awx -c /tmp/hz-awx.yaml --status
```
### 7.4 Join Aether
Generate an AWX API token for Aether:
```bash
source env
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --join-aether
# Prints the PAT token and template IDs
```
In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
- Name: `lab-awx`
- URL: `http://10.10.0.122:30080`
- Token: (paste token from `--join-aether` output)
- Verify SSL: unchecked (HTTP endpoint)
Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
### 7.5 Bind AWX to cluster
**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
endpoint returns `400 "Invalid cluster ID"` due to a bug.
**Workaround** (direct DB update):
```bash
# First confirm the AWX endpoint ID (assigned by the database):
incus exec hz-cluster:aether -- bash -c \
"su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
UPDATE incus_ovn_clusters
SET awx_endpoint_id=2,
awx_post_deploy_template_id=9,
awx_decommission_template_id=10,
awx_job_timeout_seconds=600
WHERE cluster_id=52;
\\\"\""
```
> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
> `postgres` system user. The endpoint ID is auto-assigned — always check
> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
> to see if the endpoint has been fixed. If it returns a valid schema,
> use the API instead of the DB workaround.
Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
the AWX endpoint, post-deploy, and decommission templates.
---
## 8. Observability stack
### 8.1 Deploy
```bash
source env
incusos/deploy-observability \
--remote hz-cluster \
--monitoring-target hz-node-01 \
--monitoring-ip 10.10.10.70 \
--forward-vip 10.10.0.201 \
--deploy
```
> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
> Operations Center (hz-oc) runs OC, not standard Incus — its
> `/1.0/instances` endpoint returns 404 and will fail the node
> discovery phase.
> **Grafana install may fail if net-prod containers lack internet.**
> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
> after container launch but can become unreliable. If phase 4 fails:
>
> ```bash
> # Download Grafana locally and push it in
> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
> incus exec hz-cluster:monitoring -- bash -c "
> export DEBIAN_FRONTEND=noninteractive
> dpkg -i /tmp/grafana.deb; apt-get -f install -y
> mkdir -p /etc/grafana/provisioning/datasources
> cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
> apiVersion: 1
> datasources:
> - name: Prometheus
> type: prometheus
> uid: prometheus
> access: proxy
> url: http://localhost:9090
> isDefault: true
> - name: Loki
> type: loki
> uid: loki
> access: proxy
> url: http://localhost:3100
> EOF
> systemctl daemon-reload
> systemctl enable grafana-server
> systemctl start grafana-server
> "
> ```
>
> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
> network forward, and dashboards).
> **Node-exporter containers are skipped by `--resume`.** If internet
> was unavailable during phase 5, deploy them manually:
>
> ```bash
> # Download node_exporter binary locally
> curl -sL -o /tmp/ne.tar.gz \
> https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
>
> for i in 1 2 3; do
> name="node-exp-0${i}"
> ip="10.10.10.7${i}"
> node="hz-node-0${i}"
>
> incus launch images:alpine/3.20 hz-cluster:${name} \
> -n net-prod --target ${node} \
> -c limits.memory=128MiB -c security.privileged=true
> incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
> sleep 5
>
> for dev in proc sys root; do
> src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
> incus config device add hz-cluster:${name} host-${dev} disk \
> source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
> incus config device add hz-cluster:${name} host-${dev} disk \
> source=/ path=/host/root readonly=true 2>/dev/null || true
> done
>
> incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
> hz-cluster:${name}/usr/bin/node_exporter
>
> incus exec hz-cluster:${name} -- sh -c "
> chmod +x /usr/bin/node_exporter
> cat > /etc/init.d/node-exporter << 'INIT'
> #!/sbin/openrc-run
> description=\"Node Exporter\"
> command=/usr/bin/node_exporter
> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
> command_background=true
> pidfile=/run/node-exporter.pid
> INIT
> chmod +x /etc/init.d/node-exporter
> rc-update add node-exporter default
> rc-service node-exporter start
>
> cat > /etc/network/interfaces << NETCFG
> auto lo
> iface lo inet loopback
> auto eth0
> iface eth0 inet static
> address ${ip}/24
> gateway 10.10.10.1
> NETCFG
> rc-service networking restart
> "
>
> incus config device set hz-cluster:${name} eth0 \
> security.acls=monitoring-allow \
> security.acls.default.ingress.action=allow \
> security.acls.default.egress.action=allow
> done
> ```
On partial failure (phases 6-9 only), resume:
```bash
source env
incusos/deploy-observability \
--remote hz-cluster \
--monitoring-target hz-node-01 \
--monitoring-ip 10.10.10.70 \
--forward-vip 10.10.0.201 \
--resume --deploy
```
After deploying, fix the HAProxy prometheus exporter (default stats page
doesn't expose `/metrics`):
```bash
for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
incus exec hz-cluster:${ct} -- bash -c "
sed -i 's/ stats enable/ http-request use-service prometheus-exporter if { path \/metrics }\n stats enable/' /etc/haproxy/haproxy.cfg
systemctl reload haproxy
"
done
```
### 8.2 Access
| Service | URL | Credentials |
|------------|------------------------------|-------------|
| Grafana | http://10.10.0.201:3000 | admin/admin |
| Prometheus | http://10.10.0.201:9090 | — |
9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
storage, networking, capacity planning, and log explorer.
Verify all Prometheus targets are up:
```bash
curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
import sys,json
for t in json.load(sys.stdin)['data']['activeTargets']:
print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
"
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
```
---
## 9. WireGuard access (macOS)
If you use the macOS WireGuard app from the App Store:
- The app uses Network Extension (visible as `utun5` in `ifconfig`)
- `wg show` always shows empty — this is normal with the App Store app
- The app allows only **one active tunnel at a time**
For a second simultaneous tunnel:
```bash
# Install wg-quick for macOS
brew install wireguard-tools
# Start the second tunnel alongside the App Store one
sudo wg-quick up ~/.config/wireguard/hetzner.conf
```
If the server added a new WireGuard peer (new workstation/client):
```bash
# On the Hetzner Proxmox host (via the existing tunnel or public IP)
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
ssh hetzner-lab "wg-quick save wg0"
```
---
## 10. Incus remote management
### Add remotes
```bash
# OC (Operations Center — not standard Incus)
# Note: Must be added manually — incus remote add loops prompting for token
# OC does not support the standard trust token flow.
# Edit ~/.config/incus/config.yml directly:
```
Add to `~/.config/incus/config.yml`:
```yaml
remotes:
oc-server:
addr: https://10.10.0.110:8443
auth_type: tls
protocol: incus
public: false
hz-cluster:
addr: https://10.10.0.111:8443
auth_type: tls
protocol: incus
public: false
```
Copy the OC certificate (accept it with curl first):
```bash
# Accept and save OC cert
curl -sk https://10.10.0.110:8443 > /dev/null
# The cert ends up in your browser cache — use openssl to fetch it
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
| openssl x509 > ~/.config/incus/servercerts/oc-server.crt
```
> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
> fail. OC does not expose the standard `/1.0/instances` endpoint.
### After redeploy (remote cert changed)
```bash
# Remove old remote and re-add
incus remote remove hz-cluster
incus remote add hz-cluster https://10.10.0.111:8443 \
--accept-certificate --auth-type tls
```
---
## 11. Quick reference
### Access URLs
| Service | URL | Credentials |
|-----------------|----------------------------------|---------------------|
| Operations Center | https://10.10.0.110:8443 | admin / (set on first login) |
| Aether | https://10.10.0.120:8443 | admin / admin1234 |
| AWX | http://10.10.0.122:30080 | admin / (from deploy-awx --status) |
| Grafana | http://10.10.0.201:3000 | admin / admin |
| Prometheus | http://10.10.0.201:9090 | — |
| HAProxy VIP | http://10.10.0.200/ | — |
### IP allocation
| Range | Purpose |
|-----------------------|------------------------------|
| 10.10.0.1 | Proxmox host (vmbr1 gateway) |
| 10.10.0.110-113 | IncusOS VMs (OC + nodes) |
| 10.10.0.120 | Aether VM (macvlan) |
| 10.10.0.122 | AWX VM (macvlan) |
| 10.10.0.200+ | OVN external IPs (VIPs) |
| 10.10.10.0/24 | OVN overlay (containers) |
| 10.10.10.50-51 | HAProxy containers |
| 10.10.10.60-62 | nginx test backends |
| 10.10.10.70 | monitoring container |
| 10.10.10.71-73 | node-exporter containers |
| 10.10.99.0/24 | WireGuard tunnel |
| 10.10.99.1 | WireGuard server |
| 10.10.99.2 | Your workstation |
### Cluster IDs
- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
your OC UI if different)
### Useful incus commands
```bash
# List all instances in cluster
incus list hz-cluster:
# List cluster members
incus cluster list hz-cluster:
# List networks
incus network list hz-cluster:
# Exec into a container
incus exec hz-cluster:monitoring -- bash
# Start/stop a container
incus start hz-cluster:monitoring
incus stop hz-cluster:monitoring --force
# Check container network
incus exec hz-cluster:nginx-lb-01 -- ip a
```
### Teardown
```bash
# Force destroy all VMs (no nice cleanup needed for test lab)
source env
for vmid in 910 911 912 913; do
incusos/helpers/proxmox-api POST \
"/nodes/aether-lab/qemu/${vmid}/status/stop" \
'{"forceStop":1}' 2>/dev/null || true
sleep 3
incusos/helpers/proxmox-api DELETE \
"/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
done
# Remove stale remotes
incus remote remove hz-cluster 2>/dev/null || true
incus remote remove oc-server 2>/dev/null || true
```
---
## Known issues and workarounds
### OC remote add loops (incus remote add oc-server fails)
`incus remote add` enters a trust token prompt loop for OC because OC
does not implement the standard Incus trust token flow.
**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
and save the TLS certificate separately.
### IncusOS management NIC name is `mgmt`, not `eth0`
All IncusOS instances use `mgmt` as the management NIC name. Use
`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
### Aether `PUT /api/clusters/{id}/awx-config` returns 400
Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
### deploy-observability: don't use --oc-server with OC
The `--oc-server` flag tries to list instances via the standard Incus
API (`/1.0/instances`), which OC does not implement. Omit the flag.
### OVN SNAT internet access from net-prod containers is unreliable
Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
reach the internet. This works briefly after container launch (via Proxmox
NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
stale. Connections return `Connection refused` from Fastly CDN.
**Workaround**: Download packages locally and push them in with
`incus file push` instead of relying on apt/apk inside net-prod containers.
Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
Proxmox masquerade and are unaffected.
### deploy-awx temp file permission errors on resume
Temp files created as root inside the AWX VM persist between runs.
Clean up before resuming:
```bash
incus exec hz-cluster:awx -- rm -f \
/tmp/awx-operator-kustomization.yaml \
/tmp/awx-kustomization.yaml \
/tmp/awx-instance.yaml
```
### aether_url in config files must use key-specific sed
Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
already fixed in all scripts in this repo.

356
hetzner/proxmox-setup
View File

@ -34,7 +34,6 @@ WG_PORT="51820"
SKIP_REPOS=false SKIP_REPOS=false
SKIP_WIREGUARD=false SKIP_WIREGUARD=false
SKIP_FIREWALL=false SKIP_FIREWALL=false
SKIP_STORAGE=false
# Derived (computed after argument parsing) # Derived (computed after argument parsing)
SUBNET_BASE="" # e.g. 10.10.0 SUBNET_BASE="" # e.g. 10.10.0
@ -72,12 +71,7 @@ DETECT_VMBR1_IP="" # e.g. "10.10.0.1/24" or empty
DETECT_IP_FORWARD="" # "1" or "0" DETECT_IP_FORWARD="" # "1" or "0"
DETECT_NAT_ACTIVE="" # "yes" or empty DETECT_NAT_ACTIVE="" # "yes" or empty
# Storage
STATE_STORAGE="done" # done | available | needed
DETECT_ZPOOLS="" DETECT_ZPOOLS=""
DETECT_PVE_STORAGE=""
DETECT_AVAILABLE_DISKS=""
DETECT_AVAILABLE_DISK_COUNT="0"
# WireGuard # WireGuard
STATE_WIREGUARD="needed" # done | needed STATE_WIREGUARD="needed" # done | needed
@ -91,11 +85,16 @@ STATE_FIREWALL="needed" # done | needed
DETECT_NFT_RULES="" # "yes" if nftables rules loaded with our pattern DETECT_NFT_RULES="" # "yes" if nftables rules loaded with our pattern
DETECT_NFT_FILE="" # "yes" if config file exists DETECT_NFT_FILE="" # "yes" if config file exists
# DNS (dnsmasq on vmbr1 -- required for IncusOS boot)
STATE_DNSMASQ="needed" # done | needed
DETECT_DNSMASQ_ACTIVE="" # "yes" if dnsmasq is installed and listening on vmbr1
# API token # API token
STATE_API="needed" # done | partial | needed STATE_API="needed" # done | partial | needed
DETECT_API_ROLE="" # "yes" or empty DETECT_API_ROLE="" # "yes" or empty
DETECT_API_POOL="" # "yes" or empty DETECT_API_POOL="" # "yes" or empty
DETECT_API_TOKEN="" # "yes" or empty DETECT_API_TOKEN="" # "yes" or empty
DETECT_API_ACLS="" # "yes" if /vms ACL is set
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Color and formatting # Color and formatting
@ -206,7 +205,6 @@ ${BOLD}DESCRIPTION${RESET}
- No-subscription repositories and system update - No-subscription repositories and system update
- Private bridge (vmbr1) with NAT masquerading - Private bridge (vmbr1) with NAT masquerading
- ZFS storage pool on extra disks
- WireGuard tunnel for secure remote access - WireGuard tunnel for secure remote access
- nftables firewall lockdown - nftables firewall lockdown
- API token for incusos-proxmox automation - API token for incusos-proxmox automation
@ -220,7 +218,6 @@ ${BOLD}OPTIONS${RESET}
${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET}) ${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET})
${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT}) ${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT})
${BOLD}--skip-repos${RESET} Skip repository changes ${BOLD}--skip-repos${RESET} Skip repository changes
${BOLD}--skip-storage${RESET} Skip storage pool creation
${BOLD}--skip-wireguard${RESET} Skip WireGuard setup ${BOLD}--skip-wireguard${RESET} Skip WireGuard setup
${BOLD}--skip-firewall${RESET} Skip firewall lockdown ${BOLD}--skip-firewall${RESET} Skip firewall lockdown
${BOLD}-n, --dry-run${RESET} Preview actions without executing ${BOLD}-n, --dry-run${RESET} Preview actions without executing
@ -282,10 +279,6 @@ parse_args() {
SKIP_REPOS=true SKIP_REPOS=true
shift shift
;; ;;
--skip-storage)
SKIP_STORAGE=true
shift
;;
--skip-wireguard) --skip-wireguard)
SKIP_WIREGUARD=true SKIP_WIREGUARD=true
shift shift
@ -389,8 +382,6 @@ echo '---BLOCK_DEVICES---'
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
echo '---ZPOOLS---' echo '---ZPOOLS---'
zpool list -H 2>/dev/null || true zpool list -H 2>/dev/null || true
echo '---PVE_STORAGE---'
pvesm status 2>/dev/null || true
echo '---ENTERPRISE_REPOS---' echo '---ENTERPRISE_REPOS---'
# Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats # Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats
enterprise_active='no' enterprise_active='no'
@ -466,6 +457,16 @@ echo '---NFT_FILE---'
test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no' test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
echo '---NFT_RULES---' echo '---NFT_RULES---'
nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no' nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
echo '---DNSMASQ---'
# dnsmasq installed and listening on port 53 on vmbr1?
dnsmasq_ok='no'
if command -v dnsmasq >/dev/null 2>&1 && systemctl is-active dnsmasq >/dev/null 2>&1; then
# Check it's listening on the bridge IP
if ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq; then
dnsmasq_ok='yes'
fi
fi
echo "$dnsmasq_ok"
echo '---API_ROLE---' echo '---API_ROLE---'
pveum role list --output-format json 2>/dev/null \ pveum role list --output-format json 2>/dev/null \
| python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \ | python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \
@ -478,30 +479,10 @@ echo '---API_TOKEN---'
pveum user token list automation@pve --output-format json 2>/dev/null \ pveum user token list automation@pve --output-format json 2>/dev/null \
| python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \ | python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \
|| echo 'no' || echo 'no'
echo '---AVAILABLE_DISKS---' echo '---API_ACLS---'
# Build list of whole-disk names that are part of ZFS pools. pveum acl list --output-format json 2>/dev/null \
# zpool status shows partition names (e.g. nvme0n1p3), so strip the | python3 -c 'import sys,json; acls=json.load(sys.stdin); paths={a.get("path") for a in acls if "automation@pve" in a.get("ugid","")}; ok=("/vms" in paths and "/sdn" in paths); print("yes" if ok else "no")' 2>/dev/null \
# partition suffix to get the parent disk name. || echo 'no'
zfs_disks=$(zpool status 2>/dev/null \
| grep -oP '(nvme\S+|sd[a-z]+\S*|vd[a-z]+\S*)' \
| sed -E 's/p?[0-9]+$//' \
| sort -u || true)
lsblk -dno NAME,SIZE,TYPE | while read name size type; do
[ "$type" != 'disk' ] && continue
# Skip disks used by ZFS pools
echo "$zfs_disks" | grep -qw "$name" && continue
# Skip disks with active mount points
lsblk -no MOUNTPOINT "/dev/$name" 2>/dev/null | grep -q '/' && continue
# Check for existing partition table / signatures
partcount=$(lsblk -no NAME "/dev/$name" 2>/dev/null | wc -l)
if [ "$partcount" -gt 1 ]; then
# Has partitions -- detect type
fstype=$(lsblk -no FSTYPE "/dev/$name" 2>/dev/null | grep -v '^$' | head -1)
echo "$name $size partitioned:${fstype:-unknown}"
else
echo "$name $size clean"
fi
done
echo '---END---' echo '---END---'
DETECT DETECT
) )
@ -545,7 +526,6 @@ _parse_section() {
PUBLIC_IP) DETECT_PUBLIC_IP="$content" ;; PUBLIC_IP) DETECT_PUBLIC_IP="$content" ;;
BLOCK_DEVICES) DETECT_BLOCK_DEVICES="$content" ;; BLOCK_DEVICES) DETECT_BLOCK_DEVICES="$content" ;;
ZPOOLS) DETECT_ZPOOLS="$content" ;; ZPOOLS) DETECT_ZPOOLS="$content" ;;
PVE_STORAGE) DETECT_PVE_STORAGE="$content" ;;
ENTERPRISE_REPOS) ENTERPRISE_REPOS)
if [[ "$content" == "yes" ]]; then if [[ "$content" == "yes" ]]; then
DETECT_ENTERPRISE_ACTIVE="yes" DETECT_ENTERPRISE_ACTIVE="yes"
@ -577,6 +557,9 @@ _parse_section() {
;; ;;
NFT_FILE) DETECT_NFT_FILE="$content" ;; NFT_FILE) DETECT_NFT_FILE="$content" ;;
NFT_RULES) DETECT_NFT_RULES="$content" ;; NFT_RULES) DETECT_NFT_RULES="$content" ;;
DNSMASQ)
if [[ "$content" == "yes" ]]; then DETECT_DNSMASQ_ACTIVE="yes"; fi
;;
API_ROLE) API_ROLE)
if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi
;; ;;
@ -586,13 +569,8 @@ _parse_section() {
API_TOKEN) API_TOKEN)
if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi
;; ;;
AVAILABLE_DISKS) API_ACLS)
DETECT_AVAILABLE_DISKS="$content" if [[ "$content" == "yes" ]]; then DETECT_API_ACLS="yes"; fi
if [[ -n "$content" ]]; then
DETECT_AVAILABLE_DISK_COUNT=$(echo "$content" | wc -l)
else
DETECT_AVAILABLE_DISK_COUNT="0"
fi
;; ;;
esac esac
} }
@ -625,21 +603,6 @@ _compute_states() {
STATE_BRIDGE="needed" STATE_BRIDGE="needed"
fi fi
# Storage: check if any ZFS pool is registered as PVE storage
local zfs_in_pve
zfs_in_pve=$(echo "$DETECT_PVE_STORAGE" | grep -c "zfspool" || true)
if [[ "$zfs_in_pve" -gt 0 ]]; then
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
STATE_STORAGE="available" # has pool but also unused disks
else
STATE_STORAGE="done"
fi
elif [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
STATE_STORAGE="needed"
else
STATE_STORAGE="done" # no pool, but no disks available either
fi
# WireGuard # WireGuard
if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
STATE_WIREGUARD="done" STATE_WIREGUARD="done"
@ -655,13 +618,20 @@ _compute_states() {
fi fi
# API token # API token
if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]]; then if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]] && [[ "$DETECT_API_ACLS" == "yes" ]]; then
STATE_API="done" STATE_API="done"
elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
STATE_API="partial" STATE_API="partial"
else else
STATE_API="needed" STATE_API="needed"
fi fi
# dnsmasq (DNS for VMs on vmbr1 -- required for IncusOS boot)
if [[ "$DETECT_DNSMASQ_ACTIVE" == "yes" ]]; then
STATE_DNSMASQ="done"
else
STATE_DNSMASQ="needed"
fi
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -710,12 +680,10 @@ show_status() {
local effective_repos="$STATE_REPOS" local effective_repos="$STATE_REPOS"
local effective_update="$STATE_UPDATE" local effective_update="$STATE_UPDATE"
local effective_storage="$STATE_STORAGE"
local effective_wg="$STATE_WIREGUARD" local effective_wg="$STATE_WIREGUARD"
local effective_fw="$STATE_FIREWALL" local effective_fw="$STATE_FIREWALL"
if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi
if [[ "$SKIP_STORAGE" == true ]]; then effective_storage="skipped"; fi
if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi
if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi
@ -748,14 +716,6 @@ show_status() {
needed) echo " vmbr1 not yet created" ;; needed) echo " vmbr1 not yet created" ;;
esac esac
printf " %-20s %b" "Storage" "$(status_icon "$effective_storage")"
case "$effective_storage" in
done) echo " system pool registered, no extra disks" ;;
available) echo " system pool registered, ${DETECT_AVAILABLE_DISK_COUNT} extra disk(s) available" ;;
needed) echo " ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s), no pool created" ;;
skipped) echo "" ;;
esac
printf " %-20s %b" "WireGuard" "$(status_icon "$effective_wg")" printf " %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
case "$effective_wg" in case "$effective_wg" in
done) echo " wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;; done) echo " wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
@ -774,9 +734,9 @@ show_status() {
done) echo " nftables rules active" ;; done) echo " nftables rules active" ;;
needed) needed)
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
echo " config file exists but rules not loaded" echo " config written, apply manually (see firewall step)"
else else
echo " no rules configured" echo " config not yet written"
fi fi
;; ;;
skipped) echo "" ;; skipped) echo "" ;;
@ -795,6 +755,12 @@ show_status() {
needed) echo " not configured" ;; needed) echo " not configured" ;;
esac esac
printf " %-20s %b" "DNS (dnsmasq)" "$(status_icon "$STATE_DNSMASQ")"
case "$STATE_DNSMASQ" in
done) echo " dnsmasq active, forwarding DNS on vmbr1" ;;
needed) echo " not installed (required for IncusOS boot)" ;;
esac
echo "" echo ""
# Count pending items # Count pending items
@ -802,10 +768,10 @@ show_status() {
if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi
if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi
if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi
if [[ "$effective_storage" == "needed" ]]; then pending=$((pending + 1)); fi
if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi
if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi
if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi
if [[ "$STATE_DNSMASQ" == "needed" ]]; then pending=$((pending + 1)); fi
if [[ "$pending" -eq 0 ]]; then if [[ "$pending" -eq 0 ]]; then
success "All configured -- nothing to do" success "All configured -- nothing to do"
@ -1001,126 +967,46 @@ iface vmbr1 inet static
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Apply: storage # Apply: dnsmasq (DNS for VMs on vmbr1)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
apply_storage() { apply_dnsmasq() {
if [[ "$SKIP_STORAGE" == true ]] || [[ "$STATE_STORAGE" == "done" ]]; then if [[ "$STATE_DNSMASQ" == "done" ]]; then
return 0 return 0
fi fi
step "ZFS storage pool" step "DNS (dnsmasq on vmbr1)"
if [[ "$STATE_STORAGE" == "available" ]]; then info "Will: install dnsmasq, configure it to forward DNS queries from VMs on vmbr1"
info "ZFS pool already registered with Proxmox" info " Interface: vmbr1 (${SUBNET_GW})"
info "${DETECT_AVAILABLE_DISK_COUNT} additional unused disk(s) available:" info " Upstream: 1.1.1.1, 8.8.8.8"
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do info " Required for IncusOS boot (downloads updates from images.linuxcontainers.org)"
if [[ -n "$line" ]]; then echo " /dev/$line"; fi
done
if ! confirm "Create an additional storage pool?"; then
return 0
fi
fi
if [[ "$STATE_STORAGE" == "needed" ]]; then if ! confirm "Install and configure dnsmasq?"; then
# Check if a zpool named local-zfs already exists but isn't in PVE
local existing_pool
existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true)
if [[ -n "$existing_pool" ]]; then
info "ZFS pool 'local-zfs' exists but not registered with Proxmox"
if confirm "Register local-zfs with Proxmox?"; then
if ! dry_run_guard "pvesm add zfspool local-zfs"; then
remote "pvesm add zfspool local-zfs -pool local-zfs"
remote "pvesm set local-zfs -content images,rootdir"
success "local-zfs registered"
fi
fi
return 0
fi
fi
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -eq 0 ]]; then
info "No unused disks found"
return 0
fi
info "Available disks:"
local has_partitioned=false
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
if [[ -n "$line" ]]; then
local dname dsize dstate
dname=$(echo "$line" | awk '{print $1}')
dsize=$(echo "$line" | awk '{print $2}')
dstate=$(echo "$line" | awk '{print $3}')
if [[ "$dstate" == partitioned:* ]]; then
local fsinfo="${dstate#partitioned:}"
echo -e " /dev/${dname} ${dsize} ${YELLOW}has existing partitions (${fsinfo})${RESET}"
else
echo " /dev/${dname} ${dsize} clean"
fi
fi
done
# Check for partitioned disks that need wiping
local partitioned_disks
partitioned_disks=$(echo "$DETECT_AVAILABLE_DISKS" | awk '/partitioned:/{print "/dev/"$1}' | tr '\n' ' ')
if [[ -n "${partitioned_disks// /}" ]]; then
has_partitioned=true
warn "Disk(s) with existing partitions must be wiped before use in a ZFS pool"
warn "Affected: ${partitioned_disks}"
fi
local disk_count="$DETECT_AVAILABLE_DISK_COUNT"
local disk_names
disk_names=$(echo "$DETECT_AVAILABLE_DISKS" | awk 'NF{print "/dev/"$1}' | tr '\n' ' ')
local pool_type
if [[ "$disk_count" -eq 1 ]]; then
pool_type="single"
info "Recommendation: single disk (no redundancy)"
elif [[ "$disk_count" -eq 2 ]]; then
pool_type="mirror"
info "Recommendation: mirror (2 disks, full redundancy)"
else
pool_type="raidz1"
info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)"
fi
local pool_name
pool_name=$(prompt_value "Pool name" "vm-storage")
if [[ "$has_partitioned" == true ]]; then
warn "This will wipe partition tables on: ${partitioned_disks}"
fi
if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then
info "Skipped" info "Skipped"
return 0 return 0
fi fi
if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then dry_run_guard
return 0
fi
# Wipe partition tables on disks that have them remote "apt-get install -y dnsmasq"
for disk in $partitioned_disks; do
info "Wiping partition table on ${disk}..."
remote "wipefs -a ${disk}"
remote "sgdisk -Z ${disk} 2>/dev/null || true"
done
local zpool_cmd="zpool create -f ${pool_name}" local dnsmasq_conf
case "$pool_type" in dnsmasq_conf="# DNS for VMs on vmbr1 (${SUBNET})
mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;; # Generated by proxmox-setup -- do not edit manually
raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;; interface=vmbr1
single) zpool_cmd="${zpool_cmd} ${disk_names}" ;; bind-interfaces
esac no-resolv
server=1.1.1.1
server=8.8.8.8
domain-needed
bogus-priv"
remote "$zpool_cmd" remote_write "/etc/dnsmasq.d/vmbr1.conf" "$dnsmasq_conf"
remote "pvesm add zfspool ${pool_name} -pool ${pool_name}" remote "systemctl enable dnsmasq && systemctl restart dnsmasq"
remote "pvesm set ${pool_name} -content images,rootdir" remote "ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq && echo 'OK' || echo 'WARN: dnsmasq not listening on port 53'"
success "ZFS pool '${pool_name}' created and registered" success "dnsmasq configured on vmbr1"
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1144,6 +1030,21 @@ apply_wireguard() {
return 0 return 0
fi fi
if [[ "$DRY_RUN" == true ]]; then
info "(dry-run) Client config will look like:"
echo ""
echo " [Interface]"
echo " PrivateKey = <generated on setup>"
echo " Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}"
echo ""
echo " [Peer]"
echo " PublicKey = <server public key>"
echo " Endpoint = ${DETECT_PUBLIC_IP:-<server-public-ip>}:${WG_PORT}"
echo " AllowedIPs = 10.10.0.0/16"
echo " PersistentKeepalive = 25"
echo ""
fi
if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
return 0 return 0
fi fi
@ -1206,6 +1107,22 @@ AllowedIPs = ${WG_CLIENT_IP}/32"
echo " PersistentKeepalive = 25" echo " PersistentKeepalive = 25"
echo "" echo ""
warn "Save the client config now -- private key is not stored anywhere else" warn "Save the client config now -- private key is not stored anywhere else"
echo ""
info "Verify WireGuard before applying firewall rules"
if confirm "Connect your WireGuard client now and verify the tunnel?"; then
info "Checking server-side WireGuard status..."
local wg_status
wg_status=$(remote_output "wg show wg0 2>/dev/null" || true)
if echo "$wg_status" | grep -q "latest handshake"; then
success "WireGuard is working -- handshake confirmed"
elif echo "$wg_status" | grep -q "peer:"; then
warn "Peer configured but no handshake yet -- client connected?"
info "Try: ping ${WG_SERVER_IP}"
else
warn "No peers visible -- check wg0 is up on server"
remote_output "systemctl status wg-quick@wg0 --no-pager -l" || true
fi
fi
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1219,31 +1136,6 @@ apply_firewall() {
step "Firewall lockdown (nftables)" step "Firewall lockdown (nftables)"
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
info "Config file exists but rules not loaded"
if confirm "Load existing firewall rules from /etc/nftables-hetzner.conf?"; then
if ! dry_run_guard "nft -f /etc/nftables-hetzner.conf"; then
remote "nft -f /etc/nftables-hetzner.conf"
remote "systemctl enable nftables"
success "Firewall rules loaded"
fi
fi
return 0
fi
info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only"
info "Web UI (8006) and VM traffic only via WireGuard"
warn "Make sure WireGuard is working before applying firewall rules!"
if ! confirm "Apply firewall rules?"; then
info "Skipped"
return 0
fi
if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then
return 0
fi
local nft_rules="#!/usr/sbin/nft -f local nft_rules="#!/usr/sbin/nft -f
flush ruleset flush ruleset
@ -1295,12 +1187,26 @@ table inet filter {
} }
}" }"
if [[ "$DETECT_NFT_FILE" != "yes" ]]; then
if dry_run_guard "Write /etc/nftables-hetzner.conf on ${SSH_HOST}"; then
info "Rules that would be written to /etc/nftables-hetzner.conf:"
echo "$nft_rules" | sed 's/^/ /'
echo ""
else
remote_write "/etc/nftables-hetzner.conf" "$nft_rules" remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
remote "nft -f /etc/nftables-hetzner.conf" success "Config written to /etc/nftables-hetzner.conf"
remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf" fi
remote "systemctl enable nftables" else
info "Config already at /etc/nftables-hetzner.conf (rules not yet loaded)"
fi
success "Firewall applied (SSH + WireGuard only on public interface)" warn "Firewall rules are NOT applied automatically — misfire locks out SSH on Hetzner permanently"
info "Once WireGuard is verified, apply manually:"
echo ""
echo " ssh ${SSH_HOST} nft -f /etc/nftables-hetzner.conf"
echo " ssh ${SSH_HOST} systemctl enable nftables"
echo " ssh ${SSH_HOST} nft list ruleset # verify"
echo ""
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1319,6 +1225,7 @@ apply_api_token() {
[[ "$DETECT_API_ROLE" == "yes" ]] || actions="create IncusOSDeployer role" [[ "$DETECT_API_ROLE" == "yes" ]] || actions="create IncusOSDeployer role"
[[ "$DETECT_API_POOL" == "yes" ]] || actions="${actions:+$actions, }create IncusLab pool" [[ "$DETECT_API_POOL" == "yes" ]] || actions="${actions:+$actions, }create IncusLab pool"
[[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token" [[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
[[ "$DETECT_API_ACLS" == "yes" ]] || actions="${actions:+$actions, }add missing ACLs (/vms, /sdn, storage)"
info "Will: ${actions}" info "Will: ${actions}"
@ -1335,9 +1242,8 @@ apply_api_token() {
if [[ "$DETECT_API_ROLE" != "yes" ]]; then if [[ "$DETECT_API_ROLE" != "yes" ]]; then
remote "pveum role add IncusOSDeployer -privs \ remote "pveum role add IncusOSDeployer -privs \
'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ 'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Audit \
VM.Monitor VM.Audit VM.Console \ Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
Datastore.AllocateSpace Datastore.Audit \
Pool.Allocate Pool.Audit \ Pool.Allocate Pool.Audit \
SDN.Use Sys.Modify'" SDN.Use Sys.Modify'"
success "Created IncusOSDeployer role" success "Created IncusOSDeployer role"
@ -1360,8 +1266,22 @@ apply_api_token() {
local token_secret local token_secret
token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output") token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
# Assign ACLs success "API token created"
echo ""
warn "Token secret (save now -- shown only once):"
echo ""
echo " ${token_secret}"
echo ""
fi
# Assign ACLs (idempotent -- run whenever any ACL is missing).
# PVE 9.x requires /sdn for bridge-backed VMs even with SDN.Use in the role.
# /vms and /nodes/<node> are required for VM creation and start via API.
if [[ "$DETECT_API_ACLS" != "yes" ]]; then
remote "pveum acl modify /vms -user automation@pve -role IncusOSDeployer"
remote "pveum acl modify /nodes/${DETECT_HOSTNAME} -user automation@pve -role IncusOSDeployer"
remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer" remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
remote "pveum acl modify /sdn -user automation@pve -role IncusOSDeployer"
local storage_name local storage_name
storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true) storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
@ -1370,12 +1290,7 @@ apply_api_token() {
fi fi
remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer" remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
success "API token created" success "ACLs configured"
echo ""
warn "Token secret (save now -- shown only once):"
echo ""
echo " ${token_secret}"
echo ""
fi fi
} }
@ -1406,7 +1321,6 @@ show_summary() {
echo "" echo ""
echo " # Hetzner" echo " # Hetzner"
echo " HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>" echo " HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
echo " HETZNER_PROXMOX_ROOT_PASSWORD=<your root password>"
echo "" echo ""
echo -e "${BOLD}Next steps:${RESET}" echo -e "${BOLD}Next steps:${RESET}"
@ -1457,7 +1371,7 @@ main() {
apply_repos apply_repos
apply_update apply_update
apply_bridge apply_bridge
apply_storage apply_dnsmasq
apply_wireguard apply_wireguard
apply_firewall apply_firewall
apply_api_token apply_api_token

24
incusos/deploy-awx
View File

@ -22,6 +22,7 @@ readonly AWX_NAMESPACE="awx"
VM_NAME="awx" VM_NAME="awx"
TARGET_REMOTE="oc-node-02" TARGET_REMOTE="oc-node-02"
TARGET_NODE="" # Node within cluster for --target; defaults to TARGET_REMOTE if empty
AWX_IP="192.168.102.161" AWX_IP="192.168.102.161"
AWX_PREFIX="22" AWX_PREFIX="22"
AWX_GATEWAY="192.168.100.1" AWX_GATEWAY="192.168.100.1"
@ -39,6 +40,7 @@ AETHER_CLUSTER_ID="52"
DRY_RUN=false DRY_RUN=false
VERBOSE=false VERBOSE=false
QUIET=false QUIET=false
RESUME=false
CONFIG_FILE="" CONFIG_FILE=""
ACTION="" ACTION=""
@ -163,8 +165,10 @@ parse_config() {
local val local val
val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && VM_NAME="$val" [[ -n "$val" ]] && VM_NAME="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *target_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && TARGET_REMOTE="$val" [[ -n "$val" ]] && TARGET_REMOTE="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && TARGET_NODE="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
if [[ -n "$val" ]]; then if [[ -n "$val" ]]; then
AWX_IP="${val%%/*}" AWX_IP="${val%%/*}"
@ -184,7 +188,7 @@ parse_config() {
[[ -n "$val" ]] && GIT_REPO="$val" [[ -n "$val" ]] && GIT_REPO="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && GIT_BRANCH="$val" [[ -n "$val" ]] && GIT_BRANCH="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
[[ -n "$val" ]] && AETHER_URL="$val" [[ -n "$val" ]] && AETHER_URL="$val"
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && AETHER_CLUSTER_ID="$val" [[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
@ -375,8 +379,10 @@ action_deploy() {
echo echo
if vm_exists; then if vm_exists; then
if vm_running; then if [[ "${RESUME:-false}" == true ]]; then
error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first." info "AWX VM already exists — resuming from phase 4"
elif vm_running; then
error "AWX VM already exists and is running. Use --status, --heal, --resume, or --cleanup first."
return 1 return 1
else else
warn "AWX VM exists but is not running — starting it" warn "AWX VM exists but is not running — starting it"
@ -387,9 +393,11 @@ action_deploy() {
fi fi
fi fi
if [[ "${RESUME:-false}" != true ]]; then
phase_create_vm phase_create_vm
phase_configure_network phase_configure_network
phase_install_k3s phase_install_k3s
fi
phase_deploy_awx_operator phase_deploy_awx_operator
phase_deploy_awx_instance phase_deploy_awx_instance
phase_verify phase_verify
@ -410,12 +418,13 @@ action_deploy() {
phase_create_vm() { phase_create_vm() {
step "Phase 1: Create VM" step "Phase 1: Create VM"
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then local target_node="${TARGET_NODE:-$TARGET_REMOTE}"
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${target_node} ..."; then
return 0 return 0
fi fi
incus launch images:debian/12 "$(vm_ref)" --vm \ incus launch images:debian/12 "$(vm_ref)" --vm \
--target "$TARGET_REMOTE" \ --target "$target_node" \
-c "limits.cpu=${AWX_CPU}" \ -c "limits.cpu=${AWX_CPU}" \
-c "limits.memory=${AWX_MEMORY}" \ -c "limits.memory=${AWX_MEMORY}" \
-d "root,size=${AWX_DISK}" -d "root,size=${AWX_DISK}"
@ -490,7 +499,7 @@ phase_install_k3s() {
vm_exec bash -c " vm_exec bash -c "
export DEBIAN_FRONTEND=noninteractive export DEBIAN_FRONTEND=noninteractive
apt-get update -qq apt-get update -qq
apt-get install -y -qq curl python3 2>&1 | tail -1 apt-get install -y -qq curl python3 git 2>&1 | tail -1
" "
vm_exec bash -c " vm_exec bash -c "
@ -1162,6 +1171,7 @@ parse_args() {
--cleanup) ACTION="cleanup";; --cleanup) ACTION="cleanup";;
--doctor) ACTION="doctor";; --doctor) ACTION="doctor";;
-c|--config) CONFIG_FILE="$2"; shift;; -c|--config) CONFIG_FILE="$2"; shift;;
--resume) RESUME=true;;
-n|--dry-run) DRY_RUN=true;; -n|--dry-run) DRY_RUN=true;;
-v|--verbose) VERBOSE=true;; -v|--verbose) VERBOSE=true;;
-q|--quiet) QUIET=true;; -q|--quiet) QUIET=true;;

2
incusos/deploy-haproxy
View File

@ -184,7 +184,7 @@ parse_config() {
[[ -n "$val" ]] && SERVICE_NAME="$val" [[ -n "$val" ]] && SERVICE_NAME="$val"
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
[[ -n "$val" ]] && IMAGE_VERSION="$val" [[ -n "$val" ]] && IMAGE_VERSION="$val"
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true) val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
[[ -n "$val" ]] && AETHER_URL="$val" [[ -n "$val" ]] && AETHER_URL="$val"
} }

16
incusos/deploy-observability
View File

@ -43,6 +43,7 @@ OC_SERVER_IP=""
DRY_RUN=false DRY_RUN=false
VERBOSE=false VERBOSE=false
QUIET=false QUIET=false
RESUME=false
ACTION="" ACTION=""
MANUAL_NODES="" MANUAL_NODES=""
@ -103,6 +104,9 @@ ${BOLD}ACTIONS${RESET}
${BOLD}OPTIONS${RESET} ${BOLD}OPTIONS${RESET}
-r, --remote NAME Incus remote (default: ${CLUSTER_REMOTE}) -r, --remote NAME Incus remote (default: ${CLUSTER_REMOTE})
--nodes NODE1,NODE2 Manual node list (skip auto-discovery) --nodes NODE1,NODE2 Manual node list (skip auto-discovery)
--monitoring-target NODE Node to deploy monitoring container on (default: ${MONITORING_TARGET})
--monitoring-ip IP IP for monitoring container (default: ${MONITORING_IP})
--forward-vip IP OVN external IP for Grafana/Prometheus forward (default: ${FORWARD_VIP})
--oc-server IP Monitor oc-server at this IP (proxy device for 9100) --oc-server IP Monitor oc-server at this IP (proxy device for 9100)
-n, --dry-run Preview actions without executing -n, --dry-run Preview actions without executing
-v, --verbose Show detailed output -v, --verbose Show detailed output
@ -298,16 +302,22 @@ action_deploy() {
echo echo
if container_exists "monitoring"; then if container_exists "monitoring"; then
if [[ "$RESUME" == true ]]; then
info "Monitoring container already exists — resuming from phase 5b"
else
error "Monitoring container already exists" error "Monitoring container already exists"
error "Use --status to check health, or --cleanup first." error "Use --status to check health, --cleanup first, or --resume to continue."
return 1 return 1
fi fi
fi
if [[ "$RESUME" != true ]]; then
phase_deploy_monitoring phase_deploy_monitoring
phase_install_prometheus phase_install_prometheus
phase_install_loki phase_install_loki
phase_install_grafana phase_install_grafana
phase_deploy_node_exporters phase_deploy_node_exporters
fi
phase_deploy_oc_server_exporter phase_deploy_oc_server_exporter
phase_create_acl phase_create_acl
phase_configure_haproxy_exporter phase_configure_haproxy_exporter
@ -1708,7 +1718,11 @@ parse_args() {
--doctor) ACTION="doctor";; --doctor) ACTION="doctor";;
-r|--remote) CLUSTER_REMOTE="$2"; shift;; -r|--remote) CLUSTER_REMOTE="$2"; shift;;
--nodes) MANUAL_NODES="$2"; shift;; --nodes) MANUAL_NODES="$2"; shift;;
--monitoring-target) MONITORING_TARGET="$2"; shift;;
--monitoring-ip) MONITORING_IP="$2"; shift;;
--forward-vip) FORWARD_VIP="$2"; shift;;
--oc-server) OC_SERVER_IP="$2"; shift;; --oc-server) OC_SERVER_IP="$2"; shift;;
--resume) RESUME=true;;
-n|--dry-run) DRY_RUN=true;; -n|--dry-run) DRY_RUN=true;;
-v|--verbose) VERBOSE=true;; -v|--verbose) VERBOSE=true;;
-q|--quiet) QUIET=true;; -q|--quiet) QUIET=true;;

19
incusos/incusos-proxmox
View File

@ -1405,6 +1405,13 @@ json.dump(result, sys.stdout, indent=2)
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val" val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
# api_token_secret in proxmox.yaml overrides the repo-wide env file, useful
# for per-target configs where the env file holds a different target's token.
val=$(json_get "$px_json" ".api_token_secret" "")
if [[ -n "$val" ]]; then
export PROXMOX_TOKEN_SECRET="$val"
detail "Using api_token_secret from proxmox.yaml"
fi
rm -f "$px_json" rm -f "$px_json"
} }
@ -2281,6 +2288,7 @@ build_vm_description() {
# Check if a VM has our management marker in its description # Check if a VM has our management marker in its description
pve_vm_is_managed() { pve_vm_is_managed() {
local vmid="$1" local vmid="$1"
local encoded_marker="${MANAGED_MARKER//:/%3A}"
if [[ "$DRY_RUN" == true ]]; then if [[ "$DRY_RUN" == true ]]; then
return 1 return 1
@ -2296,7 +2304,8 @@ pve_vm_is_managed() {
;; ;;
esac esac
echo "$config_output" | grep -qF "$MANAGED_MARKER" echo "$config_output" | grep -qF "$MANAGED_MARKER" || \
echo "$config_output" | grep -qF "$encoded_marker"
} }
# Check if a VM still has install media (ide2) attached. # Check if a VM still has install media (ide2) attached.
@ -2397,15 +2406,15 @@ preflight_checks() {
if [[ "$DRY_RUN" != true ]]; then if [[ "$DRY_RUN" != true ]]; then
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) ssh)
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_STORAGE"; then
error "Storage pool '${PVE_STORAGE}' not found on Proxmox" error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')" error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
error "Fix: update 'proxmox.storage' in your config file" error "Fix: update 'proxmox.storage' in your config file"
errors=$((errors + 1)) errors=$((errors + 1))
fi fi
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_ISO_STORAGE"; then
error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox" error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')" error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
error "Fix: update 'proxmox.iso_storage' in your config file" error "Fix: update 'proxmox.iso_storage' in your config file"
errors=$((errors + 1)) errors=$((errors + 1))
fi fi

27
incusos/targets/hetzner/awx.yaml Normal file
View File

@ -0,0 +1,27 @@
# awx.yaml - AWX deployment config for Hetzner lab
#
# Deploys a Debian 12 VM with K3s + AWX Operator on hz-node-03.
# VM uses macvlan parent=mgmt for direct access on the 10.10.0.0/24 network.
#
# Architecture:
# awx (10.10.0.122) - K3s + AWX Operator + AWX instance
# AWX exposed on NodePort 30080 (HTTP)
#
# Usage:
# source env
# ./incusos/deploy-awx \
# --config incusos/targets/hetzner/awx.yaml \
# --deploy
awx:
vm_name: awx
target_remote: hz-cluster
target_node: hz-node-03
ip: 10.10.0.122/24
gateway: 10.10.0.1
dns: 10.10.0.1
cpu: 4
memory: 8GiB
disk: 40GiB
aether_url: https://10.10.0.120:8443
aether_cluster_id: 52

28
incusos/targets/hetzner/haproxy.yaml Normal file
View File

@ -0,0 +1,28 @@
# haproxy.yaml - HAProxy load balancing config for Hetzner lab
#
# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
# with a VIP on the UPLINK external range (10.10.0.200-210).
#
# Architecture:
# ffsdn-haproxy-52-01 (10.10.10.50) - HAProxy instance 1 on net-prod
# ffsdn-haproxy-52-02 (10.10.10.51) - HAProxy instance 2 on net-prod
# nginx-lb-01/02/03 (10.10.10.60-62) - nginx test backends on net-prod
# VIP: 10.10.0.200:80 - OVN external IP for external access
#
# Usage:
# source env
# ./incusos/deploy-haproxy \
# --config incusos/targets/hetzner/haproxy.yaml \
# --deploy
haproxy:
cluster_remote: hz-cluster
cluster_id: 52
ovn_network: net-prod
vip: 10.10.0.200
haproxy_01_ip: 10.10.10.50
haproxy_02_ip: 10.10.10.51
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
service_name: web-test
image_version: "1.0.0"
aether_url: https://10.10.0.120:8443

12
incusos/targets/hetzner/lab-production.yaml
View File

@ -1,6 +1,6 @@
# lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster) # lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
# #
# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering. # Deploys 1 Operations Center instance and 3 Incus nodes for OC-managed clustering.
# Sized for a Hetzner dedicated server with generous resources. # Sized for a Hetzner dedicated server with generous resources.
# #
# Architecture: # Architecture:
@ -12,7 +12,7 @@
# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet). # Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
# OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external. # OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
# #
# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads) # RAM budget: ~248 GiB of 256 GiB (8 GiB OC + 3×80 GiB nodes; ~8 GiB reserved for Proxmox host)
# #
# Usage: # Usage:
# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ # incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
@ -26,8 +26,8 @@
# Connection settings from proxmox.yaml in this directory. # Connection settings from proxmox.yaml in this directory.
defaults: defaults:
cores: 8 cores: 16
memory: 16384 memory: 81920 # ~80 GiB per node (3 nodes × 80 = 240 GiB + 8 for OC = 248 GiB total)
disk: 100 disk: 100
start_vmid: 910 start_vmid: 910
@ -36,12 +36,12 @@ vms:
app: operations-center app: operations-center
apply_defaults: true apply_defaults: true
cores: 4 cores: 4
memory: 8192 memory: 8192 # OC needs very little RAM
ip: 10.10.0.110/24 ip: 10.10.0.110/24
- name: hz-node-01 - name: hz-node-01
app: incus app: incus
apply_defaults: true # init node: needs storage pool + network apply_defaults: false # OC manages cluster formation + storage/network setup
disk: 150 # extra space for OVN control plane container disk: 150 # extra space for OVN control plane container
ip: 10.10.0.111/24 ip: 10.10.0.111/24

1
incusos/targets/hetzner/proxmox.yaml.example
View File

@ -14,6 +14,7 @@
host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1) host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1)
method: api # api recommended (WireGuard may add SSH latency) method: api # api recommended (WireGuard may add SSH latency)
api_token_id: automation@pve!deploy api_token_id: automation@pve!deploy
api_token_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # token secret (overrides env PROXMOX_TOKEN_SECRET)
node: pve # Proxmox node name (adjust if different) node: pve # Proxmox node name (adjust if different)
storage: local-zfs # VM disk storage (ZFS pool) storage: local-zfs # VM disk storage (ZFS pool)
iso_storage: local # ISO/image storage iso_storage: local # ISO/image storage