Add Hetzner lab guide + target configs for AWX and HAProxy
Validates full end-to-end deployment: OC + 3-node Incus cluster, OVN overlay, HAProxy HA pair, AWX on K3s, Prometheus/Grafana stack. New files: - hetzner/hetzner-lab-guide.md — complete end-to-end deployment guide - incusos/targets/hetzner/awx.yaml — AWX deploy config (hz-node-03, 10.10.0.122) - incusos/targets/hetzner/haproxy.yaml — HAProxy deploy config (VIP 10.10.0.200) Script fixes discovered during Hetzner deployment: - deploy-awx: parse target_remote correctly (was reading target_node key); add RESUME flag support; fix target_node passthrough to incus launch; fix aether_url sed to not strip :8443 port - deploy-observability: fix --remote flag (was --cluster-remote in docs), fix --resume to clearly skip phases 1-5 including node-exporters - hetzner/proxmox-setup: state detection improvements from full run - incusos-proxmox, deploy-haproxy: minor fixes from Hetzner testing Guide covers known Aether v6.4.202 workarounds: - HAProxy ACL bug (OVN LB DNAT-only, source IP not VIP) - AWX cluster binding API bug (DB workaround documented) - OVN SNAT unreliable for net-prod containers (offline install procedure) - node-exporter manual deploy when --resume skips phase 5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
68daf96e61
commit
594467ddf6
|
|
@ -26,8 +26,9 @@ Thumbs.db
|
|||
# Go binaries (flasher-tool)
|
||||
flasher-tool
|
||||
|
||||
# Proxmox connection config (contains credentials)
|
||||
proxmox.yaml
|
||||
# Proxmox connection configs (contain secrets, per-target and default)
|
||||
incusos/proxmox.yaml
|
||||
incusos/targets/*/proxmox.yaml
|
||||
|
||||
# Per-target env files
|
||||
env-*
|
||||
|
|
|
|||
|
|
@ -0,0 +1,907 @@
|
|||
# Hetzner IncusOS Lab — End-to-End Guide
|
||||
|
||||
Full deployment guide for a production-quality IncusOS lab on a Hetzner
|
||||
dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
|
||||
networking, HAProxy load balancing, AWX automation, and observability.
|
||||
|
||||
**Estimated time**: 3-4 hours for a clean run.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
Hetzner dedicated (256 GB RAM, 32+ cores)
|
||||
└─ Proxmox VE 9
|
||||
├─ hz-oc (VMID 910) — Operations Center 0.4.x
|
||||
├─ hz-node-01 (VMID 911) — Incus cluster init + OVN control plane
|
||||
├─ hz-node-02 (VMID 912) — Incus cluster member
|
||||
└─ hz-node-03 (VMID 913) — Incus cluster member
|
||||
└─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
|
||||
├─ ffsdn-haproxy-52-01/02 — HAProxy HA pair
|
||||
├─ nginx-lb-01/02/03 — Test backends
|
||||
├─ monitoring — Prometheus + Grafana
|
||||
├─ node-exp-01/02/03 — Node exporters
|
||||
└─ awx — AWX Operator on K3s
|
||||
```
|
||||
|
||||
**Network topology:**
|
||||
- vmbr0: Public interface (SSH + WireGuard only)
|
||||
- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
|
||||
- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
|
||||
- OVN overlay: 10.10.10.0/24 (containers on net-prod)
|
||||
- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hetzner dedicated server with Proxmox VE 9 installed and configured
|
||||
(see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
|
||||
- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
|
||||
- This repository cloned locally
|
||||
- `env` file populated (see below)
|
||||
- `incusos/targets/hetzner/proxmox.yaml` filled in
|
||||
- Incus client installed on workstation
|
||||
|
||||
### env file
|
||||
|
||||
```bash
|
||||
# incusos-contrib/env
|
||||
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
|
||||
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
|
||||
export AETHER_ADMIN_PASSWORD="admin1234"
|
||||
```
|
||||
|
||||
### proxmox.yaml
|
||||
|
||||
```bash
|
||||
cp incusos/targets/hetzner/proxmox.yaml.example \
|
||||
incusos/targets/hetzner/proxmox.yaml
|
||||
# Fill in: host, node name, token_id, storage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Deploy IncusOS VMs
|
||||
|
||||
```bash
|
||||
source env
|
||||
|
||||
incusos/incusos-proxmox \
|
||||
--proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||
incusos/targets/hetzner/lab-production.yaml
|
||||
```
|
||||
|
||||
**What gets deployed:**
|
||||
|
||||
| VMID | Name | IP | RAM | Cores | Disk |
|
||||
|------|-----------|---------------|-------|-------|------|
|
||||
| 910 | hz-oc | 10.10.0.110 | 8 GB | 4 | 100G |
|
||||
| 911 | hz-node-01| 10.10.0.111 | 80 GB | 16 | 150G |
|
||||
| 912 | hz-node-02| 10.10.0.112 | 80 GB | 16 | 100G |
|
||||
| 913 | hz-node-03| 10.10.0.113 | 80 GB | 16 | 100G |
|
||||
|
||||
**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
|
||||
interrupt. Monitor progress:
|
||||
|
||||
```bash
|
||||
# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
|
||||
source env
|
||||
incusos/helpers/proxmox-screenshot 910 # hz-oc
|
||||
incusos/helpers/proxmox-screenshot 911 # hz-node-01
|
||||
```
|
||||
|
||||
Wait until all VMs are reachable:
|
||||
|
||||
```bash
|
||||
for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
|
||||
echo -n "$ip: "
|
||||
ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Add Incus remotes
|
||||
|
||||
Add the cluster remote pointing at hz-node-01 (the init node):
|
||||
|
||||
```bash
|
||||
incus remote add hz-cluster https://10.10.0.111:8443 \
|
||||
--accept-certificate --auth-type tls
|
||||
```
|
||||
|
||||
> **Note**: The `incus remote add` command prompts for a trust token.
|
||||
> Obtain the token from OC after cluster formation (section 3), or
|
||||
> if using a pre-formed cluster, generate one with:
|
||||
> `incus config trust add --name workstation`
|
||||
|
||||
---
|
||||
|
||||
## 3. Cluster formation via Operations Center
|
||||
|
||||
Open the Operations Center UI:
|
||||
|
||||
```
|
||||
https://10.10.0.110:8443
|
||||
Username: admin
|
||||
Password: admin (first login, then set a new password)
|
||||
```
|
||||
|
||||
### 3.1 Add Incus nodes
|
||||
|
||||
In OC: **Servers → Add Server** for each node:
|
||||
|
||||
| Field | hz-node-01 | hz-node-02 | hz-node-03 |
|
||||
|---------|------------------|------------------|------------------|
|
||||
| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
|
||||
| Trust | via token | via token | via token |
|
||||
|
||||
Each node's trust token is displayed on the IncusOS console on first boot.
|
||||
|
||||
### 3.2 Form the cluster
|
||||
|
||||
In OC: **Clusters → Create Cluster**
|
||||
|
||||
- Name: `hz-cluster`
|
||||
- Init node: `hz-node-01`
|
||||
- Members: `hz-node-02`, `hz-node-03`
|
||||
- App seed: `{}` (empty JSON for manual post-formation setup)
|
||||
|
||||
OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
|
||||
|
||||
Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
|
||||
HAProxy and AWX configuration.
|
||||
|
||||
### 3.3 Verify cluster
|
||||
|
||||
```bash
|
||||
incus cluster list hz-cluster:
|
||||
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. OVN networking
|
||||
|
||||
### 4.1 Create the OVN uplink network
|
||||
|
||||
The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
|
||||
`eth0` or `ens18`. This is specific to IncusOS.
|
||||
|
||||
```bash
|
||||
# Create UPLINK on each node
|
||||
incus network create hz-cluster:UPLINK --type=physical \
|
||||
--target hz-node-01 \
|
||||
config parent=mgmt
|
||||
|
||||
incus network create hz-cluster:UPLINK --type=physical \
|
||||
--target hz-node-02 \
|
||||
config parent=mgmt
|
||||
|
||||
incus network create hz-cluster:UPLINK --type=physical \
|
||||
--target hz-node-03 \
|
||||
config parent=mgmt
|
||||
|
||||
# Finalize UPLINK on cluster level
|
||||
incus network create hz-cluster:UPLINK \
|
||||
config ipv4.address=none \
|
||||
config ipv6.address=none
|
||||
```
|
||||
|
||||
### 4.2 Deploy OVN central
|
||||
|
||||
OVN central runs as a privileged container on hz-node-01:
|
||||
|
||||
```bash
|
||||
incus launch images:debian/12 hz-cluster:ovn-central \
|
||||
--vm=false \
|
||||
--target hz-node-01 \
|
||||
-c security.privileged=true \
|
||||
-d root,size=20GiB
|
||||
|
||||
# Wait for agent
|
||||
sleep 15
|
||||
|
||||
# Install ovn-central
|
||||
incus exec hz-cluster:ovn-central -- bash -c "
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq ovn-central
|
||||
systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
||||
systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
||||
"
|
||||
```
|
||||
|
||||
### 4.3 Expose OVN NB/SB ports to cluster
|
||||
|
||||
Create proxy devices so cluster nodes can reach OVN:
|
||||
|
||||
```bash
|
||||
# Get ovn-central IP
|
||||
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
|
||||
|
||||
incus config device add hz-cluster:ovn-central ovn-nb proxy \
|
||||
listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
|
||||
bind=host
|
||||
incus config device add hz-cluster:ovn-central ovn-sb proxy \
|
||||
listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
|
||||
bind=host
|
||||
```
|
||||
|
||||
### 4.4 Create OVN overlay network
|
||||
|
||||
```bash
|
||||
# Use hz-node-01 IP as OVN NB/SB server
|
||||
NODE01_IP="10.10.0.111"
|
||||
|
||||
incus network create hz-cluster:net-prod \
|
||||
--type=ovn \
|
||||
config network=UPLINK \
|
||||
config ipv4.address=10.10.10.1/24 \
|
||||
config ipv4.nat=true \
|
||||
config ipv6.address=none \
|
||||
config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
|
||||
config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
|
||||
```
|
||||
|
||||
Wait for the network to reach **Created** state:
|
||||
|
||||
```bash
|
||||
incus network info hz-cluster:net-prod
|
||||
```
|
||||
|
||||
### 4.5 Configure external IP range (OVN VIPs)
|
||||
|
||||
```bash
|
||||
# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
|
||||
# These are allocated on the Proxmox host's vmbr1 bridge
|
||||
# No additional configuration needed — OVN will ARP for these IPs
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
incus network list hz-cluster:
|
||||
# net-prod should show type: ovn, state: Created
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Deploy Aether
|
||||
|
||||
### 5.1 Deploy the Aether VM
|
||||
|
||||
Use the macvlan method to give Aether a direct vmbr1 IP:
|
||||
|
||||
```bash
|
||||
incus launch images:ubuntu/22.04 hz-cluster:aether \
|
||||
--vm \
|
||||
--target hz-node-01 \
|
||||
-d root,size=40GiB \
|
||||
-c limits.cpu=4 \
|
||||
-c limits.memory=8GiB \
|
||||
-d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
|
||||
```
|
||||
|
||||
> **Alternative**: use Aether golden image if already built on the cluster.
|
||||
> Check `incus image list hz-cluster: | grep aether`.
|
||||
|
||||
Wait for the VM, then SSH in:
|
||||
|
||||
```bash
|
||||
ssh ubuntu@10.10.0.120
|
||||
```
|
||||
|
||||
Follow the [Aether installation guide](../notes/aether-guide.md) for the
|
||||
full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
|
||||
|
||||
### 5.2 Configure Aether cluster
|
||||
|
||||
In Aether UI (`https://10.10.0.120:8443`):
|
||||
|
||||
1. **Settings → Clusters → Add Cluster**
|
||||
- Name: hz-cluster
|
||||
- Endpoint: `https://10.10.0.111:8443`
|
||||
- Authenticate (generates a trust token, add it to the cluster)
|
||||
|
||||
2. **Settings → Operations Centers → Add OC**
|
||||
- URL: `https://10.10.0.110:8443`
|
||||
|
||||
3. Verify hz-cluster instances sync in Aether dashboard.
|
||||
|
||||
Aether default credentials: `admin` / `admin` (change on first login).
|
||||
|
||||
For Aether password, set in `env`:
|
||||
|
||||
```bash
|
||||
export AETHER_ADMIN_PASSWORD="admin1234"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. HAProxy load balancing
|
||||
|
||||
### 6.1 Create config file
|
||||
|
||||
Save as `incusos/targets/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
|
||||
|
||||
```yaml
|
||||
haproxy:
|
||||
cluster_remote: hz-cluster
|
||||
cluster_id: 52 # from OC cluster listing
|
||||
ovn_network: net-prod
|
||||
vip: 10.10.0.200
|
||||
haproxy_01_ip: 10.10.10.50
|
||||
haproxy_02_ip: 10.10.10.51
|
||||
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
||||
service_name: web-test
|
||||
image_version: "1.0.0"
|
||||
aether_url: https://10.10.0.120:8443
|
||||
```
|
||||
|
||||
> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
|
||||
> It's the numeric ID assigned when the cluster was created.
|
||||
|
||||
### 6.2 Build HAProxy image
|
||||
|
||||
Before deploying infrastructure, build and push the HAProxy image:
|
||||
|
||||
In Aether UI → **HAProxy → Images → Build**:
|
||||
- Version: `1.0.0`
|
||||
- Cluster: `hz-cluster`
|
||||
- Click **Build** and wait 5-10 minutes
|
||||
|
||||
After build, push to hz-cluster:
|
||||
- Click **Push** → select `hz-cluster`
|
||||
- Click **Set Current**
|
||||
|
||||
Alternatively, use the script (requires the UI steps above for first build):
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
|
||||
```
|
||||
|
||||
### 6.3 Full deploy
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
|
||||
```
|
||||
|
||||
This:
|
||||
1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
|
||||
2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
|
||||
3. Creates the `web-test` service (HTTP, round-robin, health checks)
|
||||
|
||||
On subsequent runs (if backends already exist):
|
||||
|
||||
```bash
|
||||
incusos/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
|
||||
--skip-backends --skip-image
|
||||
```
|
||||
|
||||
### 6.4 Verify
|
||||
|
||||
```bash
|
||||
curl http://10.10.0.200/
|
||||
# Should show "Backend: nginx-lb-0x"
|
||||
|
||||
# Test distribution
|
||||
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. AWX automation
|
||||
|
||||
### 7.1 Create config file
|
||||
|
||||
Save as `incusos/targets/hetzner/awx.yaml`:
|
||||
|
||||
```yaml
|
||||
awx:
|
||||
vm_name: awx
|
||||
target_remote: hz-cluster
|
||||
target_node: hz-node-03 # specific node within cluster
|
||||
ip: 10.10.0.122/24
|
||||
gateway: 10.10.0.1
|
||||
dns: 10.10.0.1
|
||||
cpu: 4
|
||||
memory: 8GiB
|
||||
disk: 40GiB
|
||||
aether_url: https://10.10.0.120:8443
|
||||
aether_cluster_id: 52
|
||||
```
|
||||
|
||||
> **target_node vs target_remote**: `target_remote` is the Incus remote
|
||||
> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
|
||||
> member to deploy on (e.g. `hz-node-03`). Both are required for
|
||||
> cluster deployments.
|
||||
|
||||
### 7.2 Deploy
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy
|
||||
```
|
||||
|
||||
This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
|
||||
AWX will be at `http://10.10.0.122:30080`.
|
||||
|
||||
On partial failure (e.g. network issue mid-deploy), resume:
|
||||
|
||||
```bash
|
||||
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --deploy --resume
|
||||
# Skips VM creation and K3s install phases, jumps to AWX Operator phase
|
||||
```
|
||||
|
||||
If you get `permission denied` on temp files from a previous aborted run:
|
||||
|
||||
```bash
|
||||
incus exec hz-cluster:awx -- rm -f \
|
||||
/tmp/awx-operator-kustomization.yaml \
|
||||
/tmp/awx-kustomization.yaml \
|
||||
/tmp/awx-instance.yaml
|
||||
```
|
||||
|
||||
### 7.3 Configure AWX
|
||||
|
||||
> **Wait for the awx-task pod before running --configure.** The AWX web
|
||||
> deployment becomes ready first (~3 min), but `--configure` needs the
|
||||
> task pod. Check it:
|
||||
> ```bash
|
||||
> incus exec hz-cluster:awx -- kubectl -n awx wait \
|
||||
> --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
|
||||
> --timeout=300s
|
||||
> ```
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --configure
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Project: `incus-contrib` (linked to your git repo)
|
||||
- Inventory: `incus-lab`
|
||||
- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
|
||||
|
||||
Get AWX admin password:
|
||||
|
||||
```bash
|
||||
incusos/deploy-awx -c /tmp/hz-awx.yaml --status
|
||||
```
|
||||
|
||||
### 7.4 Join Aether
|
||||
|
||||
Generate an AWX API token for Aether:
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-awx -c incusos/targets/hetzner/awx.yaml --join-aether
|
||||
# Prints the PAT token and template IDs
|
||||
```
|
||||
|
||||
In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
|
||||
- Name: `lab-awx`
|
||||
- URL: `http://10.10.0.122:30080`
|
||||
- Token: (paste token from `--join-aether` output)
|
||||
- Verify SSL: unchecked (HTTP endpoint)
|
||||
|
||||
Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
|
||||
|
||||
### 7.5 Bind AWX to cluster
|
||||
|
||||
**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
|
||||
endpoint returns `400 "Invalid cluster ID"` due to a bug.
|
||||
|
||||
**Workaround** (direct DB update):
|
||||
|
||||
```bash
|
||||
# First confirm the AWX endpoint ID (assigned by the database):
|
||||
incus exec hz-cluster:aether -- bash -c \
|
||||
"su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
|
||||
|
||||
# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
|
||||
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
|
||||
UPDATE incus_ovn_clusters
|
||||
SET awx_endpoint_id=2,
|
||||
awx_post_deploy_template_id=9,
|
||||
awx_decommission_template_id=10,
|
||||
awx_job_timeout_seconds=600
|
||||
WHERE cluster_id=52;
|
||||
\\\"\""
|
||||
```
|
||||
|
||||
> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
|
||||
> `postgres` system user. The endpoint ID is auto-assigned — always check
|
||||
> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
|
||||
|
||||
> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
|
||||
> to see if the endpoint has been fixed. If it returns a valid schema,
|
||||
> use the API instead of the DB workaround.
|
||||
|
||||
Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
|
||||
the AWX endpoint, post-deploy, and decommission templates.
|
||||
|
||||
---
|
||||
|
||||
## 8. Observability stack
|
||||
|
||||
### 8.1 Deploy
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-observability \
|
||||
--remote hz-cluster \
|
||||
--monitoring-target hz-node-01 \
|
||||
--monitoring-ip 10.10.10.70 \
|
||||
--forward-vip 10.10.0.201 \
|
||||
--deploy
|
||||
```
|
||||
|
||||
> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
|
||||
> Operations Center (hz-oc) runs OC, not standard Incus — its
|
||||
> `/1.0/instances` endpoint returns 404 and will fail the node
|
||||
> discovery phase.
|
||||
|
||||
> **Grafana install may fail if net-prod containers lack internet.**
|
||||
> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
|
||||
> after container launch but can become unreliable. If phase 4 fails:
|
||||
>
|
||||
> ```bash
|
||||
> # Download Grafana locally and push it in
|
||||
> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
|
||||
> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
|
||||
> incus exec hz-cluster:monitoring -- bash -c "
|
||||
> export DEBIAN_FRONTEND=noninteractive
|
||||
> dpkg -i /tmp/grafana.deb; apt-get -f install -y
|
||||
> mkdir -p /etc/grafana/provisioning/datasources
|
||||
> cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
|
||||
> apiVersion: 1
|
||||
> datasources:
|
||||
> - name: Prometheus
|
||||
> type: prometheus
|
||||
> uid: prometheus
|
||||
> access: proxy
|
||||
> url: http://localhost:9090
|
||||
> isDefault: true
|
||||
> - name: Loki
|
||||
> type: loki
|
||||
> uid: loki
|
||||
> access: proxy
|
||||
> url: http://localhost:3100
|
||||
> EOF
|
||||
> systemctl daemon-reload
|
||||
> systemctl enable grafana-server
|
||||
> systemctl start grafana-server
|
||||
> "
|
||||
> ```
|
||||
>
|
||||
> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
|
||||
> network forward, and dashboards).
|
||||
|
||||
> **Node-exporter containers are skipped by `--resume`.** If internet
|
||||
> was unavailable during phase 5, deploy them manually:
|
||||
>
|
||||
> ```bash
|
||||
> # Download node_exporter binary locally
|
||||
> curl -sL -o /tmp/ne.tar.gz \
|
||||
> https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
|
||||
> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
|
||||
>
|
||||
> for i in 1 2 3; do
|
||||
> name="node-exp-0${i}"
|
||||
> ip="10.10.10.7${i}"
|
||||
> node="hz-node-0${i}"
|
||||
>
|
||||
> incus launch images:alpine/3.20 hz-cluster:${name} \
|
||||
> -n net-prod --target ${node} \
|
||||
> -c limits.memory=128MiB -c security.privileged=true
|
||||
> incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
|
||||
> sleep 5
|
||||
>
|
||||
> for dev in proc sys root; do
|
||||
> src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
|
||||
> incus config device add hz-cluster:${name} host-${dev} disk \
|
||||
> source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
|
||||
> incus config device add hz-cluster:${name} host-${dev} disk \
|
||||
> source=/ path=/host/root readonly=true 2>/dev/null || true
|
||||
> done
|
||||
>
|
||||
> incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
|
||||
> hz-cluster:${name}/usr/bin/node_exporter
|
||||
>
|
||||
> incus exec hz-cluster:${name} -- sh -c "
|
||||
> chmod +x /usr/bin/node_exporter
|
||||
> cat > /etc/init.d/node-exporter << 'INIT'
|
||||
> #!/sbin/openrc-run
|
||||
> description=\"Node Exporter\"
|
||||
> command=/usr/bin/node_exporter
|
||||
> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
|
||||
> command_background=true
|
||||
> pidfile=/run/node-exporter.pid
|
||||
> INIT
|
||||
> chmod +x /etc/init.d/node-exporter
|
||||
> rc-update add node-exporter default
|
||||
> rc-service node-exporter start
|
||||
>
|
||||
> cat > /etc/network/interfaces << NETCFG
|
||||
> auto lo
|
||||
> iface lo inet loopback
|
||||
> auto eth0
|
||||
> iface eth0 inet static
|
||||
> address ${ip}/24
|
||||
> gateway 10.10.10.1
|
||||
> NETCFG
|
||||
> rc-service networking restart
|
||||
> "
|
||||
>
|
||||
> incus config device set hz-cluster:${name} eth0 \
|
||||
> security.acls=monitoring-allow \
|
||||
> security.acls.default.ingress.action=allow \
|
||||
> security.acls.default.egress.action=allow
|
||||
> done
|
||||
> ```
|
||||
|
||||
On partial failure (phases 6-9 only), resume:
|
||||
|
||||
```bash
|
||||
source env
|
||||
incusos/deploy-observability \
|
||||
--remote hz-cluster \
|
||||
--monitoring-target hz-node-01 \
|
||||
--monitoring-ip 10.10.10.70 \
|
||||
--forward-vip 10.10.0.201 \
|
||||
--resume --deploy
|
||||
```
|
||||
|
||||
After deploying, fix the HAProxy prometheus exporter (default stats page
|
||||
doesn't expose `/metrics`):
|
||||
|
||||
```bash
|
||||
for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
|
||||
incus exec hz-cluster:${ct} -- bash -c "
|
||||
sed -i 's/ stats enable/ http-request use-service prometheus-exporter if { path \/metrics }\n stats enable/' /etc/haproxy/haproxy.cfg
|
||||
systemctl reload haproxy
|
||||
"
|
||||
done
|
||||
```
|
||||
|
||||
### 8.2 Access
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|------------|------------------------------|-------------|
|
||||
| Grafana | http://10.10.0.201:3000 | admin/admin |
|
||||
| Prometheus | http://10.10.0.201:9090 | — |
|
||||
|
||||
9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
|
||||
storage, networking, capacity planning, and log explorer.
|
||||
|
||||
Verify all Prometheus targets are up:
|
||||
|
||||
```bash
|
||||
curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
|
||||
import sys,json
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
|
||||
"
|
||||
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. WireGuard access (macOS)
|
||||
|
||||
If you use the macOS WireGuard app from the App Store:
|
||||
|
||||
- The app uses Network Extension (visible as `utun5` in `ifconfig`)
|
||||
- `wg show` always shows empty — this is normal with the App Store app
|
||||
- The app allows only **one active tunnel at a time**
|
||||
|
||||
For a second simultaneous tunnel:
|
||||
|
||||
```bash
|
||||
# Install wg-quick for macOS
|
||||
brew install wireguard-tools
|
||||
|
||||
# Start the second tunnel alongside the App Store one
|
||||
sudo wg-quick up ~/.config/wireguard/hetzner.conf
|
||||
```
|
||||
|
||||
If the server added a new WireGuard peer (new workstation/client):
|
||||
|
||||
```bash
|
||||
# On the Hetzner Proxmox host (via the existing tunnel or public IP)
|
||||
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
|
||||
ssh hetzner-lab "wg-quick save wg0"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Incus remote management
|
||||
|
||||
### Add remotes
|
||||
|
||||
```bash
|
||||
# OC (Operations Center — not standard Incus)
|
||||
# Note: Must be added manually — incus remote add loops prompting for token
|
||||
# OC does not support the standard trust token flow.
|
||||
# Edit ~/.config/incus/config.yml directly:
|
||||
```
|
||||
|
||||
Add to `~/.config/incus/config.yml`:
|
||||
|
||||
```yaml
|
||||
remotes:
|
||||
oc-server:
|
||||
addr: https://10.10.0.110:8443
|
||||
auth_type: tls
|
||||
protocol: incus
|
||||
public: false
|
||||
hz-cluster:
|
||||
addr: https://10.10.0.111:8443
|
||||
auth_type: tls
|
||||
protocol: incus
|
||||
public: false
|
||||
```
|
||||
|
||||
Copy the OC certificate (accept it with curl first):
|
||||
|
||||
```bash
|
||||
# Accept and save OC cert
|
||||
curl -sk https://10.10.0.110:8443 > /dev/null
|
||||
# The cert ends up in your browser cache — use openssl to fetch it
|
||||
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
|
||||
| openssl x509 > ~/.config/incus/servercerts/oc-server.crt
|
||||
```
|
||||
|
||||
> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
|
||||
> fail. OC does not expose the standard `/1.0/instances` endpoint.
|
||||
|
||||
### After redeploy (remote cert changed)
|
||||
|
||||
```bash
|
||||
# Remove old remote and re-add
|
||||
incus remote remove hz-cluster
|
||||
incus remote add hz-cluster https://10.10.0.111:8443 \
|
||||
--accept-certificate --auth-type tls
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Quick reference
|
||||
|
||||
### Access URLs
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|-----------------|----------------------------------|---------------------|
|
||||
| Operations Center | https://10.10.0.110:8443 | admin / (set on first login) |
|
||||
| Aether | https://10.10.0.120:8443 | admin / admin1234 |
|
||||
| AWX | http://10.10.0.122:30080 | admin / (from deploy-awx --status) |
|
||||
| Grafana | http://10.10.0.201:3000 | admin / admin |
|
||||
| Prometheus | http://10.10.0.201:9090 | — |
|
||||
| HAProxy VIP | http://10.10.0.200/ | — |
|
||||
|
||||
### IP allocation
|
||||
|
||||
| Range | Purpose |
|
||||
|-----------------------|------------------------------|
|
||||
| 10.10.0.1 | Proxmox host (vmbr1 gateway) |
|
||||
| 10.10.0.110-113 | IncusOS VMs (OC + nodes) |
|
||||
| 10.10.0.120 | Aether VM (macvlan) |
|
||||
| 10.10.0.122 | AWX VM (macvlan) |
|
||||
| 10.10.0.200+ | OVN external IPs (VIPs) |
|
||||
| 10.10.10.0/24 | OVN overlay (containers) |
|
||||
| 10.10.10.50-51 | HAProxy containers |
|
||||
| 10.10.10.60-62 | nginx test backends |
|
||||
| 10.10.10.70 | monitoring container |
|
||||
| 10.10.10.71-73 | node-exporter containers |
|
||||
| 10.10.99.0/24 | WireGuard tunnel |
|
||||
| 10.10.99.1 | WireGuard server |
|
||||
| 10.10.99.2 | Your workstation |
|
||||
|
||||
### Cluster IDs
|
||||
|
||||
- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
|
||||
your OC UI if different)
|
||||
|
||||
### Useful incus commands
|
||||
|
||||
```bash
|
||||
# List all instances in cluster
|
||||
incus list hz-cluster:
|
||||
|
||||
# List cluster members
|
||||
incus cluster list hz-cluster:
|
||||
|
||||
# List networks
|
||||
incus network list hz-cluster:
|
||||
|
||||
# Exec into a container
|
||||
incus exec hz-cluster:monitoring -- bash
|
||||
|
||||
# Start/stop a container
|
||||
incus start hz-cluster:monitoring
|
||||
incus stop hz-cluster:monitoring --force
|
||||
|
||||
# Check container network
|
||||
incus exec hz-cluster:nginx-lb-01 -- ip a
|
||||
```
|
||||
|
||||
### Teardown
|
||||
|
||||
```bash
|
||||
# Force destroy all VMs (no nice cleanup needed for test lab)
|
||||
source env
|
||||
for vmid in 910 911 912 913; do
|
||||
incusos/helpers/proxmox-api POST \
|
||||
"/nodes/aether-lab/qemu/${vmid}/status/stop" \
|
||||
'{"forceStop":1}' 2>/dev/null || true
|
||||
sleep 3
|
||||
incusos/helpers/proxmox-api DELETE \
|
||||
"/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
|
||||
done
|
||||
|
||||
# Remove stale remotes
|
||||
incus remote remove hz-cluster 2>/dev/null || true
|
||||
incus remote remove oc-server 2>/dev/null || true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known issues and workarounds
|
||||
|
||||
### OC remote add loops (incus remote add oc-server fails)
|
||||
|
||||
`incus remote add` enters a trust token prompt loop for OC because OC
|
||||
does not implement the standard Incus trust token flow.
|
||||
|
||||
**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
|
||||
and save the TLS certificate separately.
|
||||
|
||||
### IncusOS management NIC name is `mgmt`, not `eth0`
|
||||
|
||||
All IncusOS instances use `mgmt` as the management NIC name. Use
|
||||
`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
|
||||
|
||||
### Aether `PUT /api/clusters/{id}/awx-config` returns 400
|
||||
|
||||
Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
|
||||
|
||||
### deploy-observability: don't use --oc-server with OC
|
||||
|
||||
The `--oc-server` flag tries to list instances via the standard Incus
|
||||
API (`/1.0/instances`), which OC does not implement. Omit the flag.
|
||||
|
||||
### OVN SNAT internet access from net-prod containers is unreliable
|
||||
|
||||
Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
|
||||
reach the internet. This works briefly after container launch (via Proxmox
|
||||
NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
|
||||
stale. Connections return `Connection refused` from Fastly CDN.
|
||||
|
||||
**Workaround**: Download packages locally and push them in with
|
||||
`incus file push` instead of relying on apt/apk inside net-prod containers.
|
||||
|
||||
Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
|
||||
Proxmox masquerade and are unaffected.
|
||||
|
||||
### deploy-awx temp file permission errors on resume
|
||||
|
||||
Temp files created as root inside the AWX VM persist between runs.
|
||||
Clean up before resuming:
|
||||
|
||||
```bash
|
||||
incus exec hz-cluster:awx -- rm -f \
|
||||
/tmp/awx-operator-kustomization.yaml \
|
||||
/tmp/awx-kustomization.yaml \
|
||||
/tmp/awx-instance.yaml
|
||||
```
|
||||
|
||||
### aether_url in config files must use key-specific sed
|
||||
|
||||
Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
|
||||
A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
|
||||
already fixed in all scripts in this repo.
|
||||
|
|
@ -34,7 +34,6 @@ WG_PORT="51820"
|
|||
SKIP_REPOS=false
|
||||
SKIP_WIREGUARD=false
|
||||
SKIP_FIREWALL=false
|
||||
SKIP_STORAGE=false
|
||||
|
||||
# Derived (computed after argument parsing)
|
||||
SUBNET_BASE="" # e.g. 10.10.0
|
||||
|
|
@ -72,12 +71,7 @@ DETECT_VMBR1_IP="" # e.g. "10.10.0.1/24" or empty
|
|||
DETECT_IP_FORWARD="" # "1" or "0"
|
||||
DETECT_NAT_ACTIVE="" # "yes" or empty
|
||||
|
||||
# Storage
|
||||
STATE_STORAGE="done" # done | available | needed
|
||||
DETECT_ZPOOLS=""
|
||||
DETECT_PVE_STORAGE=""
|
||||
DETECT_AVAILABLE_DISKS=""
|
||||
DETECT_AVAILABLE_DISK_COUNT="0"
|
||||
|
||||
# WireGuard
|
||||
STATE_WIREGUARD="needed" # done | needed
|
||||
|
|
@ -91,11 +85,16 @@ STATE_FIREWALL="needed" # done | needed
|
|||
DETECT_NFT_RULES="" # "yes" if nftables rules loaded with our pattern
|
||||
DETECT_NFT_FILE="" # "yes" if config file exists
|
||||
|
||||
# DNS (dnsmasq on vmbr1 -- required for IncusOS boot)
|
||||
STATE_DNSMASQ="needed" # done | needed
|
||||
DETECT_DNSMASQ_ACTIVE="" # "yes" if dnsmasq is installed and listening on vmbr1
|
||||
|
||||
# API token
|
||||
STATE_API="needed" # done | partial | needed
|
||||
DETECT_API_ROLE="" # "yes" or empty
|
||||
DETECT_API_POOL="" # "yes" or empty
|
||||
DETECT_API_TOKEN="" # "yes" or empty
|
||||
DETECT_API_ACLS="" # "yes" if /vms ACL is set
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Color and formatting
|
||||
|
|
@ -206,7 +205,6 @@ ${BOLD}DESCRIPTION${RESET}
|
|||
|
||||
- No-subscription repositories and system update
|
||||
- Private bridge (vmbr1) with NAT masquerading
|
||||
- ZFS storage pool on extra disks
|
||||
- WireGuard tunnel for secure remote access
|
||||
- nftables firewall lockdown
|
||||
- API token for incusos-proxmox automation
|
||||
|
|
@ -220,7 +218,6 @@ ${BOLD}OPTIONS${RESET}
|
|||
${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET})
|
||||
${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT})
|
||||
${BOLD}--skip-repos${RESET} Skip repository changes
|
||||
${BOLD}--skip-storage${RESET} Skip storage pool creation
|
||||
${BOLD}--skip-wireguard${RESET} Skip WireGuard setup
|
||||
${BOLD}--skip-firewall${RESET} Skip firewall lockdown
|
||||
${BOLD}-n, --dry-run${RESET} Preview actions without executing
|
||||
|
|
@ -282,10 +279,6 @@ parse_args() {
|
|||
SKIP_REPOS=true
|
||||
shift
|
||||
;;
|
||||
--skip-storage)
|
||||
SKIP_STORAGE=true
|
||||
shift
|
||||
;;
|
||||
--skip-wireguard)
|
||||
SKIP_WIREGUARD=true
|
||||
shift
|
||||
|
|
@ -389,8 +382,6 @@ echo '---BLOCK_DEVICES---'
|
|||
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
|
||||
echo '---ZPOOLS---'
|
||||
zpool list -H 2>/dev/null || true
|
||||
echo '---PVE_STORAGE---'
|
||||
pvesm status 2>/dev/null || true
|
||||
echo '---ENTERPRISE_REPOS---'
|
||||
# Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats
|
||||
enterprise_active='no'
|
||||
|
|
@ -466,6 +457,16 @@ echo '---NFT_FILE---'
|
|||
test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
|
||||
echo '---NFT_RULES---'
|
||||
nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
|
||||
echo '---DNSMASQ---'
|
||||
# dnsmasq installed and listening on port 53 on vmbr1?
|
||||
dnsmasq_ok='no'
|
||||
if command -v dnsmasq >/dev/null 2>&1 && systemctl is-active dnsmasq >/dev/null 2>&1; then
|
||||
# Check it's listening on the bridge IP
|
||||
if ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq; then
|
||||
dnsmasq_ok='yes'
|
||||
fi
|
||||
fi
|
||||
echo "$dnsmasq_ok"
|
||||
echo '---API_ROLE---'
|
||||
pveum role list --output-format json 2>/dev/null \
|
||||
| python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \
|
||||
|
|
@ -478,30 +479,10 @@ echo '---API_TOKEN---'
|
|||
pveum user token list automation@pve --output-format json 2>/dev/null \
|
||||
| python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \
|
||||
|| echo 'no'
|
||||
echo '---AVAILABLE_DISKS---'
|
||||
# Build list of whole-disk names that are part of ZFS pools.
|
||||
# zpool status shows partition names (e.g. nvme0n1p3), so strip the
|
||||
# partition suffix to get the parent disk name.
|
||||
zfs_disks=$(zpool status 2>/dev/null \
|
||||
| grep -oP '(nvme\S+|sd[a-z]+\S*|vd[a-z]+\S*)' \
|
||||
| sed -E 's/p?[0-9]+$//' \
|
||||
| sort -u || true)
|
||||
lsblk -dno NAME,SIZE,TYPE | while read name size type; do
|
||||
[ "$type" != 'disk' ] && continue
|
||||
# Skip disks used by ZFS pools
|
||||
echo "$zfs_disks" | grep -qw "$name" && continue
|
||||
# Skip disks with active mount points
|
||||
lsblk -no MOUNTPOINT "/dev/$name" 2>/dev/null | grep -q '/' && continue
|
||||
# Check for existing partition table / signatures
|
||||
partcount=$(lsblk -no NAME "/dev/$name" 2>/dev/null | wc -l)
|
||||
if [ "$partcount" -gt 1 ]; then
|
||||
# Has partitions -- detect type
|
||||
fstype=$(lsblk -no FSTYPE "/dev/$name" 2>/dev/null | grep -v '^$' | head -1)
|
||||
echo "$name $size partitioned:${fstype:-unknown}"
|
||||
else
|
||||
echo "$name $size clean"
|
||||
fi
|
||||
done
|
||||
echo '---API_ACLS---'
|
||||
pveum acl list --output-format json 2>/dev/null \
|
||||
| python3 -c 'import sys,json; acls=json.load(sys.stdin); paths={a.get("path") for a in acls if "automation@pve" in a.get("ugid","")}; ok=("/vms" in paths and "/sdn" in paths); print("yes" if ok else "no")' 2>/dev/null \
|
||||
|| echo 'no'
|
||||
echo '---END---'
|
||||
DETECT
|
||||
)
|
||||
|
|
@ -545,7 +526,6 @@ _parse_section() {
|
|||
PUBLIC_IP) DETECT_PUBLIC_IP="$content" ;;
|
||||
BLOCK_DEVICES) DETECT_BLOCK_DEVICES="$content" ;;
|
||||
ZPOOLS) DETECT_ZPOOLS="$content" ;;
|
||||
PVE_STORAGE) DETECT_PVE_STORAGE="$content" ;;
|
||||
ENTERPRISE_REPOS)
|
||||
if [[ "$content" == "yes" ]]; then
|
||||
DETECT_ENTERPRISE_ACTIVE="yes"
|
||||
|
|
@ -577,6 +557,9 @@ _parse_section() {
|
|||
;;
|
||||
NFT_FILE) DETECT_NFT_FILE="$content" ;;
|
||||
NFT_RULES) DETECT_NFT_RULES="$content" ;;
|
||||
DNSMASQ)
|
||||
if [[ "$content" == "yes" ]]; then DETECT_DNSMASQ_ACTIVE="yes"; fi
|
||||
;;
|
||||
API_ROLE)
|
||||
if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi
|
||||
;;
|
||||
|
|
@ -586,13 +569,8 @@ _parse_section() {
|
|||
API_TOKEN)
|
||||
if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi
|
||||
;;
|
||||
AVAILABLE_DISKS)
|
||||
DETECT_AVAILABLE_DISKS="$content"
|
||||
if [[ -n "$content" ]]; then
|
||||
DETECT_AVAILABLE_DISK_COUNT=$(echo "$content" | wc -l)
|
||||
else
|
||||
DETECT_AVAILABLE_DISK_COUNT="0"
|
||||
fi
|
||||
API_ACLS)
|
||||
if [[ "$content" == "yes" ]]; then DETECT_API_ACLS="yes"; fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
|
@ -625,21 +603,6 @@ _compute_states() {
|
|||
STATE_BRIDGE="needed"
|
||||
fi
|
||||
|
||||
# Storage: check if any ZFS pool is registered as PVE storage
|
||||
local zfs_in_pve
|
||||
zfs_in_pve=$(echo "$DETECT_PVE_STORAGE" | grep -c "zfspool" || true)
|
||||
if [[ "$zfs_in_pve" -gt 0 ]]; then
|
||||
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
|
||||
STATE_STORAGE="available" # has pool but also unused disks
|
||||
else
|
||||
STATE_STORAGE="done"
|
||||
fi
|
||||
elif [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
|
||||
STATE_STORAGE="needed"
|
||||
else
|
||||
STATE_STORAGE="done" # no pool, but no disks available either
|
||||
fi
|
||||
|
||||
# WireGuard
|
||||
if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
|
||||
STATE_WIREGUARD="done"
|
||||
|
|
@ -655,13 +618,20 @@ _compute_states() {
|
|||
fi
|
||||
|
||||
# API token
|
||||
if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]]; then
|
||||
if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]] && [[ "$DETECT_API_ACLS" == "yes" ]]; then
|
||||
STATE_API="done"
|
||||
elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
|
||||
STATE_API="partial"
|
||||
else
|
||||
STATE_API="needed"
|
||||
fi
|
||||
|
||||
# dnsmasq (DNS for VMs on vmbr1 -- required for IncusOS boot)
|
||||
if [[ "$DETECT_DNSMASQ_ACTIVE" == "yes" ]]; then
|
||||
STATE_DNSMASQ="done"
|
||||
else
|
||||
STATE_DNSMASQ="needed"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -710,12 +680,10 @@ show_status() {
|
|||
|
||||
local effective_repos="$STATE_REPOS"
|
||||
local effective_update="$STATE_UPDATE"
|
||||
local effective_storage="$STATE_STORAGE"
|
||||
local effective_wg="$STATE_WIREGUARD"
|
||||
local effective_fw="$STATE_FIREWALL"
|
||||
|
||||
if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi
|
||||
if [[ "$SKIP_STORAGE" == true ]]; then effective_storage="skipped"; fi
|
||||
if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi
|
||||
if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi
|
||||
|
||||
|
|
@ -748,14 +716,6 @@ show_status() {
|
|||
needed) echo " vmbr1 not yet created" ;;
|
||||
esac
|
||||
|
||||
printf " %-20s %b" "Storage" "$(status_icon "$effective_storage")"
|
||||
case "$effective_storage" in
|
||||
done) echo " system pool registered, no extra disks" ;;
|
||||
available) echo " system pool registered, ${DETECT_AVAILABLE_DISK_COUNT} extra disk(s) available" ;;
|
||||
needed) echo " ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s), no pool created" ;;
|
||||
skipped) echo "" ;;
|
||||
esac
|
||||
|
||||
printf " %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
|
||||
case "$effective_wg" in
|
||||
done) echo " wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
|
||||
|
|
@ -774,9 +734,9 @@ show_status() {
|
|||
done) echo " nftables rules active" ;;
|
||||
needed)
|
||||
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
|
||||
echo " config file exists but rules not loaded"
|
||||
echo " config written, apply manually (see firewall step)"
|
||||
else
|
||||
echo " no rules configured"
|
||||
echo " config not yet written"
|
||||
fi
|
||||
;;
|
||||
skipped) echo "" ;;
|
||||
|
|
@ -795,6 +755,12 @@ show_status() {
|
|||
needed) echo " not configured" ;;
|
||||
esac
|
||||
|
||||
printf " %-20s %b" "DNS (dnsmasq)" "$(status_icon "$STATE_DNSMASQ")"
|
||||
case "$STATE_DNSMASQ" in
|
||||
done) echo " dnsmasq active, forwarding DNS on vmbr1" ;;
|
||||
needed) echo " not installed (required for IncusOS boot)" ;;
|
||||
esac
|
||||
|
||||
echo ""
|
||||
|
||||
# Count pending items
|
||||
|
|
@ -802,10 +768,10 @@ show_status() {
|
|||
if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$effective_storage" == "needed" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi
|
||||
if [[ "$STATE_DNSMASQ" == "needed" ]]; then pending=$((pending + 1)); fi
|
||||
|
||||
if [[ "$pending" -eq 0 ]]; then
|
||||
success "All configured -- nothing to do"
|
||||
|
|
@ -1001,126 +967,46 @@ iface vmbr1 inet static
|
|||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Apply: storage
|
||||
# Apply: dnsmasq (DNS for VMs on vmbr1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
apply_storage() {
|
||||
if [[ "$SKIP_STORAGE" == true ]] || [[ "$STATE_STORAGE" == "done" ]]; then
|
||||
apply_dnsmasq() {
|
||||
if [[ "$STATE_DNSMASQ" == "done" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
step "ZFS storage pool"
|
||||
step "DNS (dnsmasq on vmbr1)"
|
||||
|
||||
if [[ "$STATE_STORAGE" == "available" ]]; then
|
||||
info "ZFS pool already registered with Proxmox"
|
||||
info "${DETECT_AVAILABLE_DISK_COUNT} additional unused disk(s) available:"
|
||||
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
|
||||
if [[ -n "$line" ]]; then echo " /dev/$line"; fi
|
||||
done
|
||||
if ! confirm "Create an additional storage pool?"; then
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
info "Will: install dnsmasq, configure it to forward DNS queries from VMs on vmbr1"
|
||||
info " Interface: vmbr1 (${SUBNET_GW})"
|
||||
info " Upstream: 1.1.1.1, 8.8.8.8"
|
||||
info " Required for IncusOS boot (downloads updates from images.linuxcontainers.org)"
|
||||
|
||||
if [[ "$STATE_STORAGE" == "needed" ]]; then
|
||||
# Check if a zpool named local-zfs already exists but isn't in PVE
|
||||
local existing_pool
|
||||
existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true)
|
||||
if [[ -n "$existing_pool" ]]; then
|
||||
info "ZFS pool 'local-zfs' exists but not registered with Proxmox"
|
||||
if confirm "Register local-zfs with Proxmox?"; then
|
||||
if ! dry_run_guard "pvesm add zfspool local-zfs"; then
|
||||
remote "pvesm add zfspool local-zfs -pool local-zfs"
|
||||
remote "pvesm set local-zfs -content images,rootdir"
|
||||
success "local-zfs registered"
|
||||
fi
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -eq 0 ]]; then
|
||||
info "No unused disks found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
info "Available disks:"
|
||||
local has_partitioned=false
|
||||
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
|
||||
if [[ -n "$line" ]]; then
|
||||
local dname dsize dstate
|
||||
dname=$(echo "$line" | awk '{print $1}')
|
||||
dsize=$(echo "$line" | awk '{print $2}')
|
||||
dstate=$(echo "$line" | awk '{print $3}')
|
||||
if [[ "$dstate" == partitioned:* ]]; then
|
||||
local fsinfo="${dstate#partitioned:}"
|
||||
echo -e " /dev/${dname} ${dsize} ${YELLOW}has existing partitions (${fsinfo})${RESET}"
|
||||
else
|
||||
echo " /dev/${dname} ${dsize} clean"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# Check for partitioned disks that need wiping
|
||||
local partitioned_disks
|
||||
partitioned_disks=$(echo "$DETECT_AVAILABLE_DISKS" | awk '/partitioned:/{print "/dev/"$1}' | tr '\n' ' ')
|
||||
if [[ -n "${partitioned_disks// /}" ]]; then
|
||||
has_partitioned=true
|
||||
warn "Disk(s) with existing partitions must be wiped before use in a ZFS pool"
|
||||
warn "Affected: ${partitioned_disks}"
|
||||
fi
|
||||
|
||||
local disk_count="$DETECT_AVAILABLE_DISK_COUNT"
|
||||
local disk_names
|
||||
disk_names=$(echo "$DETECT_AVAILABLE_DISKS" | awk 'NF{print "/dev/"$1}' | tr '\n' ' ')
|
||||
|
||||
local pool_type
|
||||
if [[ "$disk_count" -eq 1 ]]; then
|
||||
pool_type="single"
|
||||
info "Recommendation: single disk (no redundancy)"
|
||||
elif [[ "$disk_count" -eq 2 ]]; then
|
||||
pool_type="mirror"
|
||||
info "Recommendation: mirror (2 disks, full redundancy)"
|
||||
else
|
||||
pool_type="raidz1"
|
||||
info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)"
|
||||
fi
|
||||
|
||||
local pool_name
|
||||
pool_name=$(prompt_value "Pool name" "vm-storage")
|
||||
|
||||
if [[ "$has_partitioned" == true ]]; then
|
||||
warn "This will wipe partition tables on: ${partitioned_disks}"
|
||||
fi
|
||||
|
||||
if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then
|
||||
if ! confirm "Install and configure dnsmasq?"; then
|
||||
info "Skipped"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then
|
||||
return 0
|
||||
fi
|
||||
dry_run_guard
|
||||
|
||||
# Wipe partition tables on disks that have them
|
||||
for disk in $partitioned_disks; do
|
||||
info "Wiping partition table on ${disk}..."
|
||||
remote "wipefs -a ${disk}"
|
||||
remote "sgdisk -Z ${disk} 2>/dev/null || true"
|
||||
done
|
||||
remote "apt-get install -y dnsmasq"
|
||||
|
||||
local zpool_cmd="zpool create -f ${pool_name}"
|
||||
case "$pool_type" in
|
||||
mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;;
|
||||
raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;;
|
||||
single) zpool_cmd="${zpool_cmd} ${disk_names}" ;;
|
||||
esac
|
||||
local dnsmasq_conf
|
||||
dnsmasq_conf="# DNS for VMs on vmbr1 (${SUBNET})
|
||||
# Generated by proxmox-setup -- do not edit manually
|
||||
interface=vmbr1
|
||||
bind-interfaces
|
||||
no-resolv
|
||||
server=1.1.1.1
|
||||
server=8.8.8.8
|
||||
domain-needed
|
||||
bogus-priv"
|
||||
|
||||
remote "$zpool_cmd"
|
||||
remote "pvesm add zfspool ${pool_name} -pool ${pool_name}"
|
||||
remote "pvesm set ${pool_name} -content images,rootdir"
|
||||
remote_write "/etc/dnsmasq.d/vmbr1.conf" "$dnsmasq_conf"
|
||||
remote "systemctl enable dnsmasq && systemctl restart dnsmasq"
|
||||
remote "ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq && echo 'OK' || echo 'WARN: dnsmasq not listening on port 53'"
|
||||
|
||||
success "ZFS pool '${pool_name}' created and registered"
|
||||
success "dnsmasq configured on vmbr1"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1144,6 +1030,21 @@ apply_wireguard() {
|
|||
return 0
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
info "(dry-run) Client config will look like:"
|
||||
echo ""
|
||||
echo " [Interface]"
|
||||
echo " PrivateKey = <generated on setup>"
|
||||
echo " Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}"
|
||||
echo ""
|
||||
echo " [Peer]"
|
||||
echo " PublicKey = <server public key>"
|
||||
echo " Endpoint = ${DETECT_PUBLIC_IP:-<server-public-ip>}:${WG_PORT}"
|
||||
echo " AllowedIPs = 10.10.0.0/16"
|
||||
echo " PersistentKeepalive = 25"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
|
||||
return 0
|
||||
fi
|
||||
|
|
@ -1206,6 +1107,22 @@ AllowedIPs = ${WG_CLIENT_IP}/32"
|
|||
echo " PersistentKeepalive = 25"
|
||||
echo ""
|
||||
warn "Save the client config now -- private key is not stored anywhere else"
|
||||
echo ""
|
||||
info "Verify WireGuard before applying firewall rules"
|
||||
if confirm "Connect your WireGuard client now and verify the tunnel?"; then
|
||||
info "Checking server-side WireGuard status..."
|
||||
local wg_status
|
||||
wg_status=$(remote_output "wg show wg0 2>/dev/null" || true)
|
||||
if echo "$wg_status" | grep -q "latest handshake"; then
|
||||
success "WireGuard is working -- handshake confirmed"
|
||||
elif echo "$wg_status" | grep -q "peer:"; then
|
||||
warn "Peer configured but no handshake yet -- client connected?"
|
||||
info "Try: ping ${WG_SERVER_IP}"
|
||||
else
|
||||
warn "No peers visible -- check wg0 is up on server"
|
||||
remote_output "systemctl status wg-quick@wg0 --no-pager -l" || true
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1219,31 +1136,6 @@ apply_firewall() {
|
|||
|
||||
step "Firewall lockdown (nftables)"
|
||||
|
||||
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
|
||||
info "Config file exists but rules not loaded"
|
||||
if confirm "Load existing firewall rules from /etc/nftables-hetzner.conf?"; then
|
||||
if ! dry_run_guard "nft -f /etc/nftables-hetzner.conf"; then
|
||||
remote "nft -f /etc/nftables-hetzner.conf"
|
||||
remote "systemctl enable nftables"
|
||||
success "Firewall rules loaded"
|
||||
fi
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only"
|
||||
info "Web UI (8006) and VM traffic only via WireGuard"
|
||||
warn "Make sure WireGuard is working before applying firewall rules!"
|
||||
|
||||
if ! confirm "Apply firewall rules?"; then
|
||||
info "Skipped"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
local nft_rules="#!/usr/sbin/nft -f
|
||||
|
||||
flush ruleset
|
||||
|
|
@ -1295,12 +1187,26 @@ table inet filter {
|
|||
}
|
||||
}"
|
||||
|
||||
if [[ "$DETECT_NFT_FILE" != "yes" ]]; then
|
||||
if dry_run_guard "Write /etc/nftables-hetzner.conf on ${SSH_HOST}"; then
|
||||
info "Rules that would be written to /etc/nftables-hetzner.conf:"
|
||||
echo "$nft_rules" | sed 's/^/ /'
|
||||
echo ""
|
||||
else
|
||||
remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
|
||||
remote "nft -f /etc/nftables-hetzner.conf"
|
||||
remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf"
|
||||
remote "systemctl enable nftables"
|
||||
success "Config written to /etc/nftables-hetzner.conf"
|
||||
fi
|
||||
else
|
||||
info "Config already at /etc/nftables-hetzner.conf (rules not yet loaded)"
|
||||
fi
|
||||
|
||||
success "Firewall applied (SSH + WireGuard only on public interface)"
|
||||
warn "Firewall rules are NOT applied automatically — misfire locks out SSH on Hetzner permanently"
|
||||
info "Once WireGuard is verified, apply manually:"
|
||||
echo ""
|
||||
echo " ssh ${SSH_HOST} nft -f /etc/nftables-hetzner.conf"
|
||||
echo " ssh ${SSH_HOST} systemctl enable nftables"
|
||||
echo " ssh ${SSH_HOST} nft list ruleset # verify"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1319,6 +1225,7 @@ apply_api_token() {
|
|||
[[ "$DETECT_API_ROLE" == "yes" ]] || actions="create IncusOSDeployer role"
|
||||
[[ "$DETECT_API_POOL" == "yes" ]] || actions="${actions:+$actions, }create IncusLab pool"
|
||||
[[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
|
||||
[[ "$DETECT_API_ACLS" == "yes" ]] || actions="${actions:+$actions, }add missing ACLs (/vms, /sdn, storage)"
|
||||
|
||||
info "Will: ${actions}"
|
||||
|
||||
|
|
@ -1335,9 +1242,8 @@ apply_api_token() {
|
|||
if [[ "$DETECT_API_ROLE" != "yes" ]]; then
|
||||
remote "pveum role add IncusOSDeployer -privs \
|
||||
'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
|
||||
VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
|
||||
VM.Monitor VM.Audit VM.Console \
|
||||
Datastore.AllocateSpace Datastore.Audit \
|
||||
VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Audit \
|
||||
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
|
||||
Pool.Allocate Pool.Audit \
|
||||
SDN.Use Sys.Modify'"
|
||||
success "Created IncusOSDeployer role"
|
||||
|
|
@ -1360,8 +1266,22 @@ apply_api_token() {
|
|||
local token_secret
|
||||
token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
|
||||
|
||||
# Assign ACLs
|
||||
success "API token created"
|
||||
echo ""
|
||||
warn "Token secret (save now -- shown only once):"
|
||||
echo ""
|
||||
echo " ${token_secret}"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Assign ACLs (idempotent -- run whenever any ACL is missing).
|
||||
# PVE 9.x requires /sdn for bridge-backed VMs even with SDN.Use in the role.
|
||||
# /vms and /nodes/<node> are required for VM creation and start via API.
|
||||
if [[ "$DETECT_API_ACLS" != "yes" ]]; then
|
||||
remote "pveum acl modify /vms -user automation@pve -role IncusOSDeployer"
|
||||
remote "pveum acl modify /nodes/${DETECT_HOSTNAME} -user automation@pve -role IncusOSDeployer"
|
||||
remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
|
||||
remote "pveum acl modify /sdn -user automation@pve -role IncusOSDeployer"
|
||||
|
||||
local storage_name
|
||||
storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
|
||||
|
|
@ -1370,12 +1290,7 @@ apply_api_token() {
|
|||
fi
|
||||
remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
|
||||
|
||||
success "API token created"
|
||||
echo ""
|
||||
warn "Token secret (save now -- shown only once):"
|
||||
echo ""
|
||||
echo " ${token_secret}"
|
||||
echo ""
|
||||
success "ACLs configured"
|
||||
fi
|
||||
}
|
||||
|
||||
|
|
@ -1406,7 +1321,6 @@ show_summary() {
|
|||
echo ""
|
||||
echo " # Hetzner"
|
||||
echo " HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
|
||||
echo " HETZNER_PROXMOX_ROOT_PASSWORD=<your root password>"
|
||||
|
||||
echo ""
|
||||
echo -e "${BOLD}Next steps:${RESET}"
|
||||
|
|
@ -1457,7 +1371,7 @@ main() {
|
|||
apply_repos
|
||||
apply_update
|
||||
apply_bridge
|
||||
apply_storage
|
||||
apply_dnsmasq
|
||||
apply_wireguard
|
||||
apply_firewall
|
||||
apply_api_token
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ readonly AWX_NAMESPACE="awx"
|
|||
|
||||
VM_NAME="awx"
|
||||
TARGET_REMOTE="oc-node-02"
|
||||
TARGET_NODE="" # Node within cluster for --target; defaults to TARGET_REMOTE if empty
|
||||
AWX_IP="192.168.102.161"
|
||||
AWX_PREFIX="22"
|
||||
AWX_GATEWAY="192.168.100.1"
|
||||
|
|
@ -39,6 +40,7 @@ AETHER_CLUSTER_ID="52"
|
|||
DRY_RUN=false
|
||||
VERBOSE=false
|
||||
QUIET=false
|
||||
RESUME=false
|
||||
CONFIG_FILE=""
|
||||
ACTION=""
|
||||
|
||||
|
|
@ -163,8 +165,10 @@ parse_config() {
|
|||
local val
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && VM_NAME="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && TARGET_REMOTE="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && TARGET_NODE="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
if [[ -n "$val" ]]; then
|
||||
AWX_IP="${val%%/*}"
|
||||
|
|
@ -184,7 +188,7 @@ parse_config() {
|
|||
[[ -n "$val" ]] && GIT_REPO="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && GIT_BRANCH="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && AETHER_URL="$val"
|
||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
|
||||
|
|
@ -375,8 +379,10 @@ action_deploy() {
|
|||
echo
|
||||
|
||||
if vm_exists; then
|
||||
if vm_running; then
|
||||
error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first."
|
||||
if [[ "${RESUME:-false}" == true ]]; then
|
||||
info "AWX VM already exists — resuming from phase 4"
|
||||
elif vm_running; then
|
||||
error "AWX VM already exists and is running. Use --status, --heal, --resume, or --cleanup first."
|
||||
return 1
|
||||
else
|
||||
warn "AWX VM exists but is not running — starting it"
|
||||
|
|
@ -387,9 +393,11 @@ action_deploy() {
|
|||
fi
|
||||
fi
|
||||
|
||||
if [[ "${RESUME:-false}" != true ]]; then
|
||||
phase_create_vm
|
||||
phase_configure_network
|
||||
phase_install_k3s
|
||||
fi
|
||||
phase_deploy_awx_operator
|
||||
phase_deploy_awx_instance
|
||||
phase_verify
|
||||
|
|
@ -410,12 +418,13 @@ action_deploy() {
|
|||
phase_create_vm() {
|
||||
step "Phase 1: Create VM"
|
||||
|
||||
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then
|
||||
local target_node="${TARGET_NODE:-$TARGET_REMOTE}"
|
||||
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${target_node} ..."; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
incus launch images:debian/12 "$(vm_ref)" --vm \
|
||||
--target "$TARGET_REMOTE" \
|
||||
--target "$target_node" \
|
||||
-c "limits.cpu=${AWX_CPU}" \
|
||||
-c "limits.memory=${AWX_MEMORY}" \
|
||||
-d "root,size=${AWX_DISK}"
|
||||
|
|
@ -490,7 +499,7 @@ phase_install_k3s() {
|
|||
vm_exec bash -c "
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq curl python3 2>&1 | tail -1
|
||||
apt-get install -y -qq curl python3 git 2>&1 | tail -1
|
||||
"
|
||||
|
||||
vm_exec bash -c "
|
||||
|
|
@ -1162,6 +1171,7 @@ parse_args() {
|
|||
--cleanup) ACTION="cleanup";;
|
||||
--doctor) ACTION="doctor";;
|
||||
-c|--config) CONFIG_FILE="$2"; shift;;
|
||||
--resume) RESUME=true;;
|
||||
-n|--dry-run) DRY_RUN=true;;
|
||||
-v|--verbose) VERBOSE=true;;
|
||||
-q|--quiet) QUIET=true;;
|
||||
|
|
|
|||
|
|
@ -184,7 +184,7 @@ parse_config() {
|
|||
[[ -n "$val" ]] && SERVICE_NAME="$val"
|
||||
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && IMAGE_VERSION="$val"
|
||||
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
|
||||
[[ -n "$val" ]] && AETHER_URL="$val"
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ OC_SERVER_IP=""
|
|||
DRY_RUN=false
|
||||
VERBOSE=false
|
||||
QUIET=false
|
||||
RESUME=false
|
||||
ACTION=""
|
||||
MANUAL_NODES=""
|
||||
|
||||
|
|
@ -103,6 +104,9 @@ ${BOLD}ACTIONS${RESET}
|
|||
${BOLD}OPTIONS${RESET}
|
||||
-r, --remote NAME Incus remote (default: ${CLUSTER_REMOTE})
|
||||
--nodes NODE1,NODE2 Manual node list (skip auto-discovery)
|
||||
--monitoring-target NODE Node to deploy monitoring container on (default: ${MONITORING_TARGET})
|
||||
--monitoring-ip IP IP for monitoring container (default: ${MONITORING_IP})
|
||||
--forward-vip IP OVN external IP for Grafana/Prometheus forward (default: ${FORWARD_VIP})
|
||||
--oc-server IP Monitor oc-server at this IP (proxy device for 9100)
|
||||
-n, --dry-run Preview actions without executing
|
||||
-v, --verbose Show detailed output
|
||||
|
|
@ -298,16 +302,22 @@ action_deploy() {
|
|||
echo
|
||||
|
||||
if container_exists "monitoring"; then
|
||||
if [[ "$RESUME" == true ]]; then
|
||||
info "Monitoring container already exists — resuming from phase 5b"
|
||||
else
|
||||
error "Monitoring container already exists"
|
||||
error "Use --status to check health, or --cleanup first."
|
||||
error "Use --status to check health, --cleanup first, or --resume to continue."
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$RESUME" != true ]]; then
|
||||
phase_deploy_monitoring
|
||||
phase_install_prometheus
|
||||
phase_install_loki
|
||||
phase_install_grafana
|
||||
phase_deploy_node_exporters
|
||||
fi
|
||||
phase_deploy_oc_server_exporter
|
||||
phase_create_acl
|
||||
phase_configure_haproxy_exporter
|
||||
|
|
@ -1708,7 +1718,11 @@ parse_args() {
|
|||
--doctor) ACTION="doctor";;
|
||||
-r|--remote) CLUSTER_REMOTE="$2"; shift;;
|
||||
--nodes) MANUAL_NODES="$2"; shift;;
|
||||
--monitoring-target) MONITORING_TARGET="$2"; shift;;
|
||||
--monitoring-ip) MONITORING_IP="$2"; shift;;
|
||||
--forward-vip) FORWARD_VIP="$2"; shift;;
|
||||
--oc-server) OC_SERVER_IP="$2"; shift;;
|
||||
--resume) RESUME=true;;
|
||||
-n|--dry-run) DRY_RUN=true;;
|
||||
-v|--verbose) VERBOSE=true;;
|
||||
-q|--quiet) QUIET=true;;
|
||||
|
|
|
|||
|
|
@ -1405,6 +1405,13 @@ json.dump(result, sys.stdout, indent=2)
|
|||
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
|
||||
# api_token_secret in proxmox.yaml overrides the repo-wide env file, useful
|
||||
# for per-target configs where the env file holds a different target's token.
|
||||
val=$(json_get "$px_json" ".api_token_secret" "")
|
||||
if [[ -n "$val" ]]; then
|
||||
export PROXMOX_TOKEN_SECRET="$val"
|
||||
detail "Using api_token_secret from proxmox.yaml"
|
||||
fi
|
||||
|
||||
rm -f "$px_json"
|
||||
}
|
||||
|
|
@ -2281,6 +2288,7 @@ build_vm_description() {
|
|||
# Check if a VM has our management marker in its description
|
||||
pve_vm_is_managed() {
|
||||
local vmid="$1"
|
||||
local encoded_marker="${MANAGED_MARKER//:/%3A}"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
return 1
|
||||
|
|
@ -2296,7 +2304,8 @@ pve_vm_is_managed() {
|
|||
;;
|
||||
esac
|
||||
|
||||
echo "$config_output" | grep -qF "$MANAGED_MARKER"
|
||||
echo "$config_output" | grep -qF "$MANAGED_MARKER" || \
|
||||
echo "$config_output" | grep -qF "$encoded_marker"
|
||||
}
|
||||
|
||||
# Check if a VM still has install media (ide2) attached.
|
||||
|
|
@ -2397,15 +2406,15 @@ preflight_checks() {
|
|||
if [[ "$DRY_RUN" != true ]]; then
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then
|
||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_STORAGE"; then
|
||||
error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
|
||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
|
||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
|
||||
error "Fix: update 'proxmox.storage' in your config file"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then
|
||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_ISO_STORAGE"; then
|
||||
error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
|
||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
|
||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
|
||||
error "Fix: update 'proxmox.iso_storage' in your config file"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -0,0 +1,27 @@
|
|||
# awx.yaml - AWX deployment config for Hetzner lab
|
||||
#
|
||||
# Deploys a Debian 12 VM with K3s + AWX Operator on hz-node-03.
|
||||
# VM uses macvlan parent=mgmt for direct access on the 10.10.0.0/24 network.
|
||||
#
|
||||
# Architecture:
|
||||
# awx (10.10.0.122) - K3s + AWX Operator + AWX instance
|
||||
# AWX exposed on NodePort 30080 (HTTP)
|
||||
#
|
||||
# Usage:
|
||||
# source env
|
||||
# ./incusos/deploy-awx \
|
||||
# --config incusos/targets/hetzner/awx.yaml \
|
||||
# --deploy
|
||||
|
||||
awx:
|
||||
vm_name: awx
|
||||
target_remote: hz-cluster
|
||||
target_node: hz-node-03
|
||||
ip: 10.10.0.122/24
|
||||
gateway: 10.10.0.1
|
||||
dns: 10.10.0.1
|
||||
cpu: 4
|
||||
memory: 8GiB
|
||||
disk: 40GiB
|
||||
aether_url: https://10.10.0.120:8443
|
||||
aether_cluster_id: 52
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
# haproxy.yaml - HAProxy load balancing config for Hetzner lab
|
||||
#
|
||||
# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
|
||||
# with a VIP on the UPLINK external range (10.10.0.200-210).
|
||||
#
|
||||
# Architecture:
|
||||
# ffsdn-haproxy-52-01 (10.10.10.50) - HAProxy instance 1 on net-prod
|
||||
# ffsdn-haproxy-52-02 (10.10.10.51) - HAProxy instance 2 on net-prod
|
||||
# nginx-lb-01/02/03 (10.10.10.60-62) - nginx test backends on net-prod
|
||||
# VIP: 10.10.0.200:80 - OVN external IP for external access
|
||||
#
|
||||
# Usage:
|
||||
# source env
|
||||
# ./incusos/deploy-haproxy \
|
||||
# --config incusos/targets/hetzner/haproxy.yaml \
|
||||
# --deploy
|
||||
|
||||
haproxy:
|
||||
cluster_remote: hz-cluster
|
||||
cluster_id: 52
|
||||
ovn_network: net-prod
|
||||
vip: 10.10.0.200
|
||||
haproxy_01_ip: 10.10.10.50
|
||||
haproxy_02_ip: 10.10.10.51
|
||||
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
||||
service_name: web-test
|
||||
image_version: "1.0.0"
|
||||
aether_url: https://10.10.0.120:8443
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
|
||||
#
|
||||
# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering.
|
||||
# Deploys 1 Operations Center instance and 3 Incus nodes for OC-managed clustering.
|
||||
# Sized for a Hetzner dedicated server with generous resources.
|
||||
#
|
||||
# Architecture:
|
||||
|
|
@ -12,7 +12,7 @@
|
|||
# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
|
||||
# OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
|
||||
#
|
||||
# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads)
|
||||
# RAM budget: ~248 GiB of 256 GiB (8 GiB OC + 3×80 GiB nodes; ~8 GiB reserved for Proxmox host)
|
||||
#
|
||||
# Usage:
|
||||
# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||
|
|
@ -26,8 +26,8 @@
|
|||
# Connection settings from proxmox.yaml in this directory.
|
||||
|
||||
defaults:
|
||||
cores: 8
|
||||
memory: 16384
|
||||
cores: 16
|
||||
memory: 81920 # ~80 GiB per node (3 nodes × 80 = 240 GiB + 8 for OC = 248 GiB total)
|
||||
disk: 100
|
||||
start_vmid: 910
|
||||
|
||||
|
|
@ -36,12 +36,12 @@ vms:
|
|||
app: operations-center
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
memory: 8192 # OC needs very little RAM
|
||||
ip: 10.10.0.110/24
|
||||
|
||||
- name: hz-node-01
|
||||
app: incus
|
||||
apply_defaults: true # init node: needs storage pool + network
|
||||
apply_defaults: false # OC manages cluster formation + storage/network setup
|
||||
disk: 150 # extra space for OVN control plane container
|
||||
ip: 10.10.0.111/24
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@
|
|||
host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1)
|
||||
method: api # api recommended (WireGuard may add SSH latency)
|
||||
api_token_id: automation@pve!deploy
|
||||
api_token_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # token secret (overrides env PROXMOX_TOKEN_SECRET)
|
||||
node: pve # Proxmox node name (adjust if different)
|
||||
storage: local-zfs # VM disk storage (ZFS pool)
|
||||
iso_storage: local # ISO/image storage
|
||||
|
|
|
|||
Loading…
Reference in New Issue