diff --git a/.claude/rules/hetzner-setup.md b/.claude/rules/hetzner-setup.md new file mode 100644 index 0000000..ea0d947 --- /dev/null +++ b/.claude/rules/hetzner-setup.md @@ -0,0 +1,41 @@ +# Hetzner Setup Context + +paths: + - hetzner/* + - incusos/targets/hetzner/* + +## Network Topology + +Hetzner dedicateds have a single public IP with MAC filtering. +VMs cannot use public IPs directly. + +- **vmbr0**: Public interface (SSH + WireGuard only) +- **vmbr1**: Private bridge (10.10.0.0/24) with NAT masquerading to internet +- **wg0**: WireGuard tunnel (10.10.99.0/24) for workstation access +- VMs get IPs in 10.10.0.x range, gateway 10.10.0.1 (Proxmox host) + +### IP Ranges + +| Range | Purpose | +|-------|---------| +| 10.10.0.1 | Proxmox host (on vmbr1) | +| 10.10.0.101-113 | IncusOS VMs | +| 10.10.0.200+ | OVN external IPs (future) | +| 10.10.10.0/24 | OVN overlay (future, inside VMs) | +| 10.10.99.1 | WireGuard server | +| 10.10.99.2+ | WireGuard clients | + +### Key Differences from Beelink Target + +- **No VLAN** -- dedicated private bridge instead +- **API method preferred** -- WireGuard adds latency to SSH sessions +- **ZFS storage** -- local-zfs (ZFS pool) instead of local-lvm (LVM thin) +- **Larger VMs** -- 8 cores, 16 GiB RAM, 100 GiB disk (vs 4C/4G/50G) +- **WireGuard required** -- no direct LAN access to VMs + +### proxmox-setup Script + +`hetzner/proxmox-setup` runs from the workstation over SSH. +All commands executed via `ssh `. +Follows same conventions as other scripts: setup_colors, info/warn/error, +--dry-run, --yes flags. diff --git a/.gitignore b/.gitignore index ae6dff7..4851a3d 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,9 @@ flasher-tool # Proxmox connection config (contains credentials) proxmox.yaml +# Per-target env files +env-* + # Environment secrets env diff --git a/CLAUDE.md b/CLAUDE.md index ea011a6..c893988 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,9 +39,14 @@ incus-contrib/ │ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all) │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with console screenshots +│ ├── targets/ # Per-host configs (beelink/, hetzner/) │ ├── proxmox.yaml # Proxmox connection (gitignored) │ ├── TESTING.md │ └── examples/ # Example seed + Proxmox YAML files +├── hetzner/ # Hetzner dedicated server setup +│ ├── README.md # Overview, prerequisites, quickstart +│ ├── hetzner-setup.md # Step-by-step Proxmox setup guide +│ └── proxmox-setup # Interactive setup helper (runs over SSH) ├── env # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored) └── notes/ # Reference guides (clustering, networking, OC, AWX, etc.) ``` @@ -196,6 +201,7 @@ which files are being edited: | `incusos-immutability.md` | incusos-health, incusos-break-fix | | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | | `lab-infrastructure.md` | incusos-proxmox, lab-test, examples | +| `hetzner-setup.md` | hetzner/, incusos/targets/hetzner/ | For deep reference, see the guides in `notes/`. diff --git a/README.md b/README.md index 299c144..0d79f27 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,10 @@ Scripts for building IncusOS installation media and seed configurations: YAML, and the script handles seed generation, ISO upload, VM creation with correct settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation. +- **`targets/`** -- Per-host Proxmox connection configs and lab definitions. Each + subdirectory (beelink, hetzner) has a `proxmox.yaml.example` and lab YAML files + sized for that host. See [`targets/README.md`](incusos/targets/README.md). + - **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template configuration, and Aether integration. @@ -39,6 +43,18 @@ API (not SSH) for all instance operations — works regardless of network topolo - **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance. Graceful service shutdown and decommission logging (best-effort). +### [`hetzner/`](hetzner/) + +Setup guide and automation for deploying IncusOS labs on a Hetzner dedicated server: + +- **`hetzner-setup.md`** -- Step-by-step guide: Proxmox install, private networking, + WireGuard tunnel, firewall lockdown, and API token creation. +- **`proxmox-setup`** -- Interactive helper script that automates the guide steps + over SSH. Configures repos, bridge, storage, WireGuard, firewall, and API tokens. + +See also [`incusos/targets/`](incusos/targets/) for per-host connection configs +and lab definitions. + ### [`notes/`](notes/) Guides and reference material (roughly in learning order): diff --git a/hetzner/README.md b/hetzner/README.md new file mode 100644 index 0000000..02e0d70 --- /dev/null +++ b/hetzner/README.md @@ -0,0 +1,84 @@ +# Hetzner Dedicated Server Setup + +Turn a Hetzner bare metal dedicated server into a Proxmox host ready +for IncusOS lab deployments. The result is identical to a local Proxmox +setup (like a Beelink mini PC) but with more resources and accessed +via a WireGuard tunnel instead of LAN. + +## Prerequisites + +- A Hetzner dedicated server (server auction recommended) + - Intel CPU with VT-x/VT-d (nested virtualization) + - 32+ cores, 128+ GiB RAM + - 2+ disks (ZFS mirror for system, extra disks for VM storage) +- An SSH key pair on your workstation +- This repository cloned locally + +## Quickstart + +### 1. Install Proxmox on the server + +Order the server, boot the rescue system, and install Proxmox via +`installimage` or ISO. See [hetzner-setup.md](hetzner-setup.md) section 2 +for details. + +### 2. Run the setup helper + +```bash +# Add SSH config entry for the server (see hetzner-setup.md section 5) +# Then run the interactive setup: +hetzner/proxmox-setup --host hetzner-lab --dry-run # Preview +hetzner/proxmox-setup --host hetzner-lab # Execute +``` + +The script configures repositories, private networking, ZFS storage, +WireGuard tunnel, firewall, and API tokens -- everything needed to +deploy VMs. + +### 3. Deploy IncusOS VMs + +```bash +# Copy and edit the connection config +cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml +# Fill in the API token from step 2 + +# Deploy a 3-node cluster +incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ + incusos/targets/hetzner/lab-cluster.yaml +``` + +## What's in this directory + +| File | Description | +|------|-------------| +| [hetzner-setup.md](hetzner-setup.md) | Comprehensive step-by-step guide | +| [proxmox-setup](proxmox-setup) | Interactive helper script (automates the guide) | + +## Network architecture + +``` +Internet + | + | Public IP (e.g. 5.9.x.x) + v +[vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24] + | | + | WireGuard tunnel + | | + [vmbr1: 10.10.0.1/24] Your workstation + | (10.10.99.2) + ┌─────┼─────┐ + | | | + VM-01 VM-02 VM-03 + .101 .102 .103 +``` + +- **vmbr0**: Public interface (SSH + WireGuard only after firewall lockdown) +- **vmbr1**: Private bridge for VMs (10.10.0.0/24, NAT to internet) +- **wg0**: WireGuard tunnel for workstation access (10.10.99.0/24) +- VMs are accessible from your workstation via WireGuard (routes to 10.10.0.0/16) + +## Related files + +- [`incusos/targets/hetzner/`](../incusos/targets/hetzner/) -- Proxmox connection configs and lab definitions +- [`incusos/targets/README.md`](../incusos/targets/README.md) -- Multi-target concept explanation diff --git a/hetzner/hetzner-setup.md b/hetzner/hetzner-setup.md new file mode 100644 index 0000000..f6d7b88 --- /dev/null +++ b/hetzner/hetzner-setup.md @@ -0,0 +1,619 @@ +# Hetzner Dedicated Server: Proxmox Setup Guide + +Complete guide for turning a Hetzner bare metal server into a Proxmox host +ready for IncusOS lab deployments. Each section can be done manually or +automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)). + +--- + +## 1. Server selection + +The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used +dedicated servers at significant discounts. Look for: + +- **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested + virtualization is more proven with Proxmox/IncusOS) +- **Cores**: 32+ (each IncusOS VM wants 4-8 cores) +- **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB) +- **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage) +- **Network**: 1 Gbit/s included, single IPv4 + +Order the server and wait for provisioning (usually minutes for auction +servers). Note the assigned IP address and temporary root password. + +--- + +## 2. Proxmox installation + +Hetzner provides two installation paths: + +### Option A: installimage (recommended) + +Boot the rescue system from the Hetzner Robot panel, then SSH in: + +```bash +ssh root@ +installimage +``` + +Select Proxmox VE from the list. The installer presents an editor for +disk configuration -- set up ZFS RAID1 mirror on two system disks: + +``` +SWRAID 0 +SWRAIDLEVEL 1 +PART /boot ext4 1G +PART lvm pve all +LV pve root ext4 100G +LV pve swap swap 16G +``` + +Reboot after installation completes. + +### Option B: ISO install + +Upload the Proxmox VE ISO via Robot panel → Server → ISO Images, then +boot from it. Follow the standard Proxmox installer. This gives more +control over partitioning but requires KVM console access. + +### Verify + +```bash +ssh root@ +pvesh get /version +# Should show Proxmox VE version and API info +``` + +The web UI is available at `https://:8006` (we'll lock this +down to WireGuard-only in section 9). + +--- + +## 3. Network configuration + +Hetzner dedicateds get a single public IP with MAC filtering -- you cannot +assign additional public IPs to VMs without ordering extra IPs. Instead, +we create a private bridge and NAT. + +### Create the private bridge + +Add `vmbr1` to `/etc/network/interfaces`: + +``` +auto vmbr1 +iface vmbr1 inet static + address 10.10.0.1/24 + bridge-ports none + bridge-stp off + bridge-fd 0 + post-up echo 1 > /proc/sys/net/ipv4/ip_forward + post-up iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE + post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE +``` + +This creates a private bridge where: +- VMs connect to `vmbr1` and get IPs in 10.10.0.0/24 +- The Proxmox host (10.10.0.1) is the default gateway +- NAT masquerading gives VMs internet access through the host's public IP +- IP forwarding is enabled automatically on bridge up + +### Persist IP forwarding + +Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence: + +``` +net.ipv4.ip_forward = 1 +``` + +### Apply + +```bash +ifreload -a # Apply network changes without reboot +sysctl -p /etc/sysctl.d/99-forward.conf +``` + +### Verify + +```bash +ip addr show vmbr1 # Should show 10.10.0.1/24 +cat /proc/sys/net/ipv4/ip_forward # Should show 1 +iptables -t nat -L POSTROUTING -n # Should show MASQUERADE rule +``` + +### Network diagram + +``` +Internet + | + | Public IP (5.9.x.x) + v +[vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24] + | | + | WireGuard tunnel + | | + [vmbr1: 10.10.0.1/24] Your workstation + | (10.10.99.2) + ┌─────┼─────┐ + | | | + VM-01 VM-02 VM-03 + .101 .102 .103 +``` + +--- + +## 4. DNS setup (optional) + +If you have a domain, create an A record pointing to the public IP: + +``` +pve.example.com → 5.9.x.x +``` + +This is convenient but not required -- everything is accessed via +WireGuard using private IPs anyway. If you set a hostname, update +`/etc/hosts`: + +``` +10.10.0.1 pve.example.com pve +``` + +And set the hostname: + +```bash +hostnamectl set-hostname pve +``` + +--- + +## 5. SSH hardening + +### Copy your SSH key + +```bash +ssh-copy-id root@ +``` + +### Disable password authentication + +Edit `/etc/ssh/sshd_config`: + +``` +PermitRootLogin prohibit-password +PasswordAuthentication no +``` + +Restart: + +```bash +systemctl restart sshd +``` + +### Add a colleague's key + +```bash +# From their workstation: +ssh-copy-id root@ + +# Or manually append to /root/.ssh/authorized_keys +``` + +### SSH config for convenient access + +Add to `~/.ssh/config` on your workstation: + +``` +Host hetzner-lab + HostName 5.9.x.x # Public IP (or pve.example.com) + User root + IdentityFile ~/.ssh/id_ed25519 +``` + +After WireGuard is set up (section 8), add a second entry: + +``` +Host hetzner-lab-wg + HostName 10.10.0.1 + User root + IdentityFile ~/.ssh/id_ed25519 +``` + +### Verify + +```bash +ssh hetzner-lab pvesh get /version +``` + +--- + +## 6. Disk and storage setup + +### Identify disks + +```bash +lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE +``` + +Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from +install), 2+ additional disks for VM storage. + +### Check existing pools + +```bash +zpool status # System mirror +pvesm status # Proxmox storage backends +``` + +### Create VM storage pool + +If you have additional disks (e.g. `sda`, `sdb`) not used by the system: + +```bash +# Mirror (2 disks) -- recommended +zpool create local-zfs mirror /dev/sda /dev/sdb + +# RAIDZ1 (3+ disks) -- more space, still redundant +zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc + +# Single disk (no redundancy) +zpool create local-zfs /dev/sda +``` + +Register with Proxmox: + +```bash +pvesm add zfspool local-zfs -pool local-zfs +pvesm set local-zfs -content images,rootdir +``` + +### Verify + +```bash +pvesm status +# Should list local-zfs with type zfspool +``` + +If the system install already created a suitable ZFS pool, skip pool +creation and just register it with Proxmox if not already visible. + +--- + +## 7. Repositories and system update + +### Switch to no-subscription repositories + +The default enterprise repos require a paid subscription. Switch to +community repos: + +```bash +# Disable enterprise repo +sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list + +# Add no-subscription repo +echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \ + > /etc/apt/sources.list.d/pve-no-subscription.list + +# Disable Ceph enterprise repo if present +if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then + sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list +fi +``` + +### Remove subscription nag popup + +```bash +# Standard JS patch for the web UI subscription dialog +sed -Ezi.bak \ + "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \ + /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js +systemctl restart pveproxy +``` + +### Update + +```bash +apt update && apt dist-upgrade -y +``` + +### Verify + +```bash +apt update 2>&1 | tail -5 +# Should show no errors, all repos reachable +``` + +--- + +## 8. WireGuard tunnel + +WireGuard provides secure access to VMs and the Proxmox web UI from +your workstation, without exposing anything on the public interface. + +### Install + +```bash +apt install -y wireguard +``` + +### Generate server keys + +```bash +wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key +chmod 600 /etc/wireguard/server-private.key +``` + +### Generate client keys (on your workstation) + +```bash +wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key +``` + +### Server config + +Create `/etc/wireguard/wg0.conf`: + +```ini +[Interface] +PrivateKey = +Address = 10.10.99.1/24 +ListenPort = 51820 + +# Allow forwarding between WireGuard clients and the private VM bridge +PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT +PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT + +[Peer] +# Workstation +PublicKey = +AllowedIPs = 10.10.99.2/32 +``` + +### Client config (on your workstation) + +Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app): + +```ini +[Interface] +PrivateKey = +Address = 10.10.99.2/24 + +[Peer] +PublicKey = +Endpoint = :51820 +AllowedIPs = 10.10.0.0/16 +PersistentKeepalive = 25 +``` + +**AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet +leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.) +without needing to update the WireGuard config. + +### Enable and start + +On the server: + +```bash +systemctl enable --now wg-quick@wg0 +``` + +On the workstation: + +```bash +# Linux +sudo wg-quick up hetzner-lab + +# macOS (WireGuard app) +# Import hetzner-lab.conf and activate +``` + +### Verify + +```bash +# On the server +wg show + +# From your workstation +ping 10.10.99.1 # WireGuard interface +ping 10.10.0.1 # Private bridge (should work -- routed via WireGuard) +``` + +### Adding more peers + +To add a colleague, generate a new keypair and add a `[Peer]` block to +the server config: + +```ini +[Peer] +# Colleague name +PublicKey = +AllowedIPs = 10.10.99.3/32 +``` + +Then reload: + +```bash +systemctl restart wg-quick@wg0 +``` + +Give them a client config with `Address = 10.10.99.3/24` and the +server's public key. + +--- + +## 9. Firewall lockdown + +After WireGuard is working, lock down the public interface so only SSH +and WireGuard are accessible from the internet. Proxmox web UI and VM +traffic go through the tunnel only. + +### nftables rules + +Create `/etc/nftables-hetzner.conf`: + +``` +#!/usr/sbin/nft -f + +flush ruleset + +table inet filter { + chain input { + type filter hook input priority 0; policy drop; + + # Loopback + iif lo accept + + # Established/related connections + ct state established,related accept + + # ICMP (ping) + ip protocol icmp accept + ip6 nexthdr icmpv6 accept + + # SSH on public interface + iifname "vmbr0" tcp dport 22 accept + + # WireGuard on public interface + iifname "vmbr0" udp dport 51820 accept + + # Allow everything on private bridge and WireGuard + iifname "vmbr1" accept + iifname "wg0" accept + + # Log and drop everything else + log prefix "nft-drop: " limit rate 5/minute counter drop + } + + chain forward { + type filter hook forward priority 0; policy drop; + + # Established/related + ct state established,related accept + + # WireGuard <-> private bridge + iifname "wg0" oifname "vmbr1" accept + iifname "vmbr1" oifname "wg0" accept + + # Private bridge -> internet (NAT) + iifname "vmbr1" oifname "vmbr0" accept + } + + chain output { + type filter hook output priority 0; policy accept; + } +} +``` + +### Apply + +```bash +nft -f /etc/nftables-hetzner.conf +``` + +### Persist across reboots + +```bash +cp /etc/nftables-hetzner.conf /etc/nftables.conf +systemctl enable nftables +``` + +### Verify + +From an external machine (not through WireGuard): + +```bash +# Should work +ssh hetzner-lab echo ok + +# Should time out (port 8006 blocked on public interface) +curl -sk --connect-timeout 5 https://:8006 + +# Through WireGuard -- should work +curl -sk https://10.10.0.1:8006 +``` + +**Warning**: Test SSH access through WireGuard *before* applying firewall +rules. If you lock yourself out, use Hetzner's rescue system to recover. + +--- + +## 10. API token setup + +Create a dedicated API token for `incusos-proxmox` automation. + +### Create role and user + +```bash +# Create role with required privileges +pveum role add IncusOSDeployer -privs \ + "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ + VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ + VM.Monitor VM.Audit VM.Console \ + Datastore.AllocateSpace Datastore.Audit \ + Pool.Allocate Pool.Audit \ + SDN.Use Sys.Modify" + +# Create resource pool +pveum pool add IncusLab -comment "IncusOS lab VMs" + +# Create API token +pveum user token add automation@pve deploy --privsep 0 +# Save the displayed token secret -- it's shown only once! + +# Assign role to user on the pool +pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer + +# Assign role on storage +pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer +pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer +``` + +### Record credentials + +Add to the `env` file at the repo root: + +```bash +# Hetzner +HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx +HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password +``` + +### Verify + +```bash +# Export for incusos-proxmox +export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" + +# Test API access (update host/node in proxmox-api or use curl directly) +curl -sk "https://10.10.0.1:8006/api2/json/version" \ + -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET" +``` + +--- + +## 11. Final verification checklist + +| Check | Command | Expected | +|-------|---------|----------| +| Proxmox version | `pvesh get /version` | PVE 8.x | +| Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 | +| IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 | +| NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 | +| ZFS storage | `pvesm status` | local-zfs available | +| WireGuard | `wg show` | 1+ peer, handshake recent | +| SSH via WG | `ssh hetzner-lab-wg hostname` | Responds | +| Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response | +| API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version | +| Public lockdown | `curl --connect-timeout 5 https://:8006` | Timeout | + +### Test deploy + +```bash +export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" + +# Dry run -- verify correct bridge, IPs, storage +incusos-proxmox --dry-run \ + --proxmox incusos/targets/hetzner/proxmox.yaml \ + incusos/targets/hetzner/lab-cluster.yaml +``` + +Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs` +storage, 8 cores per VM. diff --git a/hetzner/proxmox-setup b/hetzner/proxmox-setup new file mode 100755 index 0000000..77cf2f2 --- /dev/null +++ b/hetzner/proxmox-setup @@ -0,0 +1,975 @@ +#!/usr/bin/env bash +# proxmox-setup - Interactive Proxmox post-install configuration for Hetzner +# +# Configures a fresh Proxmox host with private networking, ZFS storage, +# WireGuard tunnel, firewall lockdown, and API tokens for incusos-proxmox. +# +# Runs from your workstation over SSH. Requires an SSH host alias configured +# in ~/.ssh/config (e.g. "hetzner-lab"). +# +# Usage: proxmox-setup --host [OPTIONS] +# Run 'proxmox-setup --help' for full usage information. + +set -euo pipefail + +readonly VERSION="0.1.0" +readonly SCRIPT_NAME="proxmox-setup" +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- + +SSH_HOST="" +DRY_RUN=false +VERBOSE=false +QUIET=false +YES=false +SUBNET="10.10.0.0/24" +WG_SUBNET="10.10.99.0/24" +WG_PORT="51820" +SKIP_REPOS=false +SKIP_WIREGUARD=false +SKIP_FIREWALL=false +SKIP_STORAGE=false + +# Derived (computed in detect step) +SUBNET_BASE="" # e.g. 10.10.0 +SUBNET_MASK="" # e.g. 24 +SUBNET_GW="" # e.g. 10.10.0.1 +WG_SUBNET_BASE="" # e.g. 10.10.99 +WG_SUBNET_MASK="" # e.g. 24 +WG_SERVER_IP="" # e.g. 10.10.99.1 +WG_CLIENT_IP="" # e.g. 10.10.99.2 +PUBLIC_IP="" +PVE_VERSION="" +HOSTNAME_DETECTED="" +KERNEL_VERSION="" + +# --------------------------------------------------------------------------- +# Color and formatting +# --------------------------------------------------------------------------- + +setup_colors() { + if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then + RED='\033[0;31m' + GREEN='\033[0;32m' + YELLOW='\033[0;33m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + BOLD='\033[1m' + DIM='\033[2m' + RESET='\033[0m' + else + RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET='' + fi +} + +info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; } +success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; } +warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } +error() { echo -e "${RED}[error]${RESET} $*" >&2; } +step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } + +dry_run_guard() { + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) $*" + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# SSH helpers +# --------------------------------------------------------------------------- + +# Run a command on the remote host via SSH +remote() { + ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_HOST" "$@" +} + +# Run a command and capture output +remote_output() { + ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_HOST" "$@" 2>/dev/null +} + +# Write content to a remote file via SSH +remote_write() { + local dest="$1" + local content="$2" + ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_HOST" "cat > '$dest'" <<< "$content" +} + +# Check if remote file exists +remote_file_exists() { + remote "test -f '$1'" 2>/dev/null +} + +# --------------------------------------------------------------------------- +# Interactive helpers +# --------------------------------------------------------------------------- + +confirm() { + local prompt="$1" + if [[ "$YES" == true ]]; then + return 0 + fi + echo -en "${BOLD}${prompt}${RESET} [y/N] " + read -r answer + [[ "$answer" =~ ^[Yy]$ ]] +} + +prompt_value() { + local prompt="$1" + local default="$2" + local result + if [[ "$YES" == true ]]; then + echo "$default" + return + fi + echo -en "${BOLD}${prompt}${RESET} [${default}]: " + read -r result + echo "${result:-$default}" +} + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +usage() { + cat < [OPTIONS] + +${BOLD}DESCRIPTION${RESET} + Configures a fresh Proxmox host over SSH with: + - No-subscription repositories and system update + - Private bridge (vmbr1) with NAT masquerading + - ZFS storage pool on extra disks + - WireGuard tunnel for secure remote access + - nftables firewall lockdown + - API token for incusos-proxmox automation + + Each step prompts for confirmation. Use --yes to skip prompts. + +${BOLD}OPTIONS${RESET} + ${BOLD}--host ALIAS${RESET} SSH host alias (required, e.g. hetzner-lab) + ${BOLD}--subnet CIDR${RESET} Private bridge subnet (default: ${SUBNET}) + ${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET}) + ${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT}) + ${BOLD}--skip-repos${RESET} Skip repository changes + ${BOLD}--skip-storage${RESET} Skip storage pool creation + ${BOLD}--skip-wireguard${RESET} Skip WireGuard setup + ${BOLD}--skip-firewall${RESET} Skip firewall lockdown + ${BOLD}-n, --dry-run${RESET} Preview actions without executing + ${BOLD}-y, --yes${RESET} Skip confirmations (use defaults) + ${BOLD}-v, --verbose${RESET} Show detailed output + ${BOLD}-q, --quiet${RESET} Suppress informational output + ${BOLD}-h, --help${RESET} Show this help message + ${BOLD}-V, --version${RESET} Show version + +${BOLD}EXAMPLES${RESET} + # Preview everything + ${SCRIPT_NAME} --host hetzner-lab --dry-run + + # Full setup with defaults + ${SCRIPT_NAME} --host hetzner-lab --yes + + # Custom subnets, skip firewall + ${SCRIPT_NAME} --host hetzner-lab --subnet 172.16.0.0/24 --skip-firewall + + # Just WireGuard and firewall (repos/storage already done) + ${SCRIPT_NAME} --host hetzner-lab --skip-repos --skip-storage + +${BOLD}PREREQUISITES${RESET} + - SSH access to the Proxmox host (key-based auth recommended) + - Add an entry to ~/.ssh/config: + Host hetzner-lab + HostName + User root + IdentityFile ~/.ssh/id_ed25519 + +${BOLD}SEE ALSO${RESET} + hetzner/hetzner-setup.md Full manual guide + incusos/targets/hetzner/ Target config files +EOF + exit 0 +} + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + --host) + SSH_HOST="${2:?'--host requires an SSH alias'}" + shift 2 + ;; + --subnet) + SUBNET="${2:?'--subnet requires a CIDR'}" + shift 2 + ;; + --wg-subnet) + WG_SUBNET="${2:?'--wg-subnet requires a CIDR'}" + shift 2 + ;; + --wg-port) + WG_PORT="${2:?'--wg-port requires a port number'}" + shift 2 + ;; + --skip-repos) + SKIP_REPOS=true + shift + ;; + --skip-storage) + SKIP_STORAGE=true + shift + ;; + --skip-wireguard) + SKIP_WIREGUARD=true + shift + ;; + --skip-firewall) + SKIP_FIREWALL=true + shift + ;; + -n|--dry-run) + DRY_RUN=true + shift + ;; + -y|--yes) + YES=true + shift + ;; + -v|--verbose) + VERBOSE=true + shift + ;; + -q|--quiet) + QUIET=true + shift + ;; + -h|--help) + usage + ;; + -V|--version) + echo "${SCRIPT_NAME} v${VERSION}" + exit 0 + ;; + -*) + error "Unknown option: $1" + echo "Run '${SCRIPT_NAME} --help' for usage information." >&2 + exit 1 + ;; + *) + error "Unexpected argument: $1" + echo "Run '${SCRIPT_NAME} --help' for usage information." >&2 + exit 1 + ;; + esac + done + + if [[ -z "$SSH_HOST" ]]; then + error "--host is required (SSH alias, e.g. hetzner-lab)" + echo "Run '${SCRIPT_NAME} --help' for usage information." >&2 + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Subnet helpers +# --------------------------------------------------------------------------- + +parse_subnets() { + # Parse private bridge subnet + SUBNET_MASK="${SUBNET##*/}" + local subnet_network="${SUBNET%%/*}" + SUBNET_BASE="${subnet_network%.*}" + SUBNET_GW="${SUBNET_BASE}.1" + + # Parse WireGuard subnet + WG_SUBNET_MASK="${WG_SUBNET##*/}" + local wg_network="${WG_SUBNET%%/*}" + WG_SUBNET_BASE="${wg_network%.*}" + WG_SERVER_IP="${WG_SUBNET_BASE}.1" + WG_CLIENT_IP="${WG_SUBNET_BASE}.2" +} + +# --------------------------------------------------------------------------- +# Step 1: Detect host information +# --------------------------------------------------------------------------- + +step_detect() { + step "Detecting host information" + + # Test SSH connectivity + if ! remote "true" 2>/dev/null; then + error "Cannot connect to '${SSH_HOST}' via SSH" + error "Ensure the host is in ~/.ssh/config and key auth works:" + echo " ssh ${SSH_HOST} hostname" >&2 + exit 1 + fi + success "SSH connection to ${SSH_HOST}" + + # Gather host info + HOSTNAME_DETECTED=$(remote_output "hostname") + KERNEL_VERSION=$(remote_output "uname -r") + PVE_VERSION=$(remote_output "pvesh get /version --output-format json 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"version\"])'" || echo "unknown") + PUBLIC_IP=$(remote_output "hostname -I | awk '{print \$1}'" || echo "unknown") + + info "Hostname: ${HOSTNAME_DETECTED}" + info "Kernel: ${KERNEL_VERSION}" + info "PVE version: ${PVE_VERSION}" + info "Primary IP: ${PUBLIC_IP}" + + # Block devices + echo "" + info "Block devices:" + remote "lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null" | while IFS= read -r line; do + echo " $line" + done + + # ZFS pools + local zpool_output + zpool_output=$(remote_output "zpool list -H 2>/dev/null" || true) + if [[ -n "$zpool_output" ]]; then + echo "" + info "ZFS pools:" + echo "$zpool_output" | while IFS= read -r line; do + echo " $line" + done + fi + + # Existing storage + echo "" + info "Proxmox storage:" + remote "pvesm status 2>/dev/null" | while IFS= read -r line; do + echo " $line" + done + + # Network interfaces + echo "" + info "Network bridges:" + remote "ip -br addr show type bridge 2>/dev/null" | while IFS= read -r line; do + echo " $line" + done + + # WireGuard status + local wg_status + wg_status=$(remote_output "wg show 2>/dev/null" || true) + if [[ -n "$wg_status" ]]; then + echo "" + info "WireGuard active:" + echo "$wg_status" | head -5 | while IFS= read -r line; do + echo " $line" + done + fi + + echo "" +} + +# --------------------------------------------------------------------------- +# Step 2: Repositories +# --------------------------------------------------------------------------- + +step_repos() { + if [[ "$SKIP_REPOS" == true ]]; then + info "Skipping repository configuration (--skip-repos)" + return 0 + fi + + step "Repository configuration" + + # Check current state + local has_enterprise + has_enterprise=$(remote_output "grep -l '^deb.*enterprise' /etc/apt/sources.list.d/*.list 2>/dev/null" || true) + local has_nosub + has_nosub=$(remote_output "test -f /etc/apt/sources.list.d/pve-no-subscription.list && echo yes" || true) + + if [[ -z "$has_enterprise" ]] && [[ "$has_nosub" == "yes" ]]; then + success "Already configured (no-subscription repos active)" + return 0 + fi + + info "Will disable enterprise repos and enable no-subscription repos" + + if ! confirm "Apply repository changes?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "Disable enterprise repos, enable no-subscription"; then + return 0 + fi + + # Disable enterprise repos + remote "sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list 2>/dev/null || true" + remote "test -f /etc/apt/sources.list.d/ceph.list && sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list || true" + + # Add no-subscription repo + remote "echo 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription' > /etc/apt/sources.list.d/pve-no-subscription.list" + + # Remove subscription nag + remote "sed -Ezi.bak \"s/(Ext\\.Msg\\.show\\(\\{[^}]+title: gettext\\('No valid sub)/void\\(\\{ \\/\\/\\1/\" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null || true" + remote "systemctl restart pveproxy 2>/dev/null || true" + + success "Repositories configured" +} + +# --------------------------------------------------------------------------- +# Step 3: System update +# --------------------------------------------------------------------------- + +step_update() { + if [[ "$SKIP_REPOS" == true ]]; then + info "Skipping system update (--skip-repos)" + return 0 + fi + + step "System update" + + if ! confirm "Run apt update && apt dist-upgrade?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "apt update && apt dist-upgrade -y"; then + return 0 + fi + + info "Updating package lists..." + remote "apt update -qq" + info "Upgrading packages (this may take a few minutes)..." + remote "DEBIAN_FRONTEND=noninteractive apt dist-upgrade -y -qq" + success "System updated" +} + +# --------------------------------------------------------------------------- +# Step 4: Private bridge +# --------------------------------------------------------------------------- + +step_bridge() { + step "Private bridge (vmbr1)" + + # Check if vmbr1 already exists + local existing_bridge + existing_bridge=$(remote_output "ip addr show vmbr1 2>/dev/null" || true) + if [[ -n "$existing_bridge" ]]; then + success "vmbr1 already exists" + remote "ip -br addr show vmbr1" | while IFS= read -r line; do + echo " $line" + done + return 0 + fi + + info "Will create vmbr1 with:" + info " Subnet: ${SUBNET}" + info " Gateway: ${SUBNET_GW}/${SUBNET_MASK}" + info " NAT: MASQUERADE to vmbr0" + + local bridge_config + bridge_config=" +auto vmbr1 +iface vmbr1 inet static + address ${SUBNET_GW}/${SUBNET_MASK} + bridge-ports none + bridge-stp off + bridge-fd 0 + post-up echo 1 > /proc/sys/net/ipv4/ip_forward + post-up iptables -t nat -A POSTROUTING -s ${SUBNET} -o vmbr0 -j MASQUERADE + post-down iptables -t nat -D POSTROUTING -s ${SUBNET} -o vmbr0 -j MASQUERADE" + + if [[ "$VERBOSE" == true ]]; then + echo "" + echo "$bridge_config" + echo "" + fi + + if ! confirm "Add vmbr1 to /etc/network/interfaces?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "Create vmbr1 (${SUBNET_GW}/${SUBNET_MASK})"; then + return 0 + fi + + # Append to interfaces file + remote "printf '%s\n' '' >> /etc/network/interfaces" + remote_write "/tmp/vmbr1-config.tmp" "$bridge_config" + remote "cat /tmp/vmbr1-config.tmp >> /etc/network/interfaces && rm -f /tmp/vmbr1-config.tmp" + + # Persist IP forwarding + remote_write "/etc/sysctl.d/99-forward.conf" "net.ipv4.ip_forward = 1" + remote "sysctl -p /etc/sysctl.d/99-forward.conf >/dev/null 2>&1" + + # Apply + remote "ifreload -a 2>/dev/null || ifup vmbr1" + + success "vmbr1 created (${SUBNET_GW}/${SUBNET_MASK})" +} + +# --------------------------------------------------------------------------- +# Step 5: Storage +# --------------------------------------------------------------------------- + +step_storage() { + if [[ "$SKIP_STORAGE" == true ]]; then + info "Skipping storage setup (--skip-storage)" + return 0 + fi + + step "ZFS storage pool" + + # Check if local-zfs already exists in PVE + local existing_zfs + existing_zfs=$(remote_output "pvesm status 2>/dev/null | grep -w local-zfs" || true) + if [[ -n "$existing_zfs" ]]; then + success "local-zfs already registered with Proxmox" + echo " $existing_zfs" + return 0 + fi + + # Check if a zpool named local-zfs already exists + local existing_pool + existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true) + if [[ -n "$existing_pool" ]]; then + info "ZFS pool 'local-zfs' exists but not registered with Proxmox" + if confirm "Register local-zfs with Proxmox?"; then + if dry_run_guard "pvesm add zfspool local-zfs"; then + return 0 + fi + remote "pvesm add zfspool local-zfs -pool local-zfs" + remote "pvesm set local-zfs -content images,rootdir" + success "local-zfs registered" + fi + return 0 + fi + + # Find available disks (not in any zpool, not mounted) + info "Scanning for available disks..." + local available_disks + available_disks=$(remote_output " + used_disks=\$(zpool status 2>/dev/null | grep -oP '/dev/\\S+' | sed 's|/dev/||' || true) + lsblk -dno NAME,SIZE,TYPE | while read name size type; do + [[ \"\$type\" != \"disk\" ]] && continue + # Skip if in a zpool + echo \"\$used_disks\" | grep -qw \"\$name\" && continue + # Skip if has mounted partitions + lsblk -no MOUNTPOINT /dev/\$name 2>/dev/null | grep -q '/' && continue + echo \"\$name \$size\" + done + " || true) + + if [[ -z "$available_disks" ]]; then + info "No unused disks found -- skipping storage pool creation" + info "If you have disks to use, create the pool manually:" + info " zpool create local-zfs mirror /dev/sdX /dev/sdY" + info " pvesm add zfspool local-zfs -pool local-zfs" + return 0 + fi + + info "Available disks:" + echo "$available_disks" | while IFS= read -r line; do + echo " /dev/$line" + done + + local disk_count + disk_count=$(echo "$available_disks" | wc -l) + local disk_names + disk_names=$(echo "$available_disks" | awk '{print "/dev/"$1}' | tr '\n' ' ') + + local pool_type + if [[ "$disk_count" -eq 1 ]]; then + pool_type="single" + info "Recommendation: single disk (no redundancy)" + elif [[ "$disk_count" -eq 2 ]]; then + pool_type="mirror" + info "Recommendation: mirror (2 disks, full redundancy)" + else + pool_type="raidz1" + info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)" + fi + + local pool_name + pool_name=$(prompt_value "Pool name" "local-zfs") + + if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then + return 0 + fi + + local zpool_cmd="zpool create ${pool_name}" + case "$pool_type" in + mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;; + raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;; + single) zpool_cmd="${zpool_cmd} ${disk_names}" ;; + esac + + remote "$zpool_cmd" + remote "pvesm add zfspool ${pool_name} -pool ${pool_name}" + remote "pvesm set ${pool_name} -content images,rootdir" + + success "ZFS pool '${pool_name}' created and registered" +} + +# --------------------------------------------------------------------------- +# Step 6: WireGuard +# --------------------------------------------------------------------------- + +step_wireguard() { + if [[ "$SKIP_WIREGUARD" == true ]]; then + info "Skipping WireGuard setup (--skip-wireguard)" + return 0 + fi + + step "WireGuard tunnel" + + # Check if already configured + local wg_active + wg_active=$(remote_output "wg show wg0 2>/dev/null" || true) + if [[ -n "$wg_active" ]]; then + success "WireGuard wg0 already active" + remote "wg show wg0" 2>/dev/null | head -8 | while IFS= read -r line; do + echo " $line" + done + return 0 + fi + + info "Will configure WireGuard:" + info " Server: ${WG_SERVER_IP}/${WG_SUBNET_MASK} on port ${WG_PORT}" + info " Client: ${WG_CLIENT_IP}/${WG_SUBNET_MASK}" + info " Routing: wg0 <-> vmbr1 forwarding" + + if ! confirm "Set up WireGuard?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then + return 0 + fi + + # Install + info "Installing wireguard..." + remote "DEBIAN_FRONTEND=noninteractive apt install -y -qq wireguard >/dev/null 2>&1" + + # Generate server keys + info "Generating server keypair..." + remote "wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key" + remote "chmod 600 /etc/wireguard/server-private.key" + + local server_privkey + server_privkey=$(remote_output "cat /etc/wireguard/server-private.key") + local server_pubkey + server_pubkey=$(remote_output "cat /etc/wireguard/server-public.key") + + # Generate client keys locally + info "Generating client keypair..." + local client_privkey client_pubkey + client_privkey=$(wg genkey) + client_pubkey=$(echo "$client_privkey" | wg pubkey) + + # Write server config + local server_conf="[Interface] +PrivateKey = ${server_privkey} +Address = ${WG_SERVER_IP}/${WG_SUBNET_MASK} +ListenPort = ${WG_PORT} + +# Allow forwarding between WireGuard clients and the private VM bridge +PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT +PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT + +[Peer] +# Workstation +PublicKey = ${client_pubkey} +AllowedIPs = ${WG_CLIENT_IP}/32" + + remote_write "/etc/wireguard/wg0.conf" "$server_conf" + remote "chmod 600 /etc/wireguard/wg0.conf" + + # Enable and start + remote "systemctl enable --now wg-quick@wg0" + + success "WireGuard server configured" + + # Print client config + echo "" + info "Client config -- save to /etc/wireguard/hetzner-lab.conf or import into WireGuard app:" + echo "" + echo " [Interface]" + echo " PrivateKey = ${client_privkey}" + echo " Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}" + echo "" + echo " [Peer]" + echo " PublicKey = ${server_pubkey}" + echo " Endpoint = ${PUBLIC_IP}:${WG_PORT}" + echo " AllowedIPs = 10.10.0.0/16" + echo " PersistentKeepalive = 25" + echo "" + warn "Save the client config now -- private key is not stored anywhere else" +} + +# --------------------------------------------------------------------------- +# Step 7: Firewall +# --------------------------------------------------------------------------- + +step_firewall() { + if [[ "$SKIP_FIREWALL" == true ]]; then + info "Skipping firewall setup (--skip-firewall)" + return 0 + fi + + step "Firewall lockdown (nftables)" + + # Check if already configured + local existing_rules + existing_rules=$(remote_output "test -f /etc/nftables-hetzner.conf && echo yes" || true) + if [[ "$existing_rules" == "yes" ]]; then + success "Firewall rules already present (/etc/nftables-hetzner.conf)" + return 0 + fi + + info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only" + info "Web UI (8006) and VM traffic only via WireGuard" + warn "Make sure WireGuard is working before applying firewall rules!" + + if ! confirm "Apply firewall rules?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then + return 0 + fi + + local nft_rules="#!/usr/sbin/nft -f + +flush ruleset + +table inet filter { + chain input { + type filter hook input priority 0; policy drop; + + # Loopback + iif lo accept + + # Established/related connections + ct state established,related accept + + # ICMP (ping) + ip protocol icmp accept + ip6 nexthdr icmpv6 accept + + # SSH on public interface + iifname \"vmbr0\" tcp dport 22 accept + + # WireGuard on public interface + iifname \"vmbr0\" udp dport ${WG_PORT} accept + + # Allow everything on private bridge and WireGuard + iifname \"vmbr1\" accept + iifname \"wg0\" accept + + # Log and drop everything else + log prefix \"nft-drop: \" limit rate 5/minute counter drop + } + + chain forward { + type filter hook forward priority 0; policy drop; + + # Established/related + ct state established,related accept + + # WireGuard <-> private bridge + iifname \"wg0\" oifname \"vmbr1\" accept + iifname \"vmbr1\" oifname \"wg0\" accept + + # Private bridge -> internet (NAT) + iifname \"vmbr1\" oifname \"vmbr0\" accept + } + + chain output { + type filter hook output priority 0; policy accept; + } +}" + + remote_write "/etc/nftables-hetzner.conf" "$nft_rules" + remote "nft -f /etc/nftables-hetzner.conf" + remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf" + remote "systemctl enable nftables" + + success "Firewall applied (SSH + WireGuard only on public interface)" +} + +# --------------------------------------------------------------------------- +# Step 8: API token +# --------------------------------------------------------------------------- + +step_api_token() { + step "API token for incusos-proxmox" + + # Check if role already exists + local existing_role + existing_role=$(remote_output "pveum role list --output-format json 2>/dev/null | python3 -c 'import sys,json; [print(r[\"roleid\"]) for r in json.load(sys.stdin) if r[\"roleid\"]==\"IncusOSDeployer\"]'" || true) + + if [[ -n "$existing_role" ]]; then + success "IncusOSDeployer role already exists" + else + if ! confirm "Create API role, pool, and token?"; then + info "Skipped" + return 0 + fi + + if dry_run_guard "Create IncusOSDeployer role + automation@pve!deploy token"; then + return 0 + fi + + # Create role + remote "pveum role add IncusOSDeployer -privs \ + 'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ + VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ + VM.Monitor VM.Audit VM.Console \ + Datastore.AllocateSpace Datastore.Audit \ + Pool.Allocate Pool.Audit \ + SDN.Use Sys.Modify'" + success "Created IncusOSDeployer role" + fi + + # Create pool if needed + local existing_pool + existing_pool=$(remote_output "pveum pool list --output-format json 2>/dev/null | python3 -c 'import sys,json; [print(p[\"poolid\"]) for p in json.load(sys.stdin) if p[\"poolid\"]==\"IncusLab\"]'" || true) + + if [[ -z "$existing_pool" ]] && [[ "$DRY_RUN" != true ]]; then + remote "pveum pool add IncusLab -comment 'IncusOS lab VMs'" + success "Created IncusLab pool" + fi + + # Create token if user doesn't exist yet + local existing_token + existing_token=$(remote_output "pveum user token list automation@pve --output-format json 2>/dev/null | python3 -c 'import sys,json; [print(t[\"tokenid\"]) for t in json.load(sys.stdin) if t[\"tokenid\"]==\"deploy\"]'" || true) + + if [[ -n "$existing_token" ]]; then + success "automation@pve!deploy token already exists" + info "If you need to regenerate the secret, delete and recreate the token" + return 0 + fi + + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) Would create automation@pve!deploy token" + return 0 + fi + + # Create token + local token_output + token_output=$(remote "pveum user token add automation@pve deploy --privsep 0 --output-format json 2>/dev/null" || true) + + if [[ -z "$token_output" ]]; then + # User might not exist yet + remote "pveum user add automation@pve --comment 'IncusOS automation' 2>/dev/null || true" + token_output=$(remote "pveum user token add automation@pve deploy --privsep 0 --output-format json") + fi + + local token_secret + token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output") + + # Assign ACLs + remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer" + + # Try to find the storage name for ACL assignment + local storage_name + storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true) + if [[ -n "$storage_name" ]]; then + remote "pveum acl modify /storage/${storage_name} -user automation@pve -role IncusOSDeployer" + fi + remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer" + + success "API token created" + echo "" + warn "Token secret (save now -- shown only once):" + echo "" + echo " ${token_secret}" + echo "" +} + +# --------------------------------------------------------------------------- +# Step 9: Summary output +# --------------------------------------------------------------------------- + +step_summary() { + step "Summary" + + echo "" + echo -e "${BOLD}proxmox.yaml for incusos/targets/hetzner/:${RESET}" + echo "" + echo " host: ${SUBNET_GW}" + echo " method: api" + echo " api_token_id: automation@pve!deploy" + echo " node: ${HOSTNAME_DETECTED:-pve}" + echo " storage: local-zfs" + echo " iso_storage: local" + echo " bridge: vmbr1" + echo " gateway: ${SUBNET_GW}" + echo " dns: ${SUBNET_GW}" + echo " pool: IncusLab" + echo " ssh_user: root" + + echo "" + echo -e "${BOLD}env file additions:${RESET}" + echo "" + echo " # Hetzner" + echo " HETZNER_PROXMOX_TOKEN_SECRET=" + echo " HETZNER_PROXMOX_ROOT_PASSWORD=" + + echo "" + echo -e "${BOLD}Next steps:${RESET}" + echo "" + echo " 1. Save WireGuard client config and connect" + echo " 2. Copy proxmox.yaml.example and fill in credentials:" + echo " cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml" + echo " 3. Test with dry run:" + echo " export PROXMOX_TOKEN_SECRET=\"\$HETZNER_PROXMOX_TOKEN_SECRET\"" + echo " incusos-proxmox --dry-run --proxmox incusos/targets/hetzner/proxmox.yaml \\" + echo " incusos/targets/hetzner/lab-cluster.yaml" + echo "" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +main() { + setup_colors + parse_args "$@" + parse_subnets + + echo "" + echo -e "${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} -- Proxmox post-install configuration" + if [[ "$DRY_RUN" == true ]]; then + echo -e "${YELLOW}(dry-run mode -- no changes will be made)${RESET}" + fi + echo "" + + step_detect + step_repos + step_update + step_bridge + step_storage + step_wireguard + step_firewall + step_api_token + step_summary + + success "Setup complete" +} + +main "$@" diff --git a/incusos/targets/README.md b/incusos/targets/README.md new file mode 100644 index 0000000..3f17282 --- /dev/null +++ b/incusos/targets/README.md @@ -0,0 +1,86 @@ +# Target Configurations + +Target-specific Proxmox connection configs and lab definitions. Each +subdirectory represents a physical host (or host type) with its own +network topology, storage backend, and resource sizing. + +## How it works + +`incusos-proxmox` already supports `--proxmox FILE` to point at any +connection config. Targets are just an organizational convention -- +no script changes required. + +## Directory layout + +``` +targets/ +├── beelink/ +│ ├── proxmox.yaml.example # Connection template (LAN, vmbr0, local-lvm) +│ └── lab-cluster.yaml # 3-node cluster sized for beelink (4C/4G/50G) +└── hetzner/ + ├── proxmox.yaml.example # Connection template (WireGuard, vmbr1, local-zfs) + ├── lab-cluster.yaml # 3-node cluster sized for hetzner (8C/16G/100G) + └── lab-production.yaml # OC + 3-node cluster for hetzner +``` + +## Usage + +### 1. Copy the example and fill in credentials + +```bash +cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml +# Edit with your API token, host IP, etc. +``` + +### 2. Deploy with explicit --proxmox + +```bash +incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ + incusos/targets/hetzner/lab-cluster.yaml +``` + +### 3. Or symlink for convenience + +```bash +# Point the auto-discovered proxmox.yaml at your active target +ln -sf targets/hetzner/proxmox.yaml incusos/proxmox.yaml + +# Now deploy without --proxmox +incusos-proxmox incusos/targets/hetzner/lab-cluster.yaml +``` + +## Environment variables + +`incusos-proxmox` reads `PROXMOX_TOKEN_SECRET` from the environment (or +the `env` file at the repo root). When working with multiple targets, +use different variable names per host: + +```bash +# In env file +PROXMOX_TOKEN_SECRET=beelink-token-here +HETZNER_PROXMOX_TOKEN_SECRET=hetzner-token-here +``` + +Before deploying to a specific target, export the right secret: + +```bash +# Deploy to Hetzner +export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" +incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ + incusos/targets/hetzner/lab-cluster.yaml +``` + +## Key differences between targets + +| Setting | Beelink | Hetzner | +|---------|---------|---------| +| Host | 192.168.1.10 (LAN) | 10.10.0.1 (WireGuard) | +| Method | ssh | api | +| Bridge | vmbr0 | vmbr1 (private) | +| Storage | local-lvm | local-zfs | +| VLAN | 69 | *(none)* | +| Gateway | 192.168.100.1 | 10.10.0.1 | +| VM IPs | 192.168.102.x/22 | 10.10.0.x/24 | +| VM cores | 4 | 8 | +| VM memory | 4-8 GiB | 16 GiB | +| VM disk | 50 GiB | 100 GiB | diff --git a/incusos/targets/beelink/lab-cluster.yaml b/incusos/targets/beelink/lab-cluster.yaml new file mode 100644 index 0000000..535d1f3 --- /dev/null +++ b/incusos/targets/beelink/lab-cluster.yaml @@ -0,0 +1,32 @@ +# lab-cluster.yaml - 3-node IncusOS cluster for Beelink +# +# Sized for a Beelink mini PC (typically 8-16 cores, 32-64 GiB RAM). +# Deploys 3 standalone Incus nodes that can be clustered after deployment. +# +# Usage: +# incusos-proxmox --proxmox incusos/targets/beelink/proxmox.yaml \ +# incusos/targets/beelink/lab-cluster.yaml +# +# Connection settings from proxmox.yaml in this directory. + +defaults: + cores: 4 + memory: 4096 + disk: 50 + start_vmid: 900 + +vms: + - name: incus-lab-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + ip: 192.168.102.101/22 + + - name: incus-lab-02 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.102/22 + + - name: incus-lab-03 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.103/22 diff --git a/incusos/targets/beelink/proxmox.yaml.example b/incusos/targets/beelink/proxmox.yaml.example new file mode 100644 index 0000000..7ea21da --- /dev/null +++ b/incusos/targets/beelink/proxmox.yaml.example @@ -0,0 +1,20 @@ +# proxmox.yaml.example - Beelink mini PC target +# +# Beelink SER5 (or similar) running Proxmox on the local network. +# Copy to proxmox.yaml and edit with your actual credentials. +# +# Network: LAN access via vmbr0 with VLAN 69 for VM traffic. +# Storage: local-lvm (LVM thin pool on internal NVMe). + +host: 192.168.1.10 # Proxmox host IP (LAN) +method: ssh # ssh (default) or api +# api_token_id: automation@pve!deploy # For api method +node: pve # Proxmox node name +storage: local-lvm # VM disk storage +iso_storage: local # ISO/image storage +bridge: vmbr0 # Network bridge +vlan: 69 # VLAN tag for VM NIC +gateway: 192.168.100.1 # Default gateway for static IPs +dns: 192.168.100.1 # DNS server for static IPs +pool: IncusLab # Resource pool +ssh_user: root # SSH user (for ssh method) diff --git a/incusos/targets/hetzner/lab-cluster.yaml b/incusos/targets/hetzner/lab-cluster.yaml new file mode 100644 index 0000000..6c76cc3 --- /dev/null +++ b/incusos/targets/hetzner/lab-cluster.yaml @@ -0,0 +1,35 @@ +# lab-cluster.yaml - 3-node IncusOS cluster for Hetzner +# +# Sized for a Hetzner dedicated server (32+ cores, 128+ GiB RAM). +# More resources per VM than the beelink target: 8 cores, 16 GiB RAM, 100 GiB disk. +# +# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet). +# VMs get static IPs in the 10.10.0.101-103 range. +# +# Usage: +# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ +# incusos/targets/hetzner/lab-cluster.yaml +# +# Connection settings from proxmox.yaml in this directory. + +defaults: + cores: 8 + memory: 16384 + disk: 100 + start_vmid: 900 + +vms: + - name: incus-lab-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + ip: 10.10.0.101/24 + + - name: incus-lab-02 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 10.10.0.102/24 + + - name: incus-lab-03 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 10.10.0.103/24 diff --git a/incusos/targets/hetzner/lab-production.yaml b/incusos/targets/hetzner/lab-production.yaml new file mode 100644 index 0000000..00abcc8 --- /dev/null +++ b/incusos/targets/hetzner/lab-production.yaml @@ -0,0 +1,56 @@ +# lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster) +# +# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering. +# Sized for a Hetzner dedicated server with generous resources. +# +# Architecture: +# hz-oc (910) - Operations Center dashboard and monitoring +# hz-node-01 (911) - Cluster init node + OVN control plane host (150G disk) +# hz-node-02 (912) - Cluster member +# hz-node-03 (913) - Cluster member +# +# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet). +# OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external. +# +# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads) +# +# Usage: +# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ +# incusos/targets/hetzner/lab-production.yaml +# +# After deployment, follow notes/production-lab-guide.md for: +# - Cluster formation (section 4) +# - OVN networking (section 6) +# - Workload deployment (sections 7-9) +# +# Connection settings from proxmox.yaml in this directory. + +defaults: + cores: 8 + memory: 16384 + disk: 100 + start_vmid: 910 + +vms: + - name: hz-oc + app: operations-center + apply_defaults: true + cores: 4 + memory: 8192 + ip: 10.10.0.110/24 + + - name: hz-node-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + disk: 150 # extra space for OVN control plane container + ip: 10.10.0.111/24 + + - name: hz-node-02 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 10.10.0.112/24 + + - name: hz-node-03 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 10.10.0.113/24 diff --git a/incusos/targets/hetzner/proxmox.yaml.example b/incusos/targets/hetzner/proxmox.yaml.example new file mode 100644 index 0000000..e2cf20d --- /dev/null +++ b/incusos/targets/hetzner/proxmox.yaml.example @@ -0,0 +1,25 @@ +# proxmox.yaml.example - Hetzner dedicated server target +# +# Hetzner bare metal server with Proxmox, accessed via WireGuard tunnel. +# Copy to proxmox.yaml and edit with your actual credentials. +# +# Network: Private bridge vmbr1 (10.10.0.0/24), no VLANs needed. +# Proxmox host is the gateway (10.10.0.1) with NAT masquerading. +# Access from workstation via WireGuard tunnel (10.10.99.0/24). +# +# Storage: local-zfs (ZFS pool on dedicated disks). +# +# See hetzner/hetzner-setup.md for the full setup guide. + +host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1) +method: api # api recommended (WireGuard may add SSH latency) +api_token_id: automation@pve!deploy +node: pve # Proxmox node name (adjust if different) +storage: local-zfs # VM disk storage (ZFS pool) +iso_storage: local # ISO/image storage +bridge: vmbr1 # Private bridge (10.10.0.0/24) +# vlan: # No VLAN needed (dedicated private bridge) +gateway: 10.10.0.1 # Proxmox host is the gateway +dns: 10.10.0.1 # Proxmox host forwards DNS (or use 1.1.1.1) +pool: IncusLab # Resource pool +ssh_user: root # SSH user (for ssh method fallback)