diff --git a/hetzner/README.md b/hetzner/README.md index 02e0d70..5f62061 100644 --- a/hetzner/README.md +++ b/hetzner/README.md @@ -12,37 +12,49 @@ via a WireGuard tunnel instead of LAN. - 32+ cores, 128+ GiB RAM - 2+ disks (ZFS mirror for system, extra disks for VM storage) - An SSH key pair on your workstation +- A VNC client (for the Proxmox installer) - This repository cloned locally -## Quickstart +## Workflow -### 1. Install Proxmox on the server +The setup is split into a manual guide and an automated script: -Order the server, boot the rescue system, and install Proxmox via -`installimage` or ISO. See [hetzner-setup.md](hetzner-setup.md) section 2 -for details. +### 1. Manual: rent server and install Proxmox -### 2. Run the setup helper +Follow [hetzner-setup.md](hetzner-setup.md) sections 1-2: + +- Pick and order a server from the Hetzner auction +- Install Proxmox via QEMU in the rescue system (ZFS RAID1 mirror) +- Fix the network interface name, reboot, verify SSH works + +### 2. Manual: set up SSH alias + +Follow [hetzner-setup.md](hetzner-setup.md) section 3.1-3.3: + +```bash +ssh-copy-id root@ +# Add "Host hetzner-lab" entry to ~/.ssh/config +ssh hetzner-lab pvesh get /version # Must succeed +``` + +### 3. Automated: configure everything else ```bash -# Add SSH config entry for the server (see hetzner-setup.md section 5) -# Then run the interactive setup: hetzner/proxmox-setup --host hetzner-lab --dry-run # Preview hetzner/proxmox-setup --host hetzner-lab # Execute ``` -The script configures repositories, private networking, ZFS storage, -WireGuard tunnel, firewall, and API tokens -- everything needed to -deploy VMs. +The script interactively configures: repos and updates, private bridge +(vmbr1), ZFS storage on extra disks, WireGuard tunnel, firewall lockdown, +and API tokens for `incusos-proxmox`. -### 3. Deploy IncusOS VMs +### 4. Deploy IncusOS VMs ```bash -# Copy and edit the connection config cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml -# Fill in the API token from step 2 +# Fill in the API token from step 3 -# Deploy a 3-node cluster +export PROXMOX_TOKEN_SECRET="" incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ incusos/targets/hetzner/lab-cluster.yaml ``` @@ -51,8 +63,8 @@ incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \ | File | Description | |------|-------------| -| [hetzner-setup.md](hetzner-setup.md) | Comprehensive step-by-step guide | -| [proxmox-setup](proxmox-setup) | Interactive helper script (automates the guide) | +| [hetzner-setup.md](hetzner-setup.md) | Manual guide: server selection, Proxmox install, SSH setup | +| [proxmox-setup](proxmox-setup) | Interactive script: repos, networking, storage, WireGuard, firewall, API tokens | ## Network architecture diff --git a/hetzner/hetzner-setup.md b/hetzner/hetzner-setup.md index 784535d..6d15119 100644 --- a/hetzner/hetzner-setup.md +++ b/hetzner/hetzner-setup.md @@ -1,8 +1,13 @@ # Hetzner Dedicated Server: Proxmox Setup Guide -Complete guide for turning a Hetzner bare metal server into a Proxmox host -ready for IncusOS lab deployments. Each section can be done manually or -automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)). +This guide covers the manual steps to get a Hetzner dedicated server running +Proxmox VE with SSH access from your workstation. Once SSH works, the +interactive [`proxmox-setup`](proxmox-setup) script handles everything else +(repos, networking, storage, WireGuard, firewall, API tokens). + +**Workflow:** +1. **This guide** -- rent a server, install Proxmox via QEMU, get SSH working +2. **`proxmox-setup --host `** -- configure everything else interactively --- @@ -61,8 +66,9 @@ strong favourite -- fast and reliable. Note the device names (`/dev/nvme0n1`, `/dev/nvme1n1`) for the QEMU command. If you have additional disks beyond the pair (e.g. large SATA drives), -those can be set up as a separate storage pool later (section 6). Only -pass the system disks to QEMU for now. +those can be set up as a separate storage pool later. Only pass the system +disks to QEMU for now -- `proxmox-setup` will detect and configure the +rest. ### 2.3 Check BIOS mode @@ -223,9 +229,6 @@ ssh root@ pvesh get /version # Should show Proxmox VE version and API info ``` -The web UI is available at `https://:8006` (we'll lock this -down to WireGuard-only in section 9). - > **Alternative method**: You can also install Debian 13 via Hetzner's > `installimage` and then upgrade to Proxmox following the > [official guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie). @@ -234,63 +237,123 @@ down to WireGuard-only in section 9). --- -## 3. Network configuration +## 3. SSH setup and handoff to proxmox-setup -Hetzner dedicateds get a single public IP with MAC filtering -- you cannot -assign additional public IPs to VMs without ordering extra IPs. Instead, -we create a private bridge and NAT. +At this point Proxmox is installed and reachable via its public IP. +The remaining configuration (repos, private networking, storage, +WireGuard, firewall, API tokens) is handled by the `proxmox-setup` +script. You just need SSH key access and an alias. -### Create the private bridge - -Add `vmbr1` to `/etc/network/interfaces`: - -``` -auto vmbr1 -iface vmbr1 inet static - address 10.10.0.1/24 - bridge-ports none - bridge-stp off - bridge-fd 0 - post-up echo 1 > /proc/sys/net/ipv4/ip_forward - post-up iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE - post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE -``` - -This creates a private bridge where: -- VMs connect to `vmbr1` and get IPs in 10.10.0.0/24 -- The Proxmox host (10.10.0.1) is the default gateway -- NAT masquerading gives VMs internet access through the host's public IP -- IP forwarding is enabled automatically on bridge up - -### Persist IP forwarding - -Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence: - -``` -net.ipv4.ip_forward = 1 -``` - -### Apply +### 3.1 Copy your SSH key ```bash -ifreload -a # Apply network changes without reboot -sysctl -p /etc/sysctl.d/99-forward.conf +ssh-copy-id root@ ``` -### Verify +### 3.2 Create an SSH config alias + +Add to `~/.ssh/config` on your workstation: + +``` +Host hetzner-lab + HostName + User root + IdentityFile ~/.ssh/id_ed25519 +``` + +### 3.3 Verify SSH alias works ```bash -ip addr show vmbr1 # Should show 10.10.0.1/24 -cat /proc/sys/net/ipv4/ip_forward # Should show 1 -iptables -t nat -L POSTROUTING -n # Should show MASQUERADE rule +ssh hetzner-lab pvesh get /version ``` -### Network diagram +This must succeed before continuing. If it fails, check your SSH key +and config entry. + +### 3.4 Run proxmox-setup + +```bash +# Preview what will be configured +hetzner/proxmox-setup --host hetzner-lab --dry-run + +# Run for real (interactive, prompts before each step) +hetzner/proxmox-setup --host hetzner-lab +``` + +The script walks through each step interactively: + +1. **Detect** -- reads host info, block devices, existing config +2. **Repos** -- switches to no-subscription repos, removes nag popup +3. **Update** -- `apt update && apt dist-upgrade` +4. **Bridge** -- creates vmbr1 (private bridge, 10.10.0.0/24) with NAT +5. **Storage** -- detects unused disks, creates ZFS pool, registers with PVE +6. **WireGuard** -- installs and configures tunnel, prints client config +7. **Firewall** -- locks down public interface (SSH + WireGuard only) +8. **API token** -- creates role, pool, and token for `incusos-proxmox` + +Each step prompts for confirmation. Use `--yes` to skip prompts, or +`--skip-repos`, `--skip-storage`, etc. to skip individual steps. + +At the end, the script prints: +- A ready-to-use `proxmox.yaml` config for `incusos/targets/hetzner/` +- An `env` file snippet with the API token +- A WireGuard client config to save on your workstation + +### 3.5 Connect WireGuard and verify + +Save the WireGuard client config printed by the script: + +```bash +# Linux +sudo wg-quick up hetzner-lab + +# macOS -- import into the WireGuard app +``` + +Verify access through the tunnel: + +```bash +ping 10.10.0.1 # Proxmox host via WireGuard +curl -sk https://10.10.0.1:8006 # Web UI via WireGuard +``` + +Optionally add a second SSH alias for access via WireGuard: + +``` +Host hetzner-lab-wg + HostName 10.10.0.1 + User root + IdentityFile ~/.ssh/id_ed25519 +``` + +### 3.6 Test deploy + +```bash +# Copy and fill in the connection config +cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml + +# Set the API token +export PROXMOX_TOKEN_SECRET="" + +# Dry run -- verify correct bridge, IPs, storage +incusos-proxmox --dry-run \ + --proxmox incusos/targets/hetzner/proxmox.yaml \ + incusos/targets/hetzner/lab-cluster.yaml +``` + +Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs` +storage, 8 cores per VM. + +--- + +## Network architecture (reference) + +Once `proxmox-setup` has completed, the network looks like this: ``` Internet | - | Public IP (5.9.x.x) + | Public IP (e.g. 5.9.x.x) v [vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24] | | @@ -304,481 +367,11 @@ Internet .101 .102 .103 ``` ---- - -## 4. DNS setup (optional) - -If you have a domain, create an A record pointing to the public IP: - -``` -pve.example.com → 5.9.x.x -``` - -This is convenient but not required -- everything is accessed via -WireGuard using private IPs anyway. If you set a hostname, update -`/etc/hosts`: - -``` -10.10.0.1 pve.example.com pve -``` - -And set the hostname: - -```bash -hostnamectl set-hostname pve -``` - ---- - -## 5. SSH hardening - -### Copy your SSH key - -```bash -ssh-copy-id root@ -``` - -### Disable password authentication - -Edit `/etc/ssh/sshd_config`: - -``` -PermitRootLogin prohibit-password -PasswordAuthentication no -``` - -Restart: - -```bash -systemctl restart sshd -``` - -### Add a colleague's key - -```bash -# From their workstation: -ssh-copy-id root@ - -# Or manually append to /root/.ssh/authorized_keys -``` - -### SSH config for convenient access - -Add to `~/.ssh/config` on your workstation: - -``` -Host hetzner-lab - HostName 5.9.x.x # Public IP (or pve.example.com) - User root - IdentityFile ~/.ssh/id_ed25519 -``` - -After WireGuard is set up (section 8), add a second entry: - -``` -Host hetzner-lab-wg - HostName 10.10.0.1 - User root - IdentityFile ~/.ssh/id_ed25519 -``` - -### Verify - -```bash -ssh hetzner-lab pvesh get /version -``` - ---- - -## 6. Disk and storage setup - -### Identify disks - -```bash -lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE -``` - -Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from -install), 2+ additional disks for VM storage. - -### Check existing pools - -```bash -zpool status # System mirror -pvesm status # Proxmox storage backends -``` - -### Create VM storage pool - -If you have additional disks (e.g. `sda`, `sdb`) not used by the system: - -```bash -# Mirror (2 disks) -- recommended -zpool create local-zfs mirror /dev/sda /dev/sdb - -# RAIDZ1 (3+ disks) -- more space, still redundant -zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc - -# Single disk (no redundancy) -zpool create local-zfs /dev/sda -``` - -Register with Proxmox: - -```bash -pvesm add zfspool local-zfs -pool local-zfs -pvesm set local-zfs -content images,rootdir -``` - -### Verify - -```bash -pvesm status -# Should list local-zfs with type zfspool -``` - -If the system install already created a suitable ZFS pool, skip pool -creation and just register it with Proxmox if not already visible. - ---- - -## 7. Repositories and system update - -### Switch to no-subscription repositories - -The default enterprise repos require a paid subscription. Switch to -community repos: - -```bash -# Disable enterprise repo -sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list - -# Add no-subscription repo -echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \ - > /etc/apt/sources.list.d/pve-no-subscription.list - -# Disable Ceph enterprise repo if present -if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then - sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list -fi -``` - -### Remove subscription nag popup - -```bash -# Standard JS patch for the web UI subscription dialog -sed -Ezi.bak \ - "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \ - /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js -systemctl restart pveproxy -``` - -### Update - -```bash -apt update && apt dist-upgrade -y -``` - -### Verify - -```bash -apt update 2>&1 | tail -5 -# Should show no errors, all repos reachable -``` - ---- - -## 8. WireGuard tunnel - -WireGuard provides secure access to VMs and the Proxmox web UI from -your workstation, without exposing anything on the public interface. - -### Install - -```bash -apt install -y wireguard -``` - -### Generate server keys - -```bash -wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key -chmod 600 /etc/wireguard/server-private.key -``` - -### Generate client keys (on your workstation) - -```bash -wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key -``` - -### Server config - -Create `/etc/wireguard/wg0.conf`: - -```ini -[Interface] -PrivateKey = -Address = 10.10.99.1/24 -ListenPort = 51820 - -# Allow forwarding between WireGuard clients and the private VM bridge -PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT -PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT - -[Peer] -# Workstation -PublicKey = -AllowedIPs = 10.10.99.2/32 -``` - -### Client config (on your workstation) - -Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app): - -```ini -[Interface] -PrivateKey = -Address = 10.10.99.2/24 - -[Peer] -PublicKey = -Endpoint = :51820 -AllowedIPs = 10.10.0.0/16 -PersistentKeepalive = 25 -``` - -**AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet -leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.) -without needing to update the WireGuard config. - -### Enable and start - -On the server: - -```bash -systemctl enable --now wg-quick@wg0 -``` - -On the workstation: - -```bash -# Linux -sudo wg-quick up hetzner-lab - -# macOS (WireGuard app) -# Import hetzner-lab.conf and activate -``` - -### Verify - -```bash -# On the server -wg show - -# From your workstation -ping 10.10.99.1 # WireGuard interface -ping 10.10.0.1 # Private bridge (should work -- routed via WireGuard) -``` - -### Adding more peers - -To add a colleague, generate a new keypair and add a `[Peer]` block to -the server config: - -```ini -[Peer] -# Colleague name -PublicKey = -AllowedIPs = 10.10.99.3/32 -``` - -Then reload: - -```bash -systemctl restart wg-quick@wg0 -``` - -Give them a client config with `Address = 10.10.99.3/24` and the -server's public key. - ---- - -## 9. Firewall lockdown - -After WireGuard is working, lock down the public interface so only SSH -and WireGuard are accessible from the internet. Proxmox web UI and VM -traffic go through the tunnel only. - -### nftables rules - -Create `/etc/nftables-hetzner.conf`: - -``` -#!/usr/sbin/nft -f - -flush ruleset - -table inet filter { - chain input { - type filter hook input priority 0; policy drop; - - # Loopback - iif lo accept - - # Established/related connections - ct state established,related accept - - # ICMP (ping) - ip protocol icmp accept - ip6 nexthdr icmpv6 accept - - # SSH on public interface - iifname "vmbr0" tcp dport 22 accept - - # WireGuard on public interface - iifname "vmbr0" udp dport 51820 accept - - # Allow everything on private bridge and WireGuard - iifname "vmbr1" accept - iifname "wg0" accept - - # Log and drop everything else - log prefix "nft-drop: " limit rate 5/minute counter drop - } - - chain forward { - type filter hook forward priority 0; policy drop; - - # Established/related - ct state established,related accept - - # WireGuard <-> private bridge - iifname "wg0" oifname "vmbr1" accept - iifname "vmbr1" oifname "wg0" accept - - # Private bridge -> internet (NAT) - iifname "vmbr1" oifname "vmbr0" accept - } - - chain output { - type filter hook output priority 0; policy accept; - } -} -``` - -### Apply - -```bash -nft -f /etc/nftables-hetzner.conf -``` - -### Persist across reboots - -```bash -cp /etc/nftables-hetzner.conf /etc/nftables.conf -systemctl enable nftables -``` - -### Verify - -From an external machine (not through WireGuard): - -```bash -# Should work -ssh hetzner-lab echo ok - -# Should time out (port 8006 blocked on public interface) -curl -sk --connect-timeout 5 https://:8006 - -# Through WireGuard -- should work -curl -sk https://10.10.0.1:8006 -``` - -**Warning**: Test SSH access through WireGuard *before* applying firewall -rules. If you lock yourself out, use Hetzner's rescue system to recover. - ---- - -## 10. API token setup - -Create a dedicated API token for `incusos-proxmox` automation. - -### Create role and user - -```bash -# Create role with required privileges -pveum role add IncusOSDeployer -privs \ - "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ - VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ - VM.Monitor VM.Audit VM.Console \ - Datastore.AllocateSpace Datastore.Audit \ - Pool.Allocate Pool.Audit \ - SDN.Use Sys.Modify" - -# Create resource pool -pveum pool add IncusLab -comment "IncusOS lab VMs" - -# Create API token -pveum user token add automation@pve deploy --privsep 0 -# Save the displayed token secret -- it's shown only once! - -# Assign role to user on the pool -pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer - -# Assign role on storage -pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer -pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer -``` - -### Record credentials - -Add to the `env` file at the repo root: - -```bash -# Hetzner -HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password -``` - -### Verify - -```bash -# Export for incusos-proxmox -export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" - -# Test API access (update host/node in proxmox-api or use curl directly) -curl -sk "https://10.10.0.1:8006/api2/json/version" \ - -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET" -``` - ---- - -## 11. Final verification checklist - -| Check | Command | Expected | -|-------|---------|----------| -| Proxmox version | `pvesh get /version` | PVE 8.x | -| Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 | -| IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 | -| NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 | -| ZFS storage | `pvesm status` | local-zfs available | -| WireGuard | `wg show` | 1+ peer, handshake recent | -| SSH via WG | `ssh hetzner-lab-wg hostname` | Responds | -| Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response | -| API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version | -| Public lockdown | `curl --connect-timeout 5 https://:8006` | Timeout | - -### Test deploy - -```bash -export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" - -# Dry run -- verify correct bridge, IPs, storage -incusos-proxmox --dry-run \ - --proxmox incusos/targets/hetzner/proxmox.yaml \ - incusos/targets/hetzner/lab-cluster.yaml -``` - -Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs` -storage, 8 cores per VM. +| Interface | Purpose | +|-----------|---------| +| vmbr0 | Public (SSH + WireGuard only after firewall) | +| vmbr1 | Private bridge for VMs (10.10.0.0/24, NAT to internet) | +| wg0 | WireGuard tunnel (10.10.99.0/24) | + +AllowedIPs uses `10.10.0.0/16` to leave room for future subnets +(OVN overlay at 10.10.10.0/24, etc.) without updating WireGuard config.