incus-contrib/hetzner/hetzner-setup.md

# Hetzner Dedicated Server: Proxmox Setup Guide

Complete guide for turning a Hetzner bare metal server into a Proxmox host
ready for IncusOS lab deployments. Each section can be done manually or
automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)).

---

## 1. Server selection

The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used
dedicated servers at significant discounts. Look for:

- **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested
  virtualization is more proven with Proxmox/IncusOS)
- **Cores**: 32+ (each IncusOS VM wants 4-8 cores)
- **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB)
- **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage)
- **Network**: 1 Gbit/s included, single IPv4

Order the server and wait for provisioning (usually minutes for auction
servers). Note the assigned IP address and temporary root password.

---

## 2. Proxmox installation

We use the official Proxmox VE ISO installer running inside QEMU from
the Hetzner rescue system. This gives the full graphical installer with
ZFS support, which is not available via Hetzner's `installimage`. The
method is documented in the
[Hetzner community tutorial](https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve).

### 2.1 Boot the rescue system

In the Hetzner Robot panel, activate the rescue system (Linux 64-bit)
and reboot the server. Then SSH in:

```bash
ssh root@<public-ip>
```

### 2.2 Discover disks

Identify which disks the server has:

```bash
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE
```

Example output on a typical auction server with 2 NVMe drives:

```
NAME    SIZE MODEL                ROTA TYPE
nvme0n1 477G Samsung SSD 970 EVO     0 disk
nvme1n1 477G Samsung SSD 970 EVO     0 disk
```

For ZFS RAID1 (mirror), you want two matching disks. NVMe pairs are the
strong favourite -- fast and reliable. Note the device names (`/dev/nvme0n1`,
`/dev/nvme1n1`) for the QEMU command.

If you have additional disks beyond the pair (e.g. large SATA drives),
those can be set up as a separate storage pool later (section 6). Only
pass the system disks to QEMU for now.

### 2.3 Check BIOS mode

Determine whether the server boots in UEFI or legacy BIOS mode:

```bash
[ -d "/sys/firmware/efi" ] && echo "UEFI" || echo "BIOS"
```

Most modern Hetzner servers are UEFI. The QEMU command below includes
the OVMF BIOS line for UEFI -- remove it if your server reports BIOS.

### 2.4 Download the Proxmox ISO

Get the latest Proxmox VE ISO from the official download page:

```bash
wget https://enterprise.proxmox.com/iso/proxmox-ve_9.0-1.iso
```

Check [proxmox.com/en/downloads](https://www.proxmox.com/en/downloads)
for the current version and adjust the URL accordingly.

### 2.5 Set up SSH port forwarding for VNC

On your **workstation** (not the server), open an SSH tunnel forwarding
the VNC port:

```bash
ssh -L 5900:localhost:5900 root@<public-ip>
```

This forwards local port 5900 to the server's localhost:5900, where
QEMU will expose its VNC display. Keep this session open.

### 2.6 Start the QEMU installer

In the SSH session on the server, start QEMU with the ISO and the disks
you identified in step 2.2:

```bash
qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -m 16G \
    -boot d \
    -cdrom ./proxmox-ve_9.0-1.iso \
    -drive file=/dev/nvme0n1,format=raw,if=virtio \
    -drive file=/dev/nvme1n1,format=raw,if=virtio \
    -bios /usr/share/OVMF/OVMF_CODE.fd \
    -vnc 127.0.0.1:0
```

**Notes:**
- Remove the `-bios /usr/share/OVMF/OVMF_CODE.fd \` line for legacy
  BIOS servers (step 2.3).
- For SATA drives, use `/dev/sda`, `/dev/sdb` instead of `/dev/nvmeXn1`.
- Disks appear as `/dev/vdX` inside the VM because of the virtio interface
  -- this is normal and expected.
- The `-vnc 127.0.0.1:0` flag binds VNC to localhost only (safe, no
  password needed since it's behind the SSH tunnel).

### 2.7 Connect via VNC and install

Open a VNC client on your workstation and connect to `127.0.0.1:5900`
(or just `127.0.0.1` -- most clients default to port 5900).

The Proxmox graphical installer appears. Walk through it:

1. Accept the EULA.
2. **Target disk**: Select the ZFS RAID1 (mirror) option across both
   drives (`/dev/vda` and `/dev/vdb` in the VM -- these are your NVMe
   drives passed through via virtio).
3. **Country/timezone/keyboard**: Set as appropriate.
4. **Root password and email**: Set a strong root password. This becomes
   `HETZNER_PROXMOX_ROOT_PASSWORD` in your `env` file.
5. **Network**: The installer shows a virtualized NIC. Configure it with
   the server's public IP, gateway, and hostname. This will be corrected
   in the next step since the real NIC name differs.
6. Click **Install** and wait for completion.

The VNC client may disconnect briefly during install -- just reconnect
to `127.0.0.1:5900`.

### 2.8 Predict the real network interface name

After installation completes, stop QEMU with `Ctrl+C` in the SSH
terminal. **Do not reboot yet** -- the network interface name configured
by the installer is wrong (it matches the virtual NIC, not the real one).

Use the Hetzner `predict-check` tool to discover the real interface name:

```bash
predict-check
```

Example output:

```
eth0 -> enp0s31f6
```

Note the predicted name (e.g. `enp0s31f6`). You can also check the
current rescue interface for reference:

```bash
netdata
```

### 2.9 Fix the network config before first real boot

Boot Proxmox again in QEMU, **without** the ISO (no `-cdrom` flag):

```bash
qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -m 16G \
    -boot d \
    -drive file=/dev/nvme0n1,format=raw,if=virtio \
    -drive file=/dev/nvme1n1,format=raw,if=virtio \
    -bios /usr/share/OVMF/OVMF_CODE.fd \
    -vnc 127.0.0.1:0
```

Connect via VNC, log in as root, and edit the network configuration:

```bash
nano /etc/network/interfaces
```

Replace the virtual interface name (e.g. `ens18`) with the predicted
real name (e.g. `enp0s31f6`). A minimal working config:

```
auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet static
    address <public-ip>/32
    gateway <gateway-ip>
```

Save and shut down the VM (`shutdown -h now` inside the VNC session,
or `Ctrl+C` in the SSH terminal).

### 2.10 Reboot into Proxmox

Exit the rescue system and reboot the server from the Hetzner Robot
panel (or just `reboot` from SSH). The server now boots from disk
into Proxmox with the correct network configuration.

### 2.11 Verify

```bash
ssh root@<public-ip> pvesh get /version
# Should show Proxmox VE version and API info
```

The web UI is available at `https://<public-ip>:8006` (we'll lock this
down to WireGuard-only in section 9).

> **Alternative method**: You can also install Debian 13 via Hetzner's
> `installimage` and then upgrade to Proxmox following the
> [official guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie).
> This skips the QEMU/VNC process but does not offer ZFS-on-root from
> the installer.

---

## 3. Network configuration

Hetzner dedicateds get a single public IP with MAC filtering -- you cannot
assign additional public IPs to VMs without ordering extra IPs. Instead,
we create a private bridge and NAT.

### Create the private bridge

Add `vmbr1` to `/etc/network/interfaces`:

```
auto vmbr1
iface vmbr1 inet static
    address 10.10.0.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE
```

This creates a private bridge where:
- VMs connect to `vmbr1` and get IPs in 10.10.0.0/24
- The Proxmox host (10.10.0.1) is the default gateway
- NAT masquerading gives VMs internet access through the host's public IP
- IP forwarding is enabled automatically on bridge up

### Persist IP forwarding

Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence:

```
net.ipv4.ip_forward = 1
```

### Apply

```bash
ifreload -a        # Apply network changes without reboot
sysctl -p /etc/sysctl.d/99-forward.conf
```

### Verify

```bash
ip addr show vmbr1          # Should show 10.10.0.1/24
cat /proc/sys/net/ipv4/ip_forward  # Should show 1
iptables -t nat -L POSTROUTING -n  # Should show MASQUERADE rule
```

### Network diagram

```
Internet
    |
    | Public IP (5.9.x.x)
    v
[vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24]
                  |                       |
                  |                  WireGuard tunnel
                  |                       |
             [vmbr1: 10.10.0.1/24]   Your workstation
                  |                  (10.10.99.2)
            ┌─────┼─────┐
            |     |     |
          VM-01 VM-02 VM-03
          .101  .102  .103
```

---

## 4. DNS setup (optional)

If you have a domain, create an A record pointing to the public IP:

```
pve.example.com → 5.9.x.x
```

This is convenient but not required -- everything is accessed via
WireGuard using private IPs anyway. If you set a hostname, update
`/etc/hosts`:

```
10.10.0.1    pve.example.com pve
```

And set the hostname:

```bash
hostnamectl set-hostname pve
```

---

## 5. SSH hardening

### Copy your SSH key

```bash
ssh-copy-id root@<public-ip>
```

### Disable password authentication

Edit `/etc/ssh/sshd_config`:

```
PermitRootLogin prohibit-password
PasswordAuthentication no
```

Restart:

```bash
systemctl restart sshd
```

### Add a colleague's key

```bash
# From their workstation:
ssh-copy-id root@<public-ip>

# Or manually append to /root/.ssh/authorized_keys
```

### SSH config for convenient access

Add to `~/.ssh/config` on your workstation:

```
Host hetzner-lab
    HostName 5.9.x.x           # Public IP (or pve.example.com)
    User root
    IdentityFile ~/.ssh/id_ed25519
```

After WireGuard is set up (section 8), add a second entry:

```
Host hetzner-lab-wg
    HostName 10.10.0.1
    User root
    IdentityFile ~/.ssh/id_ed25519
```

### Verify

```bash
ssh hetzner-lab pvesh get /version
```

---

## 6. Disk and storage setup

### Identify disks

```bash
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE
```

Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from
install), 2+ additional disks for VM storage.

### Check existing pools

```bash
zpool status            # System mirror
pvesm status            # Proxmox storage backends
```

### Create VM storage pool

If you have additional disks (e.g. `sda`, `sdb`) not used by the system:

```bash
# Mirror (2 disks) -- recommended
zpool create local-zfs mirror /dev/sda /dev/sdb

# RAIDZ1 (3+ disks) -- more space, still redundant
zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc

# Single disk (no redundancy)
zpool create local-zfs /dev/sda
```

Register with Proxmox:

```bash
pvesm add zfspool local-zfs -pool local-zfs
pvesm set local-zfs -content images,rootdir
```

### Verify

```bash
pvesm status
# Should list local-zfs with type zfspool
```

If the system install already created a suitable ZFS pool, skip pool
creation and just register it with Proxmox if not already visible.

---

## 7. Repositories and system update

### Switch to no-subscription repositories

The default enterprise repos require a paid subscription. Switch to
community repos:

```bash
# Disable enterprise repo
sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list

# Add no-subscription repo
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
    > /etc/apt/sources.list.d/pve-no-subscription.list

# Disable Ceph enterprise repo if present
if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then
    sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list
fi
```

### Remove subscription nag popup

```bash
# Standard JS patch for the web UI subscription dialog
sed -Ezi.bak \
    "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \
    /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
systemctl restart pveproxy
```

### Update

```bash
apt update && apt dist-upgrade -y
```

### Verify

```bash
apt update 2>&1 | tail -5
# Should show no errors, all repos reachable
```

---

## 8. WireGuard tunnel

WireGuard provides secure access to VMs and the Proxmox web UI from
your workstation, without exposing anything on the public interface.

### Install

```bash
apt install -y wireguard
```

### Generate server keys

```bash
wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key
chmod 600 /etc/wireguard/server-private.key
```

### Generate client keys (on your workstation)

```bash
wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key
```

### Server config

Create `/etc/wireguard/wg0.conf`:

```ini
[Interface]
PrivateKey = <contents of server-private.key>
Address = 10.10.99.1/24
ListenPort = 51820

# Allow forwarding between WireGuard clients and the private VM bridge
PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT

[Peer]
# Workstation
PublicKey = <contents of wg-client-public.key>
AllowedIPs = 10.10.99.2/32
```

### Client config (on your workstation)

Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app):

```ini
[Interface]
PrivateKey = <contents of wg-client-private.key>
Address = 10.10.99.2/24

[Peer]
PublicKey = <contents of server-public.key>
Endpoint = <public-ip>:51820
AllowedIPs = 10.10.0.0/16
PersistentKeepalive = 25
```

**AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet
leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.)
without needing to update the WireGuard config.

### Enable and start

On the server:

```bash
systemctl enable --now wg-quick@wg0
```

On the workstation:

```bash
# Linux
sudo wg-quick up hetzner-lab

# macOS (WireGuard app)
# Import hetzner-lab.conf and activate
```

### Verify

```bash
# On the server
wg show

# From your workstation
ping 10.10.99.1   # WireGuard interface
ping 10.10.0.1    # Private bridge (should work -- routed via WireGuard)
```

### Adding more peers

To add a colleague, generate a new keypair and add a `[Peer]` block to
the server config:

```ini
[Peer]
# Colleague name
PublicKey = <their-public-key>
AllowedIPs = 10.10.99.3/32
```

Then reload:

```bash
systemctl restart wg-quick@wg0
```

Give them a client config with `Address = 10.10.99.3/24` and the
server's public key.

---

## 9. Firewall lockdown

After WireGuard is working, lock down the public interface so only SSH
and WireGuard are accessible from the internet. Proxmox web UI and VM
traffic go through the tunnel only.

### nftables rules

Create `/etc/nftables-hetzner.conf`:

```
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;

        # Loopback
        iif lo accept

        # Established/related connections
        ct state established,related accept

        # ICMP (ping)
        ip protocol icmp accept
        ip6 nexthdr icmpv6 accept

        # SSH on public interface
        iifname "vmbr0" tcp dport 22 accept

        # WireGuard on public interface
        iifname "vmbr0" udp dport 51820 accept

        # Allow everything on private bridge and WireGuard
        iifname "vmbr1" accept
        iifname "wg0" accept

        # Log and drop everything else
        log prefix "nft-drop: " limit rate 5/minute counter drop
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # Established/related
        ct state established,related accept

        # WireGuard <-> private bridge
        iifname "wg0" oifname "vmbr1" accept
        iifname "vmbr1" oifname "wg0" accept

        # Private bridge -> internet (NAT)
        iifname "vmbr1" oifname "vmbr0" accept
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}
```

### Apply

```bash
nft -f /etc/nftables-hetzner.conf
```

### Persist across reboots

```bash
cp /etc/nftables-hetzner.conf /etc/nftables.conf
systemctl enable nftables
```

### Verify

From an external machine (not through WireGuard):

```bash
# Should work
ssh hetzner-lab echo ok

# Should time out (port 8006 blocked on public interface)
curl -sk --connect-timeout 5 https://<public-ip>:8006

# Through WireGuard -- should work
curl -sk https://10.10.0.1:8006
```

**Warning**: Test SSH access through WireGuard *before* applying firewall
rules. If you lock yourself out, use Hetzner's rescue system to recover.

---

## 10. API token setup

Create a dedicated API token for `incusos-proxmox` automation.

### Create role and user

```bash
# Create role with required privileges
pveum role add IncusOSDeployer -privs \
    "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
     VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
     VM.Monitor VM.Audit VM.Console \
     Datastore.AllocateSpace Datastore.Audit \
     Pool.Allocate Pool.Audit \
     SDN.Use Sys.Modify"

# Create resource pool
pveum pool add IncusLab -comment "IncusOS lab VMs"

# Create API token
pveum user token add automation@pve deploy --privsep 0
# Save the displayed token secret -- it's shown only once!

# Assign role to user on the pool
pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer

# Assign role on storage
pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer
pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer
```

### Record credentials

Add to the `env` file at the repo root:

```bash
# Hetzner
HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password
```

### Verify

```bash
# Export for incusos-proxmox
export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"

# Test API access (update host/node in proxmox-api or use curl directly)
curl -sk "https://10.10.0.1:8006/api2/json/version" \
    -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET"
```

---

## 11. Final verification checklist

| Check | Command | Expected |
|-------|---------|----------|
| Proxmox version | `pvesh get /version` | PVE 8.x |
| Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 |
| IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 |
| NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 |
| ZFS storage | `pvesm status` | local-zfs available |
| WireGuard | `wg show` | 1+ peer, handshake recent |
| SSH via WG | `ssh hetzner-lab-wg hostname` | Responds |
| Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response |
| API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version |
| Public lockdown | `curl --connect-timeout 5 https://<pub-ip>:8006` | Timeout |

### Test deploy

```bash
export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"

# Dry run -- verify correct bridge, IPs, storage
incusos-proxmox --dry-run \
    --proxmox incusos/targets/hetzner/proxmox.yaml \
    incusos/targets/hetzner/lab-cluster.yaml
```

Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`
storage, 8 cores per VM.