incus-contrib/hetzner/hetzner-setup.md

# Hetzner Dedicated Server: Proxmox Setup Guide

Complete guide for turning a Hetzner bare metal server into a Proxmox host
ready for IncusOS lab deployments. Each section can be done manually or
automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)).

---

## 1. Server selection

The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used
dedicated servers at significant discounts. Look for:

- **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested
  virtualization is more proven with Proxmox/IncusOS)
- **Cores**: 32+ (each IncusOS VM wants 4-8 cores)
- **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB)
- **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage)
- **Network**: 1 Gbit/s included, single IPv4

Order the server and wait for provisioning (usually minutes for auction
servers). Note the assigned IP address and temporary root password.

---

## 2. Proxmox installation

Hetzner provides two installation paths:

### Option A: installimage (recommended)

Boot the rescue system from the Hetzner Robot panel, then SSH in:

```bash
ssh root@<public-ip>
installimage
```

Select Proxmox VE from the list. The installer presents an editor for
disk configuration -- set up ZFS RAID1 mirror on two system disks:

```
SWRAID 0
SWRAIDLEVEL 1
PART /boot ext4 1G
PART lvm pve all
LV pve root ext4 100G
LV pve swap swap 16G
```

Reboot after installation completes.

### Option B: ISO install

Upload the Proxmox VE ISO via Robot panel → Server → ISO Images, then
boot from it. Follow the standard Proxmox installer. This gives more
control over partitioning but requires KVM console access.

### Verify

```bash
ssh root@<public-ip>
pvesh get /version
# Should show Proxmox VE version and API info
```

The web UI is available at `https://<public-ip>:8006` (we'll lock this
down to WireGuard-only in section 9).

---

## 3. Network configuration

Hetzner dedicateds get a single public IP with MAC filtering -- you cannot
assign additional public IPs to VMs without ordering extra IPs. Instead,
we create a private bridge and NAT.

### Create the private bridge

Add `vmbr1` to `/etc/network/interfaces`:

```
auto vmbr1
iface vmbr1 inet static
    address 10.10.0.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE
```

This creates a private bridge where:
- VMs connect to `vmbr1` and get IPs in 10.10.0.0/24
- The Proxmox host (10.10.0.1) is the default gateway
- NAT masquerading gives VMs internet access through the host's public IP
- IP forwarding is enabled automatically on bridge up

### Persist IP forwarding

Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence:

```
net.ipv4.ip_forward = 1
```

### Apply

```bash
ifreload -a        # Apply network changes without reboot
sysctl -p /etc/sysctl.d/99-forward.conf
```

### Verify

```bash
ip addr show vmbr1          # Should show 10.10.0.1/24
cat /proc/sys/net/ipv4/ip_forward  # Should show 1
iptables -t nat -L POSTROUTING -n  # Should show MASQUERADE rule
```

### Network diagram

```
Internet
    |
    | Public IP (5.9.x.x)
    v
[vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24]
                  |                       |
                  |                  WireGuard tunnel
                  |                       |
             [vmbr1: 10.10.0.1/24]   Your workstation
                  |                  (10.10.99.2)
            ┌─────┼─────┐
            |     |     |
          VM-01 VM-02 VM-03
          .101  .102  .103
```

---

## 4. DNS setup (optional)

If you have a domain, create an A record pointing to the public IP:

```
pve.example.com → 5.9.x.x
```

This is convenient but not required -- everything is accessed via
WireGuard using private IPs anyway. If you set a hostname, update
`/etc/hosts`:

```
10.10.0.1    pve.example.com pve
```

And set the hostname:

```bash
hostnamectl set-hostname pve
```

---

## 5. SSH hardening

### Copy your SSH key

```bash
ssh-copy-id root@<public-ip>
```

### Disable password authentication

Edit `/etc/ssh/sshd_config`:

```
PermitRootLogin prohibit-password
PasswordAuthentication no
```

Restart:

```bash
systemctl restart sshd
```

### Add a colleague's key

```bash
# From their workstation:
ssh-copy-id root@<public-ip>

# Or manually append to /root/.ssh/authorized_keys
```

### SSH config for convenient access

Add to `~/.ssh/config` on your workstation:

```
Host hetzner-lab
    HostName 5.9.x.x           # Public IP (or pve.example.com)
    User root
    IdentityFile ~/.ssh/id_ed25519
```

After WireGuard is set up (section 8), add a second entry:

```
Host hetzner-lab-wg
    HostName 10.10.0.1
    User root
    IdentityFile ~/.ssh/id_ed25519
```

### Verify

```bash
ssh hetzner-lab pvesh get /version
```

---

## 6. Disk and storage setup

### Identify disks

```bash
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE
```

Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from
install), 2+ additional disks for VM storage.

### Check existing pools

```bash
zpool status            # System mirror
pvesm status            # Proxmox storage backends
```

### Create VM storage pool

If you have additional disks (e.g. `sda`, `sdb`) not used by the system:

```bash
# Mirror (2 disks) -- recommended
zpool create local-zfs mirror /dev/sda /dev/sdb

# RAIDZ1 (3+ disks) -- more space, still redundant
zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc

# Single disk (no redundancy)
zpool create local-zfs /dev/sda
```

Register with Proxmox:

```bash
pvesm add zfspool local-zfs -pool local-zfs
pvesm set local-zfs -content images,rootdir
```

### Verify

```bash
pvesm status
# Should list local-zfs with type zfspool
```

If the system install already created a suitable ZFS pool, skip pool
creation and just register it with Proxmox if not already visible.

---

## 7. Repositories and system update

### Switch to no-subscription repositories

The default enterprise repos require a paid subscription. Switch to
community repos:

```bash
# Disable enterprise repo
sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list

# Add no-subscription repo
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
    > /etc/apt/sources.list.d/pve-no-subscription.list

# Disable Ceph enterprise repo if present
if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then
    sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list
fi
```

### Remove subscription nag popup

```bash
# Standard JS patch for the web UI subscription dialog
sed -Ezi.bak \
    "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \
    /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
systemctl restart pveproxy
```

### Update

```bash
apt update && apt dist-upgrade -y
```

### Verify

```bash
apt update 2>&1 | tail -5
# Should show no errors, all repos reachable
```

---

## 8. WireGuard tunnel

WireGuard provides secure access to VMs and the Proxmox web UI from
your workstation, without exposing anything on the public interface.

### Install

```bash
apt install -y wireguard
```

### Generate server keys

```bash
wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key
chmod 600 /etc/wireguard/server-private.key
```

### Generate client keys (on your workstation)

```bash
wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key
```

### Server config

Create `/etc/wireguard/wg0.conf`:

```ini
[Interface]
PrivateKey = <contents of server-private.key>
Address = 10.10.99.1/24
ListenPort = 51820

# Allow forwarding between WireGuard clients and the private VM bridge
PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT

[Peer]
# Workstation
PublicKey = <contents of wg-client-public.key>
AllowedIPs = 10.10.99.2/32
```

### Client config (on your workstation)

Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app):

```ini
[Interface]
PrivateKey = <contents of wg-client-private.key>
Address = 10.10.99.2/24

[Peer]
PublicKey = <contents of server-public.key>
Endpoint = <public-ip>:51820
AllowedIPs = 10.10.0.0/16
PersistentKeepalive = 25
```

**AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet
leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.)
without needing to update the WireGuard config.

### Enable and start

On the server:

```bash
systemctl enable --now wg-quick@wg0
```

On the workstation:

```bash
# Linux
sudo wg-quick up hetzner-lab

# macOS (WireGuard app)
# Import hetzner-lab.conf and activate
```

### Verify

```bash
# On the server
wg show

# From your workstation
ping 10.10.99.1   # WireGuard interface
ping 10.10.0.1    # Private bridge (should work -- routed via WireGuard)
```

### Adding more peers

To add a colleague, generate a new keypair and add a `[Peer]` block to
the server config:

```ini
[Peer]
# Colleague name
PublicKey = <their-public-key>
AllowedIPs = 10.10.99.3/32
```

Then reload:

```bash
systemctl restart wg-quick@wg0
```

Give them a client config with `Address = 10.10.99.3/24` and the
server's public key.

---

## 9. Firewall lockdown

After WireGuard is working, lock down the public interface so only SSH
and WireGuard are accessible from the internet. Proxmox web UI and VM
traffic go through the tunnel only.

### nftables rules

Create `/etc/nftables-hetzner.conf`:

```
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;

        # Loopback
        iif lo accept

        # Established/related connections
        ct state established,related accept

        # ICMP (ping)
        ip protocol icmp accept
        ip6 nexthdr icmpv6 accept

        # SSH on public interface
        iifname "vmbr0" tcp dport 22 accept

        # WireGuard on public interface
        iifname "vmbr0" udp dport 51820 accept

        # Allow everything on private bridge and WireGuard
        iifname "vmbr1" accept
        iifname "wg0" accept

        # Log and drop everything else
        log prefix "nft-drop: " limit rate 5/minute counter drop
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # Established/related
        ct state established,related accept

        # WireGuard <-> private bridge
        iifname "wg0" oifname "vmbr1" accept
        iifname "vmbr1" oifname "wg0" accept

        # Private bridge -> internet (NAT)
        iifname "vmbr1" oifname "vmbr0" accept
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}
```

### Apply

```bash
nft -f /etc/nftables-hetzner.conf
```

### Persist across reboots

```bash
cp /etc/nftables-hetzner.conf /etc/nftables.conf
systemctl enable nftables
```

### Verify

From an external machine (not through WireGuard):

```bash
# Should work
ssh hetzner-lab echo ok

# Should time out (port 8006 blocked on public interface)
curl -sk --connect-timeout 5 https://<public-ip>:8006

# Through WireGuard -- should work
curl -sk https://10.10.0.1:8006
```

**Warning**: Test SSH access through WireGuard *before* applying firewall
rules. If you lock yourself out, use Hetzner's rescue system to recover.

---

## 10. API token setup

Create a dedicated API token for `incusos-proxmox` automation.

### Create role and user

```bash
# Create role with required privileges
pveum role add IncusOSDeployer -privs \
    "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
     VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
     VM.Monitor VM.Audit VM.Console \
     Datastore.AllocateSpace Datastore.Audit \
     Pool.Allocate Pool.Audit \
     SDN.Use Sys.Modify"

# Create resource pool
pveum pool add IncusLab -comment "IncusOS lab VMs"

# Create API token
pveum user token add automation@pve deploy --privsep 0
# Save the displayed token secret -- it's shown only once!

# Assign role to user on the pool
pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer

# Assign role on storage
pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer
pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer
```

### Record credentials

Add to the `env` file at the repo root:

```bash
# Hetzner
HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password
```

### Verify

```bash
# Export for incusos-proxmox
export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"

# Test API access (update host/node in proxmox-api or use curl directly)
curl -sk "https://10.10.0.1:8006/api2/json/version" \
    -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET"
```

---

## 11. Final verification checklist

| Check | Command | Expected |
|-------|---------|----------|
| Proxmox version | `pvesh get /version` | PVE 8.x |
| Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 |
| IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 |
| NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 |
| ZFS storage | `pvesm status` | local-zfs available |
| WireGuard | `wg show` | 1+ peer, handshake recent |
| SSH via WG | `ssh hetzner-lab-wg hostname` | Responds |
| Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response |
| API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version |
| Public lockdown | `curl --connect-timeout 5 https://<pub-ip>:8006` | Timeout |

### Test deploy

```bash
export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"

# Dry run -- verify correct bridge, IPs, storage
incusos-proxmox --dry-run \
    --proxmox incusos/targets/hetzner/proxmox.yaml \
    incusos/targets/hetzner/lab-cluster.yaml
```

Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`
storage, 8 cores per VM.