incus-contrib/hetzner/hetzner-setup.md

# Hetzner Dedicated Server: Proxmox Setup Guide

This guide covers the manual steps to get a Hetzner dedicated server running
Proxmox VE with SSH access from your workstation. Once SSH works, the
interactive [`proxmox-setup`](proxmox-setup) script handles everything else
(repos, networking, storage, WireGuard, firewall, API tokens).

**Workflow:**
1. **This guide** -- rent a server, install Proxmox via QEMU, get SSH working
2. **`proxmox-setup --host <alias>`** -- configure everything else interactively

---

## 1. Server selection

The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used
dedicated servers at significant discounts. Look for:

- **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested
  virtualization is more proven with Proxmox/IncusOS)
- **Cores**: 32+ (each IncusOS VM wants 4-8 cores)
- **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB)
- **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage)
- **Network**: 1 Gbit/s included, single IPv4

Order the server and wait for provisioning (usually minutes for auction
servers). Note the assigned IP address and temporary root password.

---

## 2. Proxmox installation

We use the official Proxmox VE ISO installer running inside QEMU from
the Hetzner rescue system. This gives the full graphical installer with
ZFS support, which is not available via Hetzner's `installimage`. The
method is documented in the
[Hetzner community tutorial](https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve).

### 2.1 Boot the rescue system

In the Hetzner Robot panel, activate the rescue system (Linux 64-bit)
and reboot the server. Then SSH in:

```bash
ssh root@<public-ip>
```

### 2.2 Discover disks

Identify which disks the server has:

```bash
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE
```

Example output on a typical auction server with 2 NVMe drives:

```
NAME    SIZE MODEL                ROTA TYPE
nvme0n1 477G Samsung SSD 970 EVO     0 disk
nvme1n1 477G Samsung SSD 970 EVO     0 disk
```

For ZFS RAID1 (mirror), you want two matching disks. NVMe pairs are the
strong favourite -- fast and reliable. Note the device names (`/dev/nvme0n1`,
`/dev/nvme1n1`) for the QEMU command.

If you have additional disks beyond the pair (e.g. large SATA drives),
those can be set up as a separate storage pool later. Only pass the system
disks to QEMU for now -- `proxmox-setup` will detect and configure the
rest.

### 2.3 Check BIOS mode

Determine whether the server boots in UEFI or legacy BIOS mode:

```bash
[ -d "/sys/firmware/efi" ] && echo "UEFI" || echo "BIOS"
```

Most modern Hetzner servers are UEFI. The QEMU command below includes
the OVMF BIOS line for UEFI -- remove it if your server reports BIOS.

### 2.4 Download the Proxmox ISO

Get the latest Proxmox VE ISO from the official download page:

```bash
wget https://enterprise.proxmox.com/iso/proxmox-ve_9.0-1.iso
```

Check [proxmox.com/en/downloads](https://www.proxmox.com/en/downloads)
for the current version and adjust the URL accordingly.

### 2.5 Set up SSH port forwarding for VNC

On your **workstation** (not the server), open an SSH tunnel forwarding
the VNC port:

```bash
ssh -L 5900:localhost:5900 root@<public-ip>
```

This forwards local port 5900 to the server's localhost:5900, where
QEMU will expose its VNC display. Keep this session open.

### 2.6 Start the QEMU installer

In the SSH session on the server, start QEMU with the ISO and the disks
you identified in step 2.2:

```bash
qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -m 16G \
    -boot d \
    -cdrom ./proxmox-ve_9.0-1.iso \
    -drive file=/dev/nvme0n1,format=raw,if=virtio \
    -drive file=/dev/nvme1n1,format=raw,if=virtio \
    -bios /usr/share/OVMF/OVMF_CODE.fd \
    -vnc 127.0.0.1:0
```

**Notes:**
- Remove the `-bios /usr/share/OVMF/OVMF_CODE.fd \` line for legacy
  BIOS servers (step 2.3).
- For SATA drives, use `/dev/sda`, `/dev/sdb` instead of `/dev/nvmeXn1`.
- Disks appear as `/dev/vdX` inside the VM because of the virtio interface
  -- this is normal and expected.
- The `-vnc 127.0.0.1:0` flag binds VNC to localhost only (safe, no
  password needed since it's behind the SSH tunnel).

### 2.7 Connect via VNC and install

Open a VNC client on your workstation and connect to `127.0.0.1:5900`
(or just `127.0.0.1` -- most clients default to port 5900).

The Proxmox graphical installer appears. Walk through it:

1. Accept the EULA.
2. **Target disk**: Select the ZFS RAID1 (mirror) option across both
   drives (`/dev/vda` and `/dev/vdb` in the VM -- these are your NVMe
   drives passed through via virtio).
3. **Country/timezone/keyboard**: Set as appropriate.
4. **Root password and email**: Set a strong root password. This becomes
   `HETZNER_PROXMOX_ROOT_PASSWORD` in your `env` file.
5. **Network**: The installer shows a virtualized NIC. Configure it with
   the server's public IP, gateway, and hostname. This will be corrected
   in the next step since the real NIC name differs.
6. Click **Install** and wait for completion.

The VNC client may disconnect briefly during install -- just reconnect
to `127.0.0.1:5900`.

### 2.8 Predict the real network interface name

After installation completes, stop QEMU with `Ctrl+C` in the SSH
terminal. **Do not reboot yet** -- the network interface name configured
by the installer is wrong (it matches the virtual NIC, not the real one).

Use the Hetzner `predict-check` tool to discover the real interface name:

```bash
predict-check
```

Example output:

```
eth0 -> enp0s31f6
```

Note the predicted name (e.g. `enp0s31f6`). You can also check the
current rescue interface for reference:

```bash
netdata
```

### 2.9 Fix the network config before first real boot

Boot Proxmox again in QEMU, **without** the ISO (no `-cdrom` flag):

```bash
qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -m 16G \
    -boot d \
    -drive file=/dev/nvme0n1,format=raw,if=virtio \
    -drive file=/dev/nvme1n1,format=raw,if=virtio \
    -bios /usr/share/OVMF/OVMF_CODE.fd \
    -vnc 127.0.0.1:0
```

Connect via VNC, log in as root, and edit the network configuration:

```bash
nano /etc/network/interfaces
```

Replace the virtual interface name (e.g. `ens18`) with the predicted
real name (e.g. `enp0s31f6`). A minimal working config:

```
auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet static
    address <public-ip>/32
    gateway <gateway-ip>
```

Save and shut down the VM (`shutdown -h now` inside the VNC session,
or `Ctrl+C` in the SSH terminal).

### 2.10 Reboot into Proxmox

Exit the rescue system and reboot the server from the Hetzner Robot
panel (or just `reboot` from SSH). The server now boots from disk
into Proxmox with the correct network configuration.

### 2.11 Verify

```bash
ssh root@<public-ip> pvesh get /version
# Should show Proxmox VE version and API info
```

> **Alternative method**: You can also install Debian 13 via Hetzner's
> `installimage` and then upgrade to Proxmox following the
> [official guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie).
> This skips the QEMU/VNC process but does not offer ZFS-on-root from
> the installer.

---

## 3. SSH setup and handoff to proxmox-setup

At this point Proxmox is installed and reachable via its public IP.
The remaining configuration (repos, private networking, storage,
WireGuard, firewall, API tokens) is handled by the `proxmox-setup`
script. You just need SSH key access and an alias.

### 3.1 Copy your SSH key

```bash
ssh-copy-id root@<public-ip>
```

### 3.2 Create an SSH config alias

Add to `~/.ssh/config` on your workstation:

```
Host hetzner-lab
    HostName <public-ip>
    User root
    IdentityFile ~/.ssh/id_ed25519
```

### 3.3 Verify SSH alias works

```bash
ssh hetzner-lab pvesh get /version
```

This must succeed before continuing. If it fails, check your SSH key
and config entry.

### 3.4 Run proxmox-setup

```bash
# Preview what will be configured
hetzner/proxmox-setup --host hetzner-lab --dry-run

# Run for real (interactive, prompts before each step)
hetzner/proxmox-setup --host hetzner-lab
```

The script walks through each step interactively:

1. **Detect** -- reads host info, block devices, existing config
2. **Repos** -- switches to no-subscription repos, removes nag popup
3. **Update** -- `apt update && apt dist-upgrade`
4. **Bridge** -- creates vmbr1 (private bridge, 10.10.0.0/24) with NAT
5. **Storage** -- detects unused disks, creates ZFS pool, registers with PVE
6. **WireGuard** -- installs and configures tunnel, prints client config
7. **Firewall** -- locks down public interface (SSH + WireGuard only)
8. **API token** -- creates role, pool, and token for `incusos-proxmox`

Each step prompts for confirmation. Use `--yes` to skip prompts, or
`--skip-repos`, `--skip-storage`, etc. to skip individual steps.

At the end, the script prints:
- A ready-to-use `proxmox.yaml` config for `incusos/targets/hetzner/`
- An `env` file snippet with the API token
- A WireGuard client config to save on your workstation

### 3.5 Connect WireGuard and verify

Save the WireGuard client config printed by the script:

```bash
# Linux
sudo wg-quick up hetzner-lab

# macOS -- import into the WireGuard app
```

Verify access through the tunnel:

```bash
ping 10.10.0.1                          # Proxmox host via WireGuard
curl -sk https://10.10.0.1:8006         # Web UI via WireGuard
```

Optionally add a second SSH alias for access via WireGuard:

```
Host hetzner-lab-wg
    HostName 10.10.0.1
    User root
    IdentityFile ~/.ssh/id_ed25519
```

### 3.6 Test deploy

```bash
# Copy and fill in the connection config
cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml

# Set the API token
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup output>"

# Dry run -- verify correct bridge, IPs, storage
incusos-proxmox --dry-run \
    --proxmox incusos/targets/hetzner/proxmox.yaml \
    incusos/targets/hetzner/lab-cluster.yaml
```

Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`
storage, 8 cores per VM.

---

## Network architecture (reference)

Once `proxmox-setup` has completed, the network looks like this:

```
Internet
    |
    | Public IP (e.g. 5.9.x.x)
    v
[vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24]
                  |                       |
                  |                  WireGuard tunnel
                  |                       |
             [vmbr1: 10.10.0.1/24]   Your workstation
                  |                  (10.10.99.2)
            ┌─────┼─────┐
            |     |     |
          VM-01 VM-02 VM-03
          .101  .102  .103
```

| Interface | Purpose |
|-----------|---------|
| vmbr0 | Public (SSH + WireGuard only after firewall) |
| vmbr1 | Private bridge for VMs (10.10.0.0/24, NAT to internet) |
| wg0 | WireGuard tunnel (10.10.99.0/24) |

AllowedIPs uses `10.10.0.0/16` to leave room for future subnets
(OVN overlay at 10.10.10.0/24, etc.) without updating WireGuard config.