# Hetzner Dedicated Server: Proxmox Setup Guide Complete guide for turning a Hetzner bare metal server into a Proxmox host ready for IncusOS lab deployments. Each section can be done manually or automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)). --- ## 1. Server selection The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used dedicated servers at significant discounts. Look for: - **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested virtualization is more proven with Proxmox/IncusOS) - **Cores**: 32+ (each IncusOS VM wants 4-8 cores) - **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB) - **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage) - **Network**: 1 Gbit/s included, single IPv4 Order the server and wait for provisioning (usually minutes for auction servers). Note the assigned IP address and temporary root password. --- ## 2. Proxmox installation We use the official Proxmox VE ISO installer running inside QEMU from the Hetzner rescue system. This gives the full graphical installer with ZFS support, which is not available via Hetzner's `installimage`. The method is documented in the [Hetzner community tutorial](https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve). ### 2.1 Boot the rescue system In the Hetzner Robot panel, activate the rescue system (Linux 64-bit) and reboot the server. Then SSH in: ```bash ssh root@ ``` ### 2.2 Discover disks Identify which disks the server has: ```bash lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE ``` Example output on a typical auction server with 2 NVMe drives: ``` NAME SIZE MODEL ROTA TYPE nvme0n1 477G Samsung SSD 970 EVO 0 disk nvme1n1 477G Samsung SSD 970 EVO 0 disk ``` For ZFS RAID1 (mirror), you want two matching disks. NVMe pairs are the strong favourite -- fast and reliable. Note the device names (`/dev/nvme0n1`, `/dev/nvme1n1`) for the QEMU command. If you have additional disks beyond the pair (e.g. large SATA drives), those can be set up as a separate storage pool later (section 6). Only pass the system disks to QEMU for now. ### 2.3 Check BIOS mode Determine whether the server boots in UEFI or legacy BIOS mode: ```bash [ -d "/sys/firmware/efi" ] && echo "UEFI" || echo "BIOS" ``` Most modern Hetzner servers are UEFI. The QEMU command below includes the OVMF BIOS line for UEFI -- remove it if your server reports BIOS. ### 2.4 Download the Proxmox ISO Get the latest Proxmox VE ISO from the official download page: ```bash wget https://enterprise.proxmox.com/iso/proxmox-ve_9.0-1.iso ``` Check [proxmox.com/en/downloads](https://www.proxmox.com/en/downloads) for the current version and adjust the URL accordingly. ### 2.5 Set up SSH port forwarding for VNC On your **workstation** (not the server), open an SSH tunnel forwarding the VNC port: ```bash ssh -L 5900:localhost:5900 root@ ``` This forwards local port 5900 to the server's localhost:5900, where QEMU will expose its VNC display. Keep this session open. ### 2.6 Start the QEMU installer In the SSH session on the server, start QEMU with the ISO and the disks you identified in step 2.2: ```bash qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -m 16G \ -boot d \ -cdrom ./proxmox-ve_9.0-1.iso \ -drive file=/dev/nvme0n1,format=raw,if=virtio \ -drive file=/dev/nvme1n1,format=raw,if=virtio \ -bios /usr/share/OVMF/OVMF_CODE.fd \ -vnc 127.0.0.1:0 ``` **Notes:** - Remove the `-bios /usr/share/OVMF/OVMF_CODE.fd \` line for legacy BIOS servers (step 2.3). - For SATA drives, use `/dev/sda`, `/dev/sdb` instead of `/dev/nvmeXn1`. - Disks appear as `/dev/vdX` inside the VM because of the virtio interface -- this is normal and expected. - The `-vnc 127.0.0.1:0` flag binds VNC to localhost only (safe, no password needed since it's behind the SSH tunnel). ### 2.7 Connect via VNC and install Open a VNC client on your workstation and connect to `127.0.0.1:5900` (or just `127.0.0.1` -- most clients default to port 5900). The Proxmox graphical installer appears. Walk through it: 1. Accept the EULA. 2. **Target disk**: Select the ZFS RAID1 (mirror) option across both drives (`/dev/vda` and `/dev/vdb` in the VM -- these are your NVMe drives passed through via virtio). 3. **Country/timezone/keyboard**: Set as appropriate. 4. **Root password and email**: Set a strong root password. This becomes `HETZNER_PROXMOX_ROOT_PASSWORD` in your `env` file. 5. **Network**: The installer shows a virtualized NIC. Configure it with the server's public IP, gateway, and hostname. This will be corrected in the next step since the real NIC name differs. 6. Click **Install** and wait for completion. The VNC client may disconnect briefly during install -- just reconnect to `127.0.0.1:5900`. ### 2.8 Predict the real network interface name After installation completes, stop QEMU with `Ctrl+C` in the SSH terminal. **Do not reboot yet** -- the network interface name configured by the installer is wrong (it matches the virtual NIC, not the real one). Use the Hetzner `predict-check` tool to discover the real interface name: ```bash predict-check ``` Example output: ``` eth0 -> enp0s31f6 ``` Note the predicted name (e.g. `enp0s31f6`). You can also check the current rescue interface for reference: ```bash netdata ``` ### 2.9 Fix the network config before first real boot Boot Proxmox again in QEMU, **without** the ISO (no `-cdrom` flag): ```bash qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -m 16G \ -boot d \ -drive file=/dev/nvme0n1,format=raw,if=virtio \ -drive file=/dev/nvme1n1,format=raw,if=virtio \ -bios /usr/share/OVMF/OVMF_CODE.fd \ -vnc 127.0.0.1:0 ``` Connect via VNC, log in as root, and edit the network configuration: ```bash nano /etc/network/interfaces ``` Replace the virtual interface name (e.g. `ens18`) with the predicted real name (e.g. `enp0s31f6`). A minimal working config: ``` auto lo iface lo inet loopback auto enp0s31f6 iface enp0s31f6 inet static address /32 gateway ``` Save and shut down the VM (`shutdown -h now` inside the VNC session, or `Ctrl+C` in the SSH terminal). ### 2.10 Reboot into Proxmox Exit the rescue system and reboot the server from the Hetzner Robot panel (or just `reboot` from SSH). The server now boots from disk into Proxmox with the correct network configuration. ### 2.11 Verify ```bash ssh root@ pvesh get /version # Should show Proxmox VE version and API info ``` The web UI is available at `https://:8006` (we'll lock this down to WireGuard-only in section 9). > **Alternative method**: You can also install Debian 13 via Hetzner's > `installimage` and then upgrade to Proxmox following the > [official guide](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie). > This skips the QEMU/VNC process but does not offer ZFS-on-root from > the installer. --- ## 3. Network configuration Hetzner dedicateds get a single public IP with MAC filtering -- you cannot assign additional public IPs to VMs without ordering extra IPs. Instead, we create a private bridge and NAT. ### Create the private bridge Add `vmbr1` to `/etc/network/interfaces`: ``` auto vmbr1 iface vmbr1 inet static address 10.10.0.1/24 bridge-ports none bridge-stp off bridge-fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE ``` This creates a private bridge where: - VMs connect to `vmbr1` and get IPs in 10.10.0.0/24 - The Proxmox host (10.10.0.1) is the default gateway - NAT masquerading gives VMs internet access through the host's public IP - IP forwarding is enabled automatically on bridge up ### Persist IP forwarding Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence: ``` net.ipv4.ip_forward = 1 ``` ### Apply ```bash ifreload -a # Apply network changes without reboot sysctl -p /etc/sysctl.d/99-forward.conf ``` ### Verify ```bash ip addr show vmbr1 # Should show 10.10.0.1/24 cat /proc/sys/net/ipv4/ip_forward # Should show 1 iptables -t nat -L POSTROUTING -n # Should show MASQUERADE rule ``` ### Network diagram ``` Internet | | Public IP (5.9.x.x) v [vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24] | | | WireGuard tunnel | | [vmbr1: 10.10.0.1/24] Your workstation | (10.10.99.2) ┌─────┼─────┐ | | | VM-01 VM-02 VM-03 .101 .102 .103 ``` --- ## 4. DNS setup (optional) If you have a domain, create an A record pointing to the public IP: ``` pve.example.com → 5.9.x.x ``` This is convenient but not required -- everything is accessed via WireGuard using private IPs anyway. If you set a hostname, update `/etc/hosts`: ``` 10.10.0.1 pve.example.com pve ``` And set the hostname: ```bash hostnamectl set-hostname pve ``` --- ## 5. SSH hardening ### Copy your SSH key ```bash ssh-copy-id root@ ``` ### Disable password authentication Edit `/etc/ssh/sshd_config`: ``` PermitRootLogin prohibit-password PasswordAuthentication no ``` Restart: ```bash systemctl restart sshd ``` ### Add a colleague's key ```bash # From their workstation: ssh-copy-id root@ # Or manually append to /root/.ssh/authorized_keys ``` ### SSH config for convenient access Add to `~/.ssh/config` on your workstation: ``` Host hetzner-lab HostName 5.9.x.x # Public IP (or pve.example.com) User root IdentityFile ~/.ssh/id_ed25519 ``` After WireGuard is set up (section 8), add a second entry: ``` Host hetzner-lab-wg HostName 10.10.0.1 User root IdentityFile ~/.ssh/id_ed25519 ``` ### Verify ```bash ssh hetzner-lab pvesh get /version ``` --- ## 6. Disk and storage setup ### Identify disks ```bash lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE ``` Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from install), 2+ additional disks for VM storage. ### Check existing pools ```bash zpool status # System mirror pvesm status # Proxmox storage backends ``` ### Create VM storage pool If you have additional disks (e.g. `sda`, `sdb`) not used by the system: ```bash # Mirror (2 disks) -- recommended zpool create local-zfs mirror /dev/sda /dev/sdb # RAIDZ1 (3+ disks) -- more space, still redundant zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc # Single disk (no redundancy) zpool create local-zfs /dev/sda ``` Register with Proxmox: ```bash pvesm add zfspool local-zfs -pool local-zfs pvesm set local-zfs -content images,rootdir ``` ### Verify ```bash pvesm status # Should list local-zfs with type zfspool ``` If the system install already created a suitable ZFS pool, skip pool creation and just register it with Proxmox if not already visible. --- ## 7. Repositories and system update ### Switch to no-subscription repositories The default enterprise repos require a paid subscription. Switch to community repos: ```bash # Disable enterprise repo sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list # Add no-subscription repo echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \ > /etc/apt/sources.list.d/pve-no-subscription.list # Disable Ceph enterprise repo if present if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list fi ``` ### Remove subscription nag popup ```bash # Standard JS patch for the web UI subscription dialog sed -Ezi.bak \ "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \ /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js systemctl restart pveproxy ``` ### Update ```bash apt update && apt dist-upgrade -y ``` ### Verify ```bash apt update 2>&1 | tail -5 # Should show no errors, all repos reachable ``` --- ## 8. WireGuard tunnel WireGuard provides secure access to VMs and the Proxmox web UI from your workstation, without exposing anything on the public interface. ### Install ```bash apt install -y wireguard ``` ### Generate server keys ```bash wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key chmod 600 /etc/wireguard/server-private.key ``` ### Generate client keys (on your workstation) ```bash wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key ``` ### Server config Create `/etc/wireguard/wg0.conf`: ```ini [Interface] PrivateKey = Address = 10.10.99.1/24 ListenPort = 51820 # Allow forwarding between WireGuard clients and the private VM bridge PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT [Peer] # Workstation PublicKey = AllowedIPs = 10.10.99.2/32 ``` ### Client config (on your workstation) Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app): ```ini [Interface] PrivateKey = Address = 10.10.99.2/24 [Peer] PublicKey = Endpoint = :51820 AllowedIPs = 10.10.0.0/16 PersistentKeepalive = 25 ``` **AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.) without needing to update the WireGuard config. ### Enable and start On the server: ```bash systemctl enable --now wg-quick@wg0 ``` On the workstation: ```bash # Linux sudo wg-quick up hetzner-lab # macOS (WireGuard app) # Import hetzner-lab.conf and activate ``` ### Verify ```bash # On the server wg show # From your workstation ping 10.10.99.1 # WireGuard interface ping 10.10.0.1 # Private bridge (should work -- routed via WireGuard) ``` ### Adding more peers To add a colleague, generate a new keypair and add a `[Peer]` block to the server config: ```ini [Peer] # Colleague name PublicKey = AllowedIPs = 10.10.99.3/32 ``` Then reload: ```bash systemctl restart wg-quick@wg0 ``` Give them a client config with `Address = 10.10.99.3/24` and the server's public key. --- ## 9. Firewall lockdown After WireGuard is working, lock down the public interface so only SSH and WireGuard are accessible from the internet. Proxmox web UI and VM traffic go through the tunnel only. ### nftables rules Create `/etc/nftables-hetzner.conf`: ``` #!/usr/sbin/nft -f flush ruleset table inet filter { chain input { type filter hook input priority 0; policy drop; # Loopback iif lo accept # Established/related connections ct state established,related accept # ICMP (ping) ip protocol icmp accept ip6 nexthdr icmpv6 accept # SSH on public interface iifname "vmbr0" tcp dport 22 accept # WireGuard on public interface iifname "vmbr0" udp dport 51820 accept # Allow everything on private bridge and WireGuard iifname "vmbr1" accept iifname "wg0" accept # Log and drop everything else log prefix "nft-drop: " limit rate 5/minute counter drop } chain forward { type filter hook forward priority 0; policy drop; # Established/related ct state established,related accept # WireGuard <-> private bridge iifname "wg0" oifname "vmbr1" accept iifname "vmbr1" oifname "wg0" accept # Private bridge -> internet (NAT) iifname "vmbr1" oifname "vmbr0" accept } chain output { type filter hook output priority 0; policy accept; } } ``` ### Apply ```bash nft -f /etc/nftables-hetzner.conf ``` ### Persist across reboots ```bash cp /etc/nftables-hetzner.conf /etc/nftables.conf systemctl enable nftables ``` ### Verify From an external machine (not through WireGuard): ```bash # Should work ssh hetzner-lab echo ok # Should time out (port 8006 blocked on public interface) curl -sk --connect-timeout 5 https://:8006 # Through WireGuard -- should work curl -sk https://10.10.0.1:8006 ``` **Warning**: Test SSH access through WireGuard *before* applying firewall rules. If you lock yourself out, use Hetzner's rescue system to recover. --- ## 10. API token setup Create a dedicated API token for `incusos-proxmox` automation. ### Create role and user ```bash # Create role with required privileges pveum role add IncusOSDeployer -privs \ "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ VM.Monitor VM.Audit VM.Console \ Datastore.AllocateSpace Datastore.Audit \ Pool.Allocate Pool.Audit \ SDN.Use Sys.Modify" # Create resource pool pveum pool add IncusLab -comment "IncusOS lab VMs" # Create API token pveum user token add automation@pve deploy --privsep 0 # Save the displayed token secret -- it's shown only once! # Assign role to user on the pool pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer # Assign role on storage pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer ``` ### Record credentials Add to the `env` file at the repo root: ```bash # Hetzner HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password ``` ### Verify ```bash # Export for incusos-proxmox export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" # Test API access (update host/node in proxmox-api or use curl directly) curl -sk "https://10.10.0.1:8006/api2/json/version" \ -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET" ``` --- ## 11. Final verification checklist | Check | Command | Expected | |-------|---------|----------| | Proxmox version | `pvesh get /version` | PVE 8.x | | Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 | | IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 | | NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 | | ZFS storage | `pvesm status` | local-zfs available | | WireGuard | `wg show` | 1+ peer, handshake recent | | SSH via WG | `ssh hetzner-lab-wg hostname` | Responds | | Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response | | API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version | | Public lockdown | `curl --connect-timeout 5 https://:8006` | Timeout | ### Test deploy ```bash export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" # Dry run -- verify correct bridge, IPs, storage incusos-proxmox --dry-run \ --proxmox incusos/targets/hetzner/proxmox.yaml \ incusos/targets/hetzner/lab-cluster.yaml ``` Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs` storage, 8 cores per VM.