# Hetzner Dedicated Server: Proxmox Setup Guide Complete guide for turning a Hetzner bare metal server into a Proxmox host ready for IncusOS lab deployments. Each section can be done manually or automated with `proxmox-setup` (see [proxmox-setup](proxmox-setup)). --- ## 1. Server selection The [Hetzner Server Auction](https://www.hetzner.com/sb/) offers used dedicated servers at significant discounts. Look for: - **CPU**: Intel with VT-x and VT-d (AMD works too, but Intel nested virtualization is more proven with Proxmox/IncusOS) - **Cores**: 32+ (each IncusOS VM wants 4-8 cores) - **RAM**: 128+ GiB (256 GiB ideal -- 4 VMs at 16 GiB still leaves 192 GiB) - **Disks**: 2+ NVMe or SSD (ZFS mirror for system, extras for VM storage) - **Network**: 1 Gbit/s included, single IPv4 Order the server and wait for provisioning (usually minutes for auction servers). Note the assigned IP address and temporary root password. --- ## 2. Proxmox installation Hetzner provides two installation paths: ### Option A: installimage (recommended) Boot the rescue system from the Hetzner Robot panel, then SSH in: ```bash ssh root@ installimage ``` Select Proxmox VE from the list. The installer presents an editor for disk configuration -- set up ZFS RAID1 mirror on two system disks: ``` SWRAID 0 SWRAIDLEVEL 1 PART /boot ext4 1G PART lvm pve all LV pve root ext4 100G LV pve swap swap 16G ``` Reboot after installation completes. ### Option B: ISO install Upload the Proxmox VE ISO via Robot panel → Server → ISO Images, then boot from it. Follow the standard Proxmox installer. This gives more control over partitioning but requires KVM console access. ### Verify ```bash ssh root@ pvesh get /version # Should show Proxmox VE version and API info ``` The web UI is available at `https://:8006` (we'll lock this down to WireGuard-only in section 9). --- ## 3. Network configuration Hetzner dedicateds get a single public IP with MAC filtering -- you cannot assign additional public IPs to VMs without ordering extra IPs. Instead, we create a private bridge and NAT. ### Create the private bridge Add `vmbr1` to `/etc/network/interfaces`: ``` auto vmbr1 iface vmbr1 inet static address 10.10.0.1/24 bridge-ports none bridge-stp off bridge-fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 10.10.0.0/24 -o vmbr0 -j MASQUERADE ``` This creates a private bridge where: - VMs connect to `vmbr1` and get IPs in 10.10.0.0/24 - The Proxmox host (10.10.0.1) is the default gateway - NAT masquerading gives VMs internet access through the host's public IP - IP forwarding is enabled automatically on bridge up ### Persist IP forwarding Also add to `/etc/sysctl.d/99-forward.conf` for boot persistence: ``` net.ipv4.ip_forward = 1 ``` ### Apply ```bash ifreload -a # Apply network changes without reboot sysctl -p /etc/sysctl.d/99-forward.conf ``` ### Verify ```bash ip addr show vmbr1 # Should show 10.10.0.1/24 cat /proc/sys/net/ipv4/ip_forward # Should show 1 iptables -t nat -L POSTROUTING -n # Should show MASQUERADE rule ``` ### Network diagram ``` Internet | | Public IP (5.9.x.x) v [vmbr0] ──── Proxmox host ──── [wg0: 10.10.99.1/24] | | | WireGuard tunnel | | [vmbr1: 10.10.0.1/24] Your workstation | (10.10.99.2) ┌─────┼─────┐ | | | VM-01 VM-02 VM-03 .101 .102 .103 ``` --- ## 4. DNS setup (optional) If you have a domain, create an A record pointing to the public IP: ``` pve.example.com → 5.9.x.x ``` This is convenient but not required -- everything is accessed via WireGuard using private IPs anyway. If you set a hostname, update `/etc/hosts`: ``` 10.10.0.1 pve.example.com pve ``` And set the hostname: ```bash hostnamectl set-hostname pve ``` --- ## 5. SSH hardening ### Copy your SSH key ```bash ssh-copy-id root@ ``` ### Disable password authentication Edit `/etc/ssh/sshd_config`: ``` PermitRootLogin prohibit-password PasswordAuthentication no ``` Restart: ```bash systemctl restart sshd ``` ### Add a colleague's key ```bash # From their workstation: ssh-copy-id root@ # Or manually append to /root/.ssh/authorized_keys ``` ### SSH config for convenient access Add to `~/.ssh/config` on your workstation: ``` Host hetzner-lab HostName 5.9.x.x # Public IP (or pve.example.com) User root IdentityFile ~/.ssh/id_ed25519 ``` After WireGuard is set up (section 8), add a second entry: ``` Host hetzner-lab-wg HostName 10.10.0.1 User root IdentityFile ~/.ssh/id_ed25519 ``` ### Verify ```bash ssh hetzner-lab pvesh get /version ``` --- ## 6. Disk and storage setup ### Identify disks ```bash lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE ``` Typical Hetzner setup: 2x NVMe for system (already in ZFS mirror from install), 2+ additional disks for VM storage. ### Check existing pools ```bash zpool status # System mirror pvesm status # Proxmox storage backends ``` ### Create VM storage pool If you have additional disks (e.g. `sda`, `sdb`) not used by the system: ```bash # Mirror (2 disks) -- recommended zpool create local-zfs mirror /dev/sda /dev/sdb # RAIDZ1 (3+ disks) -- more space, still redundant zpool create local-zfs raidz1 /dev/sda /dev/sdb /dev/sdc # Single disk (no redundancy) zpool create local-zfs /dev/sda ``` Register with Proxmox: ```bash pvesm add zfspool local-zfs -pool local-zfs pvesm set local-zfs -content images,rootdir ``` ### Verify ```bash pvesm status # Should list local-zfs with type zfspool ``` If the system install already created a suitable ZFS pool, skip pool creation and just register it with Proxmox if not already visible. --- ## 7. Repositories and system update ### Switch to no-subscription repositories The default enterprise repos require a paid subscription. Switch to community repos: ```bash # Disable enterprise repo sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list # Add no-subscription repo echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \ > /etc/apt/sources.list.d/pve-no-subscription.list # Disable Ceph enterprise repo if present if [[ -f /etc/apt/sources.list.d/ceph.list ]]; then sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list fi ``` ### Remove subscription nag popup ```bash # Standard JS patch for the web UI subscription dialog sed -Ezi.bak \ "s/(Ext\.Msg\.show\(\{[^}]+title: gettext\('No valid sub)/void\(\{ \/\/\1/" \ /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js systemctl restart pveproxy ``` ### Update ```bash apt update && apt dist-upgrade -y ``` ### Verify ```bash apt update 2>&1 | tail -5 # Should show no errors, all repos reachable ``` --- ## 8. WireGuard tunnel WireGuard provides secure access to VMs and the Proxmox web UI from your workstation, without exposing anything on the public interface. ### Install ```bash apt install -y wireguard ``` ### Generate server keys ```bash wg genkey | tee /etc/wireguard/server-private.key | wg pubkey > /etc/wireguard/server-public.key chmod 600 /etc/wireguard/server-private.key ``` ### Generate client keys (on your workstation) ```bash wg genkey | tee wg-client-private.key | wg pubkey > wg-client-public.key ``` ### Server config Create `/etc/wireguard/wg0.conf`: ```ini [Interface] PrivateKey = Address = 10.10.99.1/24 ListenPort = 51820 # Allow forwarding between WireGuard clients and the private VM bridge PostUp = iptables -A FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -A FORWARD -i vmbr1 -o wg0 -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o vmbr1 -j ACCEPT; iptables -D FORWARD -i vmbr1 -o wg0 -j ACCEPT [Peer] # Workstation PublicKey = AllowedIPs = 10.10.99.2/32 ``` ### Client config (on your workstation) Save as `/etc/wireguard/hetzner-lab.conf` (or import into your WireGuard app): ```ini [Interface] PrivateKey = Address = 10.10.99.2/24 [Peer] PublicKey = Endpoint = :51820 AllowedIPs = 10.10.0.0/16 PersistentKeepalive = 25 ``` **AllowedIPs note**: Using `10.10.0.0/16` instead of just the VM subnet leaves room for future subnets (OVN overlay at 10.10.10.0/24, etc.) without needing to update the WireGuard config. ### Enable and start On the server: ```bash systemctl enable --now wg-quick@wg0 ``` On the workstation: ```bash # Linux sudo wg-quick up hetzner-lab # macOS (WireGuard app) # Import hetzner-lab.conf and activate ``` ### Verify ```bash # On the server wg show # From your workstation ping 10.10.99.1 # WireGuard interface ping 10.10.0.1 # Private bridge (should work -- routed via WireGuard) ``` ### Adding more peers To add a colleague, generate a new keypair and add a `[Peer]` block to the server config: ```ini [Peer] # Colleague name PublicKey = AllowedIPs = 10.10.99.3/32 ``` Then reload: ```bash systemctl restart wg-quick@wg0 ``` Give them a client config with `Address = 10.10.99.3/24` and the server's public key. --- ## 9. Firewall lockdown After WireGuard is working, lock down the public interface so only SSH and WireGuard are accessible from the internet. Proxmox web UI and VM traffic go through the tunnel only. ### nftables rules Create `/etc/nftables-hetzner.conf`: ``` #!/usr/sbin/nft -f flush ruleset table inet filter { chain input { type filter hook input priority 0; policy drop; # Loopback iif lo accept # Established/related connections ct state established,related accept # ICMP (ping) ip protocol icmp accept ip6 nexthdr icmpv6 accept # SSH on public interface iifname "vmbr0" tcp dport 22 accept # WireGuard on public interface iifname "vmbr0" udp dport 51820 accept # Allow everything on private bridge and WireGuard iifname "vmbr1" accept iifname "wg0" accept # Log and drop everything else log prefix "nft-drop: " limit rate 5/minute counter drop } chain forward { type filter hook forward priority 0; policy drop; # Established/related ct state established,related accept # WireGuard <-> private bridge iifname "wg0" oifname "vmbr1" accept iifname "vmbr1" oifname "wg0" accept # Private bridge -> internet (NAT) iifname "vmbr1" oifname "vmbr0" accept } chain output { type filter hook output priority 0; policy accept; } } ``` ### Apply ```bash nft -f /etc/nftables-hetzner.conf ``` ### Persist across reboots ```bash cp /etc/nftables-hetzner.conf /etc/nftables.conf systemctl enable nftables ``` ### Verify From an external machine (not through WireGuard): ```bash # Should work ssh hetzner-lab echo ok # Should time out (port 8006 blocked on public interface) curl -sk --connect-timeout 5 https://:8006 # Through WireGuard -- should work curl -sk https://10.10.0.1:8006 ``` **Warning**: Test SSH access through WireGuard *before* applying firewall rules. If you lock yourself out, use Hetzner's rescue system to recover. --- ## 10. API token setup Create a dedicated API token for `incusos-proxmox` automation. ### Create role and user ```bash # Create role with required privileges pveum role add IncusOSDeployer -privs \ "VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \ VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \ VM.Monitor VM.Audit VM.Console \ Datastore.AllocateSpace Datastore.Audit \ Pool.Allocate Pool.Audit \ SDN.Use Sys.Modify" # Create resource pool pveum pool add IncusLab -comment "IncusOS lab VMs" # Create API token pveum user token add automation@pve deploy --privsep 0 # Save the displayed token secret -- it's shown only once! # Assign role to user on the pool pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer # Assign role on storage pveum acl modify /storage/local-zfs -user automation@pve -role IncusOSDeployer pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer ``` ### Record credentials Add to the `env` file at the repo root: ```bash # Hetzner HETZNER_PROXMOX_TOKEN_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx HETZNER_PROXMOX_ROOT_PASSWORD=your-root-password ``` ### Verify ```bash # Export for incusos-proxmox export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" # Test API access (update host/node in proxmox-api or use curl directly) curl -sk "https://10.10.0.1:8006/api2/json/version" \ -H "Authorization: PVEAPIToken=automation@pve!deploy=$PROXMOX_TOKEN_SECRET" ``` --- ## 11. Final verification checklist | Check | Command | Expected | |-------|---------|----------| | Proxmox version | `pvesh get /version` | PVE 8.x | | Private bridge | `ip addr show vmbr1` | 10.10.0.1/24 | | IP forwarding | `cat /proc/sys/net/ipv4/ip_forward` | 1 | | NAT | `iptables -t nat -L -n` | MASQUERADE for 10.10.0.0/24 | | ZFS storage | `pvesm status` | local-zfs available | | WireGuard | `wg show` | 1+ peer, handshake recent | | SSH via WG | `ssh hetzner-lab-wg hostname` | Responds | | Web UI via WG | `curl -sk https://10.10.0.1:8006` | HTML response | | API token | `curl -sk .../api2/json/version -H Auth...` | JSON with version | | Public lockdown | `curl --connect-timeout 5 https://:8006` | Timeout | ### Test deploy ```bash export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET" # Dry run -- verify correct bridge, IPs, storage incusos-proxmox --dry-run \ --proxmox incusos/targets/hetzner/proxmox.yaml \ incusos/targets/hetzner/lab-cluster.yaml ``` Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs` storage, 8 cores per VM.