diff --git a/CLAUDE.md b/CLAUDE.md index f745d22..0e60e66 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,6 +28,7 @@ incu-contrib/ └── notes/ # Research notes and reference material ├── clustering-guide.md # Detailed Incus clustering walkthrough ├── operations-center-guide.md # Operations Center provisioning & management + ├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN) ├── migration-guide.md # Migration paths into Incus from other hypervisors └── utm-support.md # UTM support design document (future) ``` @@ -117,6 +118,13 @@ incu-contrib/ - `balloon=0` -- IncusOS manages memory internally - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM - Minimum 50 GiB disk, minimum 4096 MiB RAM +- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag + to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM + on the tagged VLAN instead of the native/untagged network. The current lab + uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag, + VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the + Proxmox VM level only — IncusOS and Incus instances inside the VM are + unaware of it. The setting is optional: omit `vlan:` for untagged access. - **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs. IncusOS does **literal** string matching (not glob) on disk device IDs. `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit @@ -744,6 +752,54 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ - **No hardcoded package managers**: say "install the Incus client" with a link, not "sudo apt install incus". +### Incus networking (OVN) + +- **Bridge networks are node-local**: each cluster member has its own + independent bridge. Instances on the same bridge (same node) can + communicate; cross-node instances CANNOT. Each bridge has the same subnet + (e.g., 10.0.0.1/24) but they are separate L2 domains. +- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes. + Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane + + client services + physical uplink network. +- **IncusOS OVN services are disabled by default**: must be enabled via the + IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring + Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection` + fails with `db.sock not found`. +- **OVN service enable API call**: + ```bash + incus query :/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp::6642", + "enabled": true, + "tunnel_address": "", + "tunnel_protocol": "geneve" + }, + "state": {} + }' + ``` + `database` is the **southbound** DB (port 6642), NOT northbound (6641). +- **OVN control plane as container**: deploy `ovn-central` package in a + Debian container on the cluster. Use proxy devices to expose NB (6641) + and SB (6642) ports on the host's LAN IP so all nodes can reach it. +- **Setup sequence** (order matters): + 1. Deploy OVN control plane container + 2. Enable OVN services on ALL IncusOS nodes + 3. `incus config set network.ovn.northbound_connection tcp::6641` + 4. `incus cluster role add : ovn-chassis` (all nodes) + 5. Create physical uplink network (two-step cluster pattern) + 6. Create OVN network with `--type=ovn network=UPLINK` +- **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC), + `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, + `ipv4.gateway` is the LAN gateway in CIDR format. +- **OVN network isolation**: multiple OVN networks are fully isolated. + Instances on different networks cannot communicate, even on the same node. + Network peering (`incus network peer create`) enables cross-network routing. +- **OVN features tested**: cross-node connectivity, network isolation, + ACLs (per-source blocking), network peering, L4 load balancers (connection- + based hashing, not round-robin), network forwards (port forwarding to LAN + IPs), DNS resolution (per-network, hostname.incus domain). +- See `notes/networking-guide.md` for full tutorial with test results. + ### Migration into Incus - **`incus-migrate`**: official tool for importing disk images, running diff --git a/incusos/README.md b/incusos/README.md index f2f8e5e..4e3a6a6 100644 --- a/incusos/README.md +++ b/incusos/README.md @@ -389,6 +389,7 @@ node: pve storage: local-lvm iso_storage: local bridge: vmbr0 +vlan: 69 pool: IncusLab ssh_user: root ``` diff --git a/incusos/examples/lab-networking.yaml b/incusos/examples/lab-networking.yaml new file mode 100644 index 0000000..a481470 --- /dev/null +++ b/incusos/examples/lab-networking.yaml @@ -0,0 +1,42 @@ +# lab-networking.yaml - 4-node cluster for networking exploration +# +# Deploys 4 Incus nodes using an OC-provisioned ISO for a networking +# tutorial covering bridge, OVN overlay, ACLs, load balancers, and +# LAN integration. +# +# Usage: +# # First: deploy OC server with lab-oc-deploy.yaml +# # Then: create OC provisioning token + download ISO +# # Then deploy with --iso: +# incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run examples/lab-networking.yaml +# incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-networking.yaml +# incusos-proxmox --status examples/lab-networking.yaml +# incusos-proxmox --cleanup --deep examples/lab-networking.yaml +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# +# Resource budget: 4 nodes x 8 GiB = 32 GiB RAM +# Plus OC server (4 GiB) = 36 GiB total + +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 400 + +vms: + - name: net-node-01 + app: incus + apply_defaults: false + + - name: net-node-02 + app: incus + apply_defaults: false + + - name: net-node-03 + app: incus + apply_defaults: false + + - name: net-node-04 + app: incus + apply_defaults: false diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index a3b1799..c539aad 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -166,6 +166,7 @@ ${BOLD}PROXMOX CONNECTION CONFIG${RESET} storage: local-lvm # VM disk storage (default: local-lvm) iso_storage: local # ISO storage (default: local) bridge: vmbr0 # Network bridge (default: vmbr0) + vlan: 69 # VLAN tag for VM NIC (optional) pool: IncusLab # Resource pool (optional) ssh_user: root # SSH user (default: root) @@ -1294,6 +1295,7 @@ PVE_NODE="" PVE_STORAGE="" PVE_ISO_STORAGE="" PVE_BRIDGE="" +PVE_VLAN="" PVE_METHOD="" PVE_SSH_USER="" PVE_API_TOKEN_ID="" @@ -1393,6 +1395,7 @@ json.dump(result, sys.stdout, indent=2) val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val" val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" + val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val" val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" @@ -1415,6 +1418,7 @@ load_config() { val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" @@ -1852,7 +1856,9 @@ pve_create_vm() { cmd+=" --scsi0 ${PVE_STORAGE}:${disk_gib},iothread=1" cmd+=" --ide2 ${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom" cmd+=" --ide3 ${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom" - cmd+=" --net0 virtio,bridge=${PVE_BRIDGE}" + local net0_spec="virtio,bridge=${PVE_BRIDGE}" + [[ -n "$PVE_VLAN" ]] && net0_spec+=",tag=${PVE_VLAN}" + cmd+=" --net0 ${net0_spec}" cmd+=" --boot order=ide2\\;scsi0" cmd+=" --agent 1" if [[ -n "$PVE_POOL" ]]; then @@ -1892,10 +1898,14 @@ payload = { 'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1', 'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom', 'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom', - 'net0': 'virtio,bridge=${PVE_BRIDGE}', 'boot': 'order=ide2;scsi0', 'agent': 1, } +net0 = 'virtio,bridge=${PVE_BRIDGE}' +vlan = '${PVE_VLAN}' +if vlan: + net0 += ',tag=' + vlan +payload['net0'] = net0 pool = '${PVE_POOL}' if pool: payload['pool'] = pool @@ -2866,7 +2876,11 @@ print_plan() { echo -e " Proxmox host: ${CYAN}${PVE_HOST}${RESET} (${PVE_METHOD})" echo -e " Node: ${PVE_NODE}" echo -e " Storage: ${PVE_STORAGE} (disk), ${PVE_ISO_STORAGE} (iso)" - echo -e " Bridge: ${PVE_BRIDGE}" + if [[ -n "$PVE_VLAN" ]]; then + echo -e " Bridge: ${PVE_BRIDGE} (VLAN ${PVE_VLAN})" + else + echo -e " Bridge: ${PVE_BRIDGE}" + fi if [[ -n "$PVE_POOL" ]]; then echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}" fi diff --git a/notes/networking-guide.md b/notes/networking-guide.md new file mode 100644 index 0000000..a308b22 --- /dev/null +++ b/notes/networking-guide.md @@ -0,0 +1,799 @@ +# Incus Networking Guide — Bridge, OVN, and LAN Integration + +A hands-on tutorial covering Incus networking from basic bridge networking +to OVN overlay networks, tested on a 4-node IncusOS cluster managed by +Operations Center on Proxmox VE. + +## Prerequisites + +- 4-node IncusOS cluster (this guide uses `net-node-01` through `net-node-04`) +- Operations Center server (for OC-managed deployment) +- Proxmox VE host with `incusos-proxmox` deployment tool +- `incus` client configured with remotes for all cluster nodes + +### Lab configuration + +```yaml +# incusos/examples/lab-networking.yaml +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 400 + +vms: + - name: net-node-01 + app: incus + apply_defaults: false + - name: net-node-02 + app: incus + apply_defaults: false + - name: net-node-03 + app: incus + apply_defaults: false + - name: net-node-04 + app: incus + apply_defaults: false +``` + +**Resource budget:** OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total. + +### Cluster formation + +Nodes were deployed with `apply_defaults: false`, so no default storage pool, +network, or profile devices exist after installation. After forming the cluster +manually (OC's `cluster add` requires "needs update: false" which was blocked +by a v0.3.0 tracking issue), we created these resources: + +```bash +# Storage pool (per-member source, then cluster-wide finalize) +for node in net-node-01 net-node-02 net-node-03 net-node-04; do + incus storage create net-node-01:local zfs \ + source=local/incus zfs.pool_name=local/incus --target "$node" +done +incus storage create net-node-01:local zfs + +# Bridge network (per-member pending, then finalize with config) +for node in net-node-01 net-node-02 net-node-03 net-node-04; do + incus network create net-node-01:incusbr0 --type bridge --target "$node" +done +incus network create net-node-01:incusbr0 --type bridge \ + ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none + +# Default profile +incus profile device add net-node-01:default root disk path=/ pool=local +incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0 +``` + +--- + +## Part 1: Bridge Networking (Default) + +### What is bridge networking? + +When Incus creates a bridge network (type `bridge`), it creates a Linux bridge +on each cluster node. Each bridge is **independent** — there is no connectivity +between bridges on different nodes. This is the simplest networking mode and +works without any additional infrastructure. + +### Bridge topology + +``` +Node 1 (net-node-01) Node 2 (net-node-02) +┌─────────────────────┐ ┌─────────────────────┐ +│ incusbr0 │ │ incusbr0 │ +│ 10.0.0.1/24 │ │ 10.0.0.1/24 │ +│ ┌───┐ ┌───┐ │ │ ┌───┐ │ +│ │c1 │ │c2 │ │ │ │c3 │ │ +│ │.202│ │.40│ │ │ │.52│ │ +│ └───┘ └───┘ │ │ └───┘ │ +│ │ │ │ │ │ │ +│ ──┴──────┴── │ │ ──┴── │ +│ (bridge) │ │ (bridge) │ +│ │ │ │ │ │ +│ NAT ──> ens18 │ │ NAT ──> ens18 │ +│ 192.168.1.209 │ │ 192.168.1.150 │ +└─────────────────────┘ └─────────────────────┘ + │ │ + ────┴──────────────────────────────────┴──── + LAN (192.168.1.0/24) +``` + +**Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24). +The bridges are NOT connected to each other. + +### Test results + +#### Same-node communication: PASS + +Containers on the same node share the same bridge (same L2 domain): + +```bash +# test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01 +$ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40 +64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms +# 0% packet loss, ~0.03ms latency +``` + +#### NAT to internet: PASS + +The bridge has `ipv4.nat=true`, so containers can reach the internet: + +```bash +$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1 +64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms +# 0% packet loss, ~10ms to Cloudflare DNS +``` + +#### Cross-node communication: FAIL + +Containers on different nodes CANNOT reach each other: + +```bash +# test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52) +$ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52 +From 10.0.0.202 icmp_seq=1 Destination Host Unreachable +# 100% packet loss +``` + +### When to use bridge networking + +- Single-node setups (development, testing) +- Workloads that don't need cross-node communication +- Simple NAT-to-internet access +- As an uplink for OVN networks (see Part 2) + +### Bridge network configuration options + +```bash +# View current config +incus network show net-node-01:incusbr0 + +# Key settings +ipv4.address # Bridge IP and subnet (e.g., 10.0.0.1/24) +ipv4.nat # Enable NAT for internet access (true/false) +ipv4.dhcp # Enable DHCP server (true/false, default true) +ipv4.dhcp.ranges # DHCP range (e.g., 10.0.0.100-10.0.0.200) +dns.domain # DNS domain for instances (e.g., incus) +dns.mode # DNS mode: managed, dynamic, or none +ipv6.address # IPv6 address (set to "none" to disable) +``` + +--- + +## Part 2: OVN Overlay Networking + +### Why OVN? + +Bridge networking is node-local. To enable cross-node communication without +routing hacks, you need an overlay network. OVN (Open Virtual Network) +provides: + +- **Cross-node L2 connectivity** via Geneve tunnels +- **Network isolation** (multiple independent virtual networks) +- **Distributed routing** (no single point of failure) +- **Network ACLs** (firewall rules at the network level) +- **Load balancers** (built-in L4 load balancing) +- **Network forwards** (port forwarding from external IPs) +- **DNS** (automatic hostname resolution between instances) + +### OVN architecture on IncusOS + +IncusOS includes OVN **client** components only: +- `ovn-controller` (data plane) +- `ovs-vswitchd` (Open vSwitch) + +The OVN **control plane** must run separately: +- `ovn-northd` (translates logical to physical flows) +- `ovsdb-server` (northbound + southbound databases) + +Options for deploying the control plane: +1. As a container on the Incus cluster itself (used in this guide) +2. On an external host + +### OVN topology + +``` + ┌──────────────────────────────────┐ + │ OVN Control Plane │ + │ (container: ovn-central) │ + │ ovn-northd + ovsdb-server │ + │ NB: tcp:192.168.1.209:6641 │ + │ SB: tcp:192.168.1.209:6642 │ + └──────────┬───────────────────────┘ + │ (proxy devices) + ┌─────────────────────────┼─────────────────────────┐ + │ │ │ +Node 1 (net-node-01) Node 2 (net-node-02) Node 3 (net-node-03) +┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ +│ ovn-controller │ │ ovn-controller │ │ ovn-controller │ +│ ovs-vswitchd │ │ ovs-vswitchd │ │ ovs-vswitchd │ +│ │ │ │ │ │ +│ ┌───┐ ┌───┐ │ │ ┌───┐ │ │ ┌───┐ │ +│ │c1 │ │c2 │ │ │ │c3 │ │ │ │c4 │ │ +│ │.2 │ │.3 │ │ │ │.4 │ │ │ │.5 │ │ +│ └─┬─┘ └─┬─┘ │ │ └─┬─┘ │ │ └─┬─┘ │ +│ └──┬───┘ │ │ │ │ │ │ │ +│ OVN logical │ │ OVN logical │ │ OVN logical │ +│ switch │ │ switch │ │ switch │ +│ 10.10.10.0/24 │ │ 10.10.10.0/24 │ │ 10.10.10.0/24 │ +│ │ │ │ │ │ │ │ │ +│ Geneve tunnel ──┼──┼── Geneve tunnel ──┼──┼── Geneve tunnel │ +│ 192.168.1.209 │ │ 192.168.1.150 │ │ 192.168.1.13 │ +└───────────────────┘ └───────────────────┘ └───────────────────┘ + │ │ │ + ────┴──────────────────────┴──────────────────────┴──── + LAN (192.168.1.0/24) +``` + +**Key difference from bridge:** All instances share a single logical switch +connected via Geneve tunnels. Cross-node traffic is encapsulated and sent +over the LAN between nodes. + +### Step 1: Deploy OVN control plane + +Launch a container to run the OVN central services: + +```bash +# Launch OVN central container +incus launch images:debian/12 net-node-01:ovn-central --target net-node-01 + +# Install OVN central +incus exec net-node-01:ovn-central -- apt-get update -qq +incus exec net-node-01:ovn-central -- apt-get install -y ovn-central + +# Configure listening on all interfaces +incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641 +incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642 + +# Verify +incus exec net-node-01:ovn-central -- ovn-nbctl get-connection # ptcp:6641 +incus exec net-node-01:ovn-central -- ovn-sbctl get-connection # ptcp:6642 +``` + +The container runs on `incusbr0` (node-local bridge at 10.0.0.x). Other nodes +can't reach this IP directly. Use proxy devices to expose the ports on the +host IP: + +```bash +# Forward NB and SB ports from host to container +incus config device add net-node-01:ovn-central ovn-nb proxy \ + listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641 +incus config device add net-node-01:ovn-central ovn-sb proxy \ + listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642 +``` + +Now all nodes can reach the OVN databases via `tcp:192.168.1.209:6641` and +`tcp:192.168.1.209:6642`. + +### Step 2: Enable OVN on IncusOS nodes + +**Critical:** The OVN client services on IncusOS are disabled by default. +You MUST enable them on every node before configuring Incus. + +Without this step, `incus config set network.ovn.northbound_connection` fails: +``` +Error: failed to notify peer: Failed to connect to OVS: + failed to connect to unix:/run/openvswitch/db.sock: no such file or directory +``` + +Enable OVN via the IncusOS REST API on each node: + +```bash +# Enable OVN on each node (set database to SOUTHBOUND port 6642) +incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.1.209:6642", + "enabled": true, + "tunnel_address": "192.168.1.209", + "tunnel_protocol": "geneve" + }, + "state": {} +}' + +incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.1.209:6642", + "enabled": true, + "tunnel_address": "192.168.1.150", + "tunnel_protocol": "geneve" + }, + "state": {} +}' + +# Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169) +# with their respective tunnel_address values +``` + +Key fields: +- `database`: OVN **southbound** DB connection (port 6642, NOT 6641) +- `tunnel_address`: this node's LAN IP for Geneve tunnel encapsulation +- `tunnel_protocol`: `geneve` (default and recommended) +- `enabled`: `true` to start `ovsdb-server`, `ovs-vswitchd`, and `ovn-controller` + +Wait ~10 seconds for services to start, then verify: + +```bash +incus query net-node-01:/os/1.0/services/ovn +# Should show enabled: true +``` + +### Step 3: Configure Incus OVN connection + +Tell Incus where the OVN **northbound** database is: + +```bash +incus config set net-node-01: network.ovn.northbound_connection \ + tcp:192.168.1.209:6641 +``` + +This is a cluster-wide setting that propagates to all members. It WILL FAIL +if any member doesn't have OVS running (Step 2 not complete). + +Verify all chassis are registered: + +```bash +$ incus exec net-node-01:ovn-central -- ovn-sbctl show +Chassis "c1a92a50-..." + hostname: net-node-01 + Encap geneve + ip: "192.168.1.209" +Chassis "f4fb8f29-..." + hostname: net-node-02 + Encap geneve + ip: "192.168.1.150" +Chassis "88f8cd95-..." + hostname: net-node-03 + Encap geneve + ip: "192.168.1.13" +Chassis "6be485be-..." + hostname: net-node-04 + Encap geneve + ip: "192.168.1.169" +``` + +### Step 4: Assign ovn-chassis roles + +Designate which nodes participate in OVN gateway HA: + +```bash +for node in net-node-01 net-node-02 net-node-03 net-node-04; do + incus cluster role add net-node-01:"$node" ovn-chassis +done +``` + +### Step 5: Create physical uplink network + +OVN networks need an uplink for external/LAN connectivity: + +```bash +# Per-member definition (two-step cluster pattern) +for node in net-node-01 net-node-02 net-node-03 net-node-04; do + incus network create net-node-01:UPLINK --type=physical \ + parent=ens18 --target "$node" +done + +# Finalize with external IP range and gateway +incus network create net-node-01:UPLINK --type=physical \ + ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \ + ipv4.gateway=192.168.1.1/24 \ + dns.nameservers=192.168.1.1 +``` + +- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox) +- `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses +- `ipv4.gateway`: LAN gateway in CIDR format + +### Step 6: Create OVN network + +```bash +incus network create net-node-01:ovn-net1 --type=ovn \ + network=UPLINK \ + ipv4.address=10.10.10.1/24 \ + ipv4.nat=true \ + ipv6.address=none +``` + +Incus automatically creates: +- **Logical router** with external IP from the uplink range (e.g., 192.168.1.230) +- **Internal logical switch** (10.10.10.0/24) +- **External logical switch** connected to the physical uplink +- **SNAT rule**: 10.10.10.0/24 → 192.168.1.230 + +--- + +## Part 3: OVN Test Results + +### Cross-node communication: PASS + +The fundamental OVN benefit — instances on different nodes can communicate: + +```bash +# test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3) +$ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3 +64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms +64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms +# 0% packet loss, ~0.1-0.8ms via Geneve tunnel +``` + +Compare with bridge networking (Part 1) where cross-node ping FAILED. + +### Full mesh connectivity: PASS + +All 12 directional paths between 4 containers on 4 different nodes: + +``` +test-c1 -> test-c2: PASS (0.672ms) test-c2 -> test-c1: PASS (0.289ms) +test-c1 -> test-c3: PASS (0.525ms) test-c3 -> test-c1: PASS (0.335ms) +test-c1 -> test-c4: PASS (0.709ms) test-c4 -> test-c1: PASS (0.412ms) +test-c2 -> test-c3: PASS (0.539ms) test-c3 -> test-c2: PASS (0.344ms) +test-c2 -> test-c4: PASS (0.730ms) test-c4 -> test-c2: PASS (0.366ms) +test-c3 -> test-c4: PASS (0.704ms) test-c4 -> test-c3: PASS (0.356ms) +``` + +### Internet access via NAT: PASS + +```bash +$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1 +64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms +# NAT through OVN router -> physical uplink -> LAN -> internet +``` + +### LAN access via physical uplink: PASS + +OVN instances can reach any host on the LAN: + +```bash +# LAN gateway +$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1 +64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms + +# Proxmox host +$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29 +64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms + +# Cluster nodes +$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13 +64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms +``` + +--- + +## Part 4: Network Isolation (Micro-Segmentation) + +### Multiple OVN networks + +Create a second OVN network with a different subnet: + +```bash +incus network create net-node-01:ovn-net2 --type=ovn \ + network=UPLINK \ + ipv4.address=10.10.20.1/24 \ + ipv4.nat=true \ + ipv6.address=none +``` + +Each OVN network gets its own external IP from the uplink range +(ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231). + +### Isolation test results + +``` +Same network, cross-node: + test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms) + +Different network, same node: + test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss) + +Different network, different node: + test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss) + +Both networks have independent internet access: + test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms) +``` + +**Each OVN network is fully isolated.** Instances on different networks cannot +communicate, even when running on the same physical node. + +--- + +## Part 5: Network ACLs + +### Basic ACL: block ICMP + +```bash +# Create ACL +incus network acl create net-node-01:block-ping +incus network acl rule add net-node-01:block-ping ingress \ + action=drop protocol=icmp4 + +# Apply to an instance's NIC +incus config device set net-node-01:test-c1 eth0 security.acls=block-ping +``` + +**Result:** All inbound ICMP to test-c1 is dropped, including echo replies. +This means ping FROM test-c1 also fails (reply packets are dropped by the +ingress rule). + +### Targeted ACL: block specific source + +```bash +# Create ACL with default allow + specific drop +incus network acl create net-node-01:allow-except-c2 +incus network acl rule add net-node-01:allow-except-c2 ingress action=allow +incus network acl rule add net-node-01:allow-except-c2 ingress \ + action=drop source=10.10.10.3/32 + +# Apply to test-c1 +incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2 +``` + +**Results:** +``` +test-c3 (allowed) -> test-c1: PASS (0.514ms) +test-c4 (allowed) -> test-c1: PASS (0.786ms) +test-c2 (blocked) -> test-c1: BLOCKED (100% loss) +``` + +### Remove ACL + +```bash +incus config device set net-node-01:test-c1 eth0 security.acls="" +incus network acl delete net-node-01:allow-except-c2 +``` + +--- + +## Part 6: Network Peering + +Connect two previously isolated OVN networks: + +```bash +# Create mutual peering (both directions required) +incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2 +incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1 +``` + +### Before peering + +```bash +# test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss) +``` + +### After peering + +```bash +# test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms) +# test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms) +``` + +Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers. + +### Remove peering + +```bash +incus network peer delete net-node-01:ovn-net1 peer-to-net2 +incus network peer delete net-node-01:ovn-net2 peer-to-net1 +``` + +--- + +## Part 7: Load Balancers + +OVN provides built-in L4 load balancing with LAN-routable virtual IPs. + +### Create load balancer + +```bash +# Create LB with a listen address from the uplink range +incus network load-balancer create net-node-01:ovn-net1 192.168.1.232 + +# Add backends +incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \ + backend-c1 10.10.10.2 80 +incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \ + backend-c3 10.10.10.4 80 + +# Map frontend port to backends +incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \ + tcp 80 backend-c1,backend-c3 +``` + +### Test results + +```bash +# 6 requests to the load balancer VIP +Request 1: Hello from test-c1 (node-01) +Request 2: Hello from test-c1 (node-01) +Request 3: Hello from test-c1 (node-01) +Request 4: Hello from test-c3 (node-03) +Request 5: Hello from test-c3 (node-03) +Request 6: Hello from test-c1 (node-01) +``` + +Traffic is distributed across both backends. OVN uses connection-based +hashing (not strict round-robin). + +### Clean up + +```bash +incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232 +``` + +--- + +## Part 8: Network Forwards (Port Forwarding) + +Expose internal services on LAN-routable IPs without load balancing. + +### Create forward + +```bash +# Allocate a LAN IP for forwarding +incus network forward create net-node-01:ovn-net1 192.168.1.233 + +# Forward external port 8080 to internal instance port 80 +incus network forward port add net-node-01:ovn-net1 192.168.1.233 \ + tcp 8080 10.10.10.2 80 +``` + +### Test results + +```bash +# From LAN (dev machine) +$ curl http://192.168.1.233:8080/ +Hello from test-c1 (node-01) + +# From another OVN network +$ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/ +Hello from test-c1 (node-01) +``` + +### Clean up + +```bash +incus network forward delete net-node-01:ovn-net1 192.168.1.233 +``` + +--- + +## Part 9: DNS Resolution + +OVN networks provide automatic DNS for instances within the same network. + +### Configuration + +Instances get `search incus` in `/etc/resolv.conf` via DHCP. DNS queries +for instance hostnames are resolved by the OVN DHCP server. + +### Test results + +```bash +# Same network: hostname resolution works +$ incus exec net-node-01:test-c2 -- python3 -c " +import socket +print(socket.gethostbyname('test-c1')) # 10.10.10.2 +print(socket.gethostbyname('test-c3')) # 10.10.10.4 +print(socket.gethostbyname('test-c4')) # 10.10.10.5 +" + +# Cross-network: resolution fails (expected) +$ incus exec net-node-01:test-c5 -- python3 -c " +import socket +socket.gethostbyname('test-c1') # Raises: Name or service not known +" + +# Ping by hostname works within the same OVN network +$ incus exec net-node-01:test-c2 -- ping -c 1 test-c1 +PING test-c1 (10.10.10.2) ... +64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms +``` + +DNS is per-network — instances on `ovn-net1` can resolve each other but +NOT instances on `ovn-net2`. + +--- + +## Part 10: Final Network State + +After all setup: + +``` ++----------+----------+---------+---------------+-------------+---------+ +| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | STATE | ++----------+----------+---------+---------------+-------------+---------+ +| UPLINK | physical | YES | | | CREATED | +| incusbr0 | bridge | YES | 10.0.0.1/24 | | CREATED | +| ovn-net1 | ovn | YES | 10.10.10.1/24 | | CREATED | +| ovn-net2 | ovn | YES | 10.10.20.1/24 | | CREATED | ++----------+----------+---------+---------------+-------------+---------+ +``` + +### Network summary + +| Network | Type | Subnet | External IP | Purpose | +|----------|----------|----------------|----------------|------------------------| +| incusbr0 | bridge | 10.0.0.1/24 | NAT via ens18 | Default, node-local | +| UPLINK | physical | - | LAN bridge | OVN external gateway | +| ovn-net1 | ovn | 10.10.10.1/24 | 192.168.1.230 | Cross-node workloads | +| ovn-net2 | ovn | 10.10.20.1/24 | 192.168.1.231 | Isolated workloads | + +--- + +## Deployment Notes + +### OC-provisioned ISO gotchas + +**Stale ISO on Proxmox storage**: When using `incusos-proxmox --iso`, the +script checks if an ISO with the same filename already exists on Proxmox +storage. If found, it skips the upload. This means a stale ISO from a +previous OC server (with a different IP baked in) will be reused, causing +all nodes to fail with "no route to host" errors when trying to reach +the old OC server for updates. + +**Fix**: Always delete the old ISO from Proxmox before deploying with a +new OC server: +```bash +# Check for stale ISOs +ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso" +# Delete if present +ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso" +``` + +The `--cleanup-all --deep` command deletes ISOs matching `IncusOS_*.iso` +(underscore pattern) but NOT `IncusOS-oc.iso` (custom hyphenated name). + +**Symptom on console** (verified via screenshots): +``` +ERROR Failed to check for Secure Boot key updates + err=http request timed out after five seconds: + Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1": + dial tcp OLD_IP:8443: connect: no route to host + provider=operations-center +``` + +### OC update prerequisites for clustering + +OC v0.3.0 requires all nodes to show `needs update: false` before +`provisioning cluster add` will work. Even when nodes are running the +latest IncusOS version, OC may report "update pending" indefinitely. + +**Workaround**: Form the cluster manually using `incus cluster enable` +and `incus cluster join` instead of `operations-center provisioning +cluster add`. + +### Cluster resource creation with apply_defaults: false + +When nodes are deployed with `apply_defaults: false` (recommended for +OC-managed nodes and cluster joining nodes), no storage pool, network, +or profile devices are created. After cluster formation, create them +manually using the two-step pattern: + +1. Define per-member config with `--target ` +2. Finalize cluster-wide with the creation command (no `--target`) + +This is required because cluster-specific settings like `source` and +`zfs.pool_name` must be defined per-member. + +### IncusOS OVN service activation + +The OVN client services (`ovsdb-server`, `ovs-vswitchd`, `ovn-controller`) +on IncusOS are **disabled by default**. They must be explicitly enabled +via the IncusOS REST API (`/os/1.0/services/ovn`) before Incus can +configure OVN networking. + +Setting `network.ovn.northbound_connection` before enabling OVN services +on all nodes will fail with: +``` +Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock +``` + +Enable OVN on each node with the correct southbound database address and +tunnel IP **before** setting the Incus northbound connection. + +### OVN control plane in a container + +When running the OVN control plane as a container on `incusbr0` (bridge +network), the container is only directly reachable from the node it runs on. +Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the +host's LAN IP so all cluster nodes can reach it.