# Incus Networking Guide — Bridge, OVN, and LAN Integration A hands-on tutorial covering Incus networking from basic bridge networking to OVN overlay networks, tested on a 4-node IncusOS cluster managed by Operations Center on Proxmox VE. ## Prerequisites - 4-node IncusOS cluster (this guide uses `net-node-01` through `net-node-04`) - Operations Center server (for OC-managed deployment) - Proxmox VE host with `incusos-proxmox` deployment tool - `incus` client configured with remotes for all cluster nodes ### Lab configuration ```yaml # incusos/examples/lab-networking.yaml defaults: cores: 4 memory: 8192 disk: 50 start_vmid: 400 vms: - name: net-node-01 app: incus apply_defaults: false - name: net-node-02 app: incus apply_defaults: false - name: net-node-03 app: incus apply_defaults: false - name: net-node-04 app: incus apply_defaults: false ``` **Resource budget:** OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total. ### Cluster formation Nodes were deployed with `apply_defaults: false`, so no default storage pool, network, or profile devices exist after installation. After forming the cluster manually (OC's `cluster add` requires "needs update: false" which was blocked by a v0.3.0 tracking issue), we created these resources: ```bash # Storage pool (per-member source, then cluster-wide finalize) for node in net-node-01 net-node-02 net-node-03 net-node-04; do incus storage create net-node-01:local zfs \ source=local/incus zfs.pool_name=local/incus --target "$node" done incus storage create net-node-01:local zfs # Bridge network (per-member pending, then finalize with config) for node in net-node-01 net-node-02 net-node-03 net-node-04; do incus network create net-node-01:incusbr0 --type bridge --target "$node" done incus network create net-node-01:incusbr0 --type bridge \ ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none # Default profile incus profile device add net-node-01:default root disk path=/ pool=local incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0 ``` --- ## Part 1: Bridge Networking (Default) ### What is bridge networking? When Incus creates a bridge network (type `bridge`), it creates a Linux bridge on each cluster node. Each bridge is **independent** — there is no connectivity between bridges on different nodes. This is the simplest networking mode and works without any additional infrastructure. ### Bridge topology ```mermaid flowchart TD subgraph node1["Node 1 · net-node-01"] c1["c1 · .202"] & c2["c2 · .40"] --- br1("incusbr0
10.0.0.1/24") br1 --> nat1["NAT · 192.168.1.209"] end subgraph node2["Node 2 · net-node-02"] c3["c3 · .52"] --- br2("incusbr0
10.0.0.1/24") br2 --> nat2["NAT · 192.168.1.150"] end nat1 & nat2 --- lan(("LAN
192.168.1.0/24")) classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf classDef network fill:#0072B2,color:#fff,stroke:#005a8e classDef node fill:#009E73,color:#fff,stroke:#007a5e class c1,c2,c3 instance class br1,br2,lan network class nat1,nat2 node style node1 fill:#e6f5f0,stroke:#009E73 style node2 fill:#e6f5f0,stroke:#009E73 ``` **Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24). The bridges are NOT connected to each other. ### Test results #### Same-node communication: PASS Containers on the same node share the same bridge (same L2 domain): ```bash # test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01 $ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40 64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms # 0% packet loss, ~0.03ms latency ``` #### NAT to internet: PASS The bridge has `ipv4.nat=true`, so containers can reach the internet: ```bash $ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1 64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms # 0% packet loss, ~10ms to Cloudflare DNS ``` #### Cross-node communication: FAIL Containers on different nodes CANNOT reach each other: ```bash # test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52) $ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52 From 10.0.0.202 icmp_seq=1 Destination Host Unreachable # 100% packet loss ``` ### When to use bridge networking - Single-node setups (development, testing) - Workloads that don't need cross-node communication - Simple NAT-to-internet access - As an uplink for OVN networks (see Part 2) ### Bridge network configuration options ```bash # View current config incus network show net-node-01:incusbr0 # Key settings ipv4.address # Bridge IP and subnet (e.g., 10.0.0.1/24) ipv4.nat # Enable NAT for internet access (true/false) ipv4.dhcp # Enable DHCP server (true/false, default true) ipv4.dhcp.ranges # DHCP range (e.g., 10.0.0.100-10.0.0.200) dns.domain # DNS domain for instances (e.g., incus) dns.mode # DNS mode: managed, dynamic, or none ipv6.address # IPv6 address (set to "none" to disable) ``` --- ## Part 2: OVN Overlay Networking ### Why OVN? Bridge networking is node-local. To enable cross-node communication without routing hacks, you need an overlay network. OVN (Open Virtual Network) provides: - **Cross-node L2 connectivity** via Geneve tunnels - **Network isolation** (multiple independent virtual networks) - **Distributed routing** (no single point of failure) - **Network ACLs** (firewall rules at the network level) - **Load balancers** (built-in L4 load balancing) - **Network forwards** (port forwarding from external IPs) - **DNS** (automatic hostname resolution between instances) ### OVN architecture on IncusOS IncusOS includes OVN **client** components only: - `ovn-controller` (data plane) - `ovs-vswitchd` (Open vSwitch) The OVN **control plane** must run separately: - `ovn-northd` (translates logical to physical flows) - `ovsdb-server` (northbound + southbound databases) Options for deploying the control plane: 1. As a container on the Incus cluster itself (used in this guide) 2. On an external host ### OVN topology ```mermaid flowchart TD subgraph cp["OVN Control Plane"] ovnc["ovn-central
NB :6641 · SB :6642"] end subgraph n1["Node 1 · net-node-01"] ctrl1["ovn-controller"] ~~~ ls1 c1["c1 · .2"] & c2["c2 · .3"] --- ls1("logical switch
10.10.10.0/24") end subgraph n2["Node 2 · net-node-02"] ctrl2["ovn-controller"] ~~~ ls2 c3["c3 · .4"] --- ls2("logical switch
10.10.10.0/24") end subgraph n3["Node 3 · net-node-03"] ctrl3["ovn-controller"] ~~~ ls3 c4["c4 · .5"] --- ls3("logical switch
10.10.10.0/24") end ovnc -.->|"proxy devices"| ctrl1 & ctrl2 & ctrl3 ls1 <-->|Geneve| ls2 ls2 <-->|Geneve| ls3 ls1 <-->|Geneve| ls3 n1 & n2 & n3 --- lan(("LAN
192.168.1.0/24")) classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088 classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf classDef network fill:#0072B2,color:#fff,stroke:#005a8e classDef node fill:#009E73,color:#fff,stroke:#007a5e class ovnc mgmt class ctrl1,ctrl2,ctrl3 node class c1,c2,c3,c4 instance class ls1,ls2,ls3,lan network style cp fill:#f5e6f0,stroke:#CC79A7 style n1 fill:#e6f5f0,stroke:#009E73 style n2 fill:#e6f5f0,stroke:#009E73 style n3 fill:#e6f5f0,stroke:#009E73 ``` **Key difference from bridge:** All instances share a single logical switch connected via Geneve tunnels. Cross-node traffic is encapsulated and sent over the LAN between nodes. ### Step 1: Deploy OVN control plane Launch a container to run the OVN central services: ```bash # Launch OVN central container incus launch images:debian/12 net-node-01:ovn-central --target net-node-01 # Install OVN central incus exec net-node-01:ovn-central -- apt-get update -qq incus exec net-node-01:ovn-central -- apt-get install -y ovn-central # Configure listening on all interfaces incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641 incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642 # Verify incus exec net-node-01:ovn-central -- ovn-nbctl get-connection # ptcp:6641 incus exec net-node-01:ovn-central -- ovn-sbctl get-connection # ptcp:6642 ``` The container runs on `incusbr0` (node-local bridge at 10.0.0.x). Other nodes can't reach this IP directly. Use proxy devices to expose the ports on the host IP: ```bash # Forward NB and SB ports from host to container incus config device add net-node-01:ovn-central ovn-nb proxy \ listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641 incus config device add net-node-01:ovn-central ovn-sb proxy \ listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642 ``` Now all nodes can reach the OVN databases via `tcp:192.168.1.209:6641` and `tcp:192.168.1.209:6642`. ### Step 2: Enable OVN on IncusOS nodes **Critical:** The OVN client services on IncusOS are disabled by default. You MUST enable them on every node before configuring Incus. Without this step, `incus config set network.ovn.northbound_connection` fails: ``` Error: failed to notify peer: Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock: no such file or directory ``` Enable OVN via the IncusOS REST API on each node: ```bash # Enable OVN on each node (set database to SOUTHBOUND port 6642) incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{ "config": { "database": "tcp:192.168.1.209:6642", "enabled": true, "tunnel_address": "192.168.1.209", "tunnel_protocol": "geneve" }, "state": {} }' incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{ "config": { "database": "tcp:192.168.1.209:6642", "enabled": true, "tunnel_address": "192.168.1.150", "tunnel_protocol": "geneve" }, "state": {} }' # Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169) # with their respective tunnel_address values ``` Key fields: - `database`: OVN **southbound** DB connection (port 6642, NOT 6641) - `tunnel_address`: this node's LAN IP for Geneve tunnel encapsulation - `tunnel_protocol`: `geneve` (default and recommended) - `enabled`: `true` to start `ovsdb-server`, `ovs-vswitchd`, and `ovn-controller` Wait ~10 seconds for services to start, then verify: ```bash incus query net-node-01:/os/1.0/services/ovn # Should show enabled: true ``` ### Step 3: Configure Incus OVN connection Tell Incus where the OVN **northbound** database is: ```bash incus config set net-node-01: network.ovn.northbound_connection \ tcp:192.168.1.209:6641 ``` This is a cluster-wide setting that propagates to all members. It WILL FAIL if any member doesn't have OVS running (Step 2 not complete). Verify all chassis are registered: ```bash $ incus exec net-node-01:ovn-central -- ovn-sbctl show Chassis "c1a92a50-..." hostname: net-node-01 Encap geneve ip: "192.168.1.209" Chassis "f4fb8f29-..." hostname: net-node-02 Encap geneve ip: "192.168.1.150" Chassis "88f8cd95-..." hostname: net-node-03 Encap geneve ip: "192.168.1.13" Chassis "6be485be-..." hostname: net-node-04 Encap geneve ip: "192.168.1.169" ``` ### Step 4: Assign ovn-chassis roles Designate which nodes participate in OVN gateway HA: ```bash for node in net-node-01 net-node-02 net-node-03 net-node-04; do incus cluster role add net-node-01:"$node" ovn-chassis done ``` ### Step 5: Create physical uplink network OVN networks need an uplink for external/LAN connectivity: ```bash # Per-member definition (two-step cluster pattern) for node in net-node-01 net-node-02 net-node-03 net-node-04; do incus network create net-node-01:UPLINK --type=physical \ parent=ens18 --target "$node" done # Finalize with external IP range and gateway incus network create net-node-01:UPLINK --type=physical \ ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \ ipv4.gateway=192.168.1.1/24 \ dns.nameservers=192.168.1.1 ``` - `parent=ens18`: the node's physical NIC. **Note**: newer IncusOS versions name this interface `mgmt`. If `ens18` fails with "Parent interface not found", use `parent=mgmt` instead. Check with: `incus query REMOTE:/os/1.0/system/network` - `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses - `ipv4.gateway`: LAN gateway in CIDR format ### Step 6: Create OVN network ```bash incus network create net-node-01:ovn-net1 --type=ovn \ network=UPLINK \ ipv4.address=10.10.10.1/24 \ ipv4.nat=true \ ipv6.address=none ``` Incus automatically creates: - **Logical router** with external IP from the uplink range (e.g., 192.168.1.230) - **Internal logical switch** (10.10.10.0/24) - **External logical switch** connected to the physical uplink - **SNAT rule**: 10.10.10.0/24 → 192.168.1.230 --- ## Part 3: OVN Test Results ### Cross-node communication: PASS The fundamental OVN benefit — instances on different nodes can communicate: ```bash # test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3) $ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3 64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms 64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms # 0% packet loss, ~0.1-0.8ms via Geneve tunnel ``` Compare with bridge networking (Part 1) where cross-node ping FAILED. ### Full mesh connectivity: PASS All 12 directional paths between 4 containers on 4 different nodes: ``` test-c1 -> test-c2: PASS (0.672ms) test-c2 -> test-c1: PASS (0.289ms) test-c1 -> test-c3: PASS (0.525ms) test-c3 -> test-c1: PASS (0.335ms) test-c1 -> test-c4: PASS (0.709ms) test-c4 -> test-c1: PASS (0.412ms) test-c2 -> test-c3: PASS (0.539ms) test-c3 -> test-c2: PASS (0.344ms) test-c2 -> test-c4: PASS (0.730ms) test-c4 -> test-c2: PASS (0.366ms) test-c3 -> test-c4: PASS (0.704ms) test-c4 -> test-c3: PASS (0.356ms) ``` ### Internet access via NAT: PASS ```bash $ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1 64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms # NAT through OVN router -> physical uplink -> LAN -> internet ``` ### LAN access via physical uplink: PASS OVN instances can reach any host on the LAN: ```bash # LAN gateway $ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1 64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms # Proxmox host $ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29 64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms # Cluster nodes $ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13 64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms ``` --- ## Part 4: Network Isolation (Micro-Segmentation) ### Multiple OVN networks Create a second OVN network with a different subnet: ```bash incus network create net-node-01:ovn-net2 --type=ovn \ network=UPLINK \ ipv4.address=10.10.20.1/24 \ ipv4.nat=true \ ipv6.address=none ``` Each OVN network gets its own external IP from the uplink range (ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231). ### Isolation test results ``` Same network, cross-node: test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms) Different network, same node: test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss) Different network, different node: test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss) Both networks have independent internet access: test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms) ``` **Each OVN network is fully isolated.** Instances on different networks cannot communicate, even when running on the same physical node. --- ## Part 5: Network ACLs ### Basic ACL: block ICMP ```bash # Create ACL incus network acl create net-node-01:block-ping incus network acl rule add net-node-01:block-ping ingress \ action=drop protocol=icmp4 # Apply to an instance's NIC incus config device set net-node-01:test-c1 eth0 security.acls=block-ping ``` **Result:** All inbound ICMP to test-c1 is dropped, including echo replies. This means ping FROM test-c1 also fails (reply packets are dropped by the ingress rule). ### Targeted ACL: block specific source ```bash # Create ACL with default allow + specific drop incus network acl create net-node-01:allow-except-c2 incus network acl rule add net-node-01:allow-except-c2 ingress action=allow incus network acl rule add net-node-01:allow-except-c2 ingress \ action=drop source=10.10.10.3/32 # Apply to test-c1 incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2 ``` **Results:** ``` test-c3 (allowed) -> test-c1: PASS (0.514ms) test-c4 (allowed) -> test-c1: PASS (0.786ms) test-c2 (blocked) -> test-c1: BLOCKED (100% loss) ``` ### Remove ACL ```bash incus config device set net-node-01:test-c1 eth0 security.acls="" incus network acl delete net-node-01:allow-except-c2 ``` --- ## Part 6: Network Peering Connect two previously isolated OVN networks: ```bash # Create mutual peering (both directions required) incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2 incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1 ``` ### Before peering ```bash # test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss) ``` ### After peering ```bash # test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms) # test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms) ``` Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers. ### Remove peering ```bash incus network peer delete net-node-01:ovn-net1 peer-to-net2 incus network peer delete net-node-01:ovn-net2 peer-to-net1 ``` --- ## Part 7: Load Balancers OVN provides built-in L4 load balancing with LAN-routable virtual IPs. ### Create load balancer ```bash # Create LB with a listen address from the uplink range incus network load-balancer create net-node-01:ovn-net1 192.168.1.232 # Add backends incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \ backend-c1 10.10.10.2 80 incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \ backend-c3 10.10.10.4 80 # Map frontend port to backends incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \ tcp 80 backend-c1,backend-c3 ``` ### Test results ```bash # 6 requests to the load balancer VIP Request 1: Hello from test-c1 (node-01) Request 2: Hello from test-c1 (node-01) Request 3: Hello from test-c1 (node-01) Request 4: Hello from test-c3 (node-03) Request 5: Hello from test-c3 (node-03) Request 6: Hello from test-c1 (node-01) ``` Traffic is distributed across both backends. OVN uses connection-based hashing (not strict round-robin). ### Clean up ```bash incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232 ``` --- ## Part 8: Network Forwards (Port Forwarding) Expose internal services on LAN-routable IPs without load balancing. ### Create forward ```bash # Allocate a LAN IP for forwarding incus network forward create net-node-01:ovn-net1 192.168.1.233 # Forward external port 8080 to internal instance port 80 incus network forward port add net-node-01:ovn-net1 192.168.1.233 \ tcp 8080 10.10.10.2 80 ``` ### Test results ```bash # From LAN (dev machine) $ curl http://192.168.1.233:8080/ Hello from test-c1 (node-01) # From another OVN network $ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/ Hello from test-c1 (node-01) ``` ### Clean up ```bash incus network forward delete net-node-01:ovn-net1 192.168.1.233 ``` --- ## Part 9: DNS Resolution OVN networks provide automatic DNS for instances within the same network. ### Configuration Instances get `search incus` in `/etc/resolv.conf` via DHCP. DNS queries for instance hostnames are resolved by the OVN DHCP server. ### Test results ```bash # Same network: hostname resolution works $ incus exec net-node-01:test-c2 -- python3 -c " import socket print(socket.gethostbyname('test-c1')) # 10.10.10.2 print(socket.gethostbyname('test-c3')) # 10.10.10.4 print(socket.gethostbyname('test-c4')) # 10.10.10.5 " # Cross-network: resolution fails (expected) $ incus exec net-node-01:test-c5 -- python3 -c " import socket socket.gethostbyname('test-c1') # Raises: Name or service not known " # Ping by hostname works within the same OVN network $ incus exec net-node-01:test-c2 -- ping -c 1 test-c1 PING test-c1 (10.10.10.2) ... 64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms ``` DNS is per-network — instances on `ovn-net1` can resolve each other but NOT instances on `ovn-net2`. --- ## Part 10: Final Network State After all setup: ``` +----------+----------+---------+---------------+-------------+---------+ | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | STATE | +----------+----------+---------+---------------+-------------+---------+ | UPLINK | physical | YES | | | CREATED | | incusbr0 | bridge | YES | 10.0.0.1/24 | | CREATED | | ovn-net1 | ovn | YES | 10.10.10.1/24 | | CREATED | | ovn-net2 | ovn | YES | 10.10.20.1/24 | | CREATED | +----------+----------+---------+---------------+-------------+---------+ ``` ### Network summary | Network | Type | Subnet | External IP | Purpose | |----------|----------|----------------|----------------|------------------------| | incusbr0 | bridge | 10.0.0.1/24 | NAT via ens18 | Default, node-local | | UPLINK | physical | - | LAN bridge | OVN external gateway | | ovn-net1 | ovn | 10.10.10.1/24 | 192.168.1.230 | Cross-node workloads | | ovn-net2 | ovn | 10.10.20.1/24 | 192.168.1.231 | Isolated workloads | --- ## Deployment Notes ### OC-provisioned ISO gotchas **Stale ISO on Proxmox storage**: When using `incusos-proxmox --iso`, the script checks if an ISO with the same filename already exists on Proxmox storage. If found, it skips the upload. This means a stale ISO from a previous OC server (with a different IP baked in) will be reused, causing all nodes to fail with "no route to host" errors when trying to reach the old OC server for updates. **Fix**: Always delete the old ISO from Proxmox before deploying with a new OC server: ```bash # Check for stale ISOs ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso" # Delete if present ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso" ``` The `--cleanup-all --deep` command deletes ISOs matching `IncusOS_*.iso` (underscore pattern) but NOT `IncusOS-oc.iso` (custom hyphenated name). **Symptom on console** (verified via screenshots): ``` ERROR Failed to check for Secure Boot key updates err=http request timed out after five seconds: Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1": dial tcp OLD_IP:8443: connect: no route to host provider=operations-center ``` ### OC update prerequisites for clustering OC v0.3.0 requires all nodes to show `needs update: false` before `provisioning cluster add` will work. Even when nodes are running the latest IncusOS version, OC may report "update pending" indefinitely. **Workaround**: Form the cluster manually using `incus cluster enable` and `incus cluster join` instead of `operations-center provisioning cluster add`. ### Cluster resource creation with apply_defaults: false When nodes are deployed with `apply_defaults: false` (recommended for OC-managed nodes and cluster joining nodes), no storage pool, network, or profile devices are created. After cluster formation, create them manually using the two-step pattern: 1. Define per-member config with `--target ` 2. Finalize cluster-wide with the creation command (no `--target`) This is required because cluster-specific settings like `source` and `zfs.pool_name` must be defined per-member. ### IncusOS OVN service activation The OVN client services (`ovsdb-server`, `ovs-vswitchd`, `ovn-controller`) on IncusOS are **disabled by default**. They must be explicitly enabled via the IncusOS REST API (`/os/1.0/services/ovn`) before Incus can configure OVN networking. Setting `network.ovn.northbound_connection` before enabling OVN services on all nodes will fail with: ``` Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock ``` Enable OVN on each node with the correct southbound database address and tunnel IP **before** setting the Incus northbound connection. ### OVN control plane in a container When running the OVN control plane as a container on `incusbr0` (bridge network), the container is only directly reachable from the node it runs on. Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the host's LAN IP so all cluster nodes can reach it.