24 KiB

Raw Blame History

Incus Networking Guide — Bridge, OVN, and LAN Integration

A hands-on tutorial covering Incus networking from basic bridge networking to OVN overlay networks, tested on a 4-node IncusOS cluster managed by Operations Center on Proxmox VE.

Prerequisites

4-node IncusOS cluster (this guide uses net-node-01 through net-node-04)
Operations Center server (for OC-managed deployment)
Proxmox VE host with incusos-proxmox deployment tool
incus client configured with remotes for all cluster nodes

Lab configuration

# incusos/examples/lab-networking.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400

vms:
  - name: net-node-01
    app: incus
    apply_defaults: false
  - name: net-node-02
    app: incus
    apply_defaults: false
  - name: net-node-03
    app: incus
    apply_defaults: false
  - name: net-node-04
    app: incus
    apply_defaults: false

Resource budget: OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total.

Cluster formation

Nodes were deployed with apply_defaults: false, so no default storage pool, network, or profile devices exist after installation. After forming the cluster manually (OC's cluster add requires "needs update: false" which was blocked by a v0.3.0 tracking issue), we created these resources:

# Storage pool (per-member source, then cluster-wide finalize)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
    incus storage create net-node-01:local zfs \
        source=local/incus zfs.pool_name=local/incus --target "$node"
done
incus storage create net-node-01:local zfs

# Bridge network (per-member pending, then finalize with config)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
    incus network create net-node-01:incusbr0 --type bridge --target "$node"
done
incus network create net-node-01:incusbr0 --type bridge \
    ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none

# Default profile
incus profile device add net-node-01:default root disk path=/ pool=local
incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0

Part 1: Bridge Networking (Default)

What is bridge networking?

When Incus creates a bridge network (type bridge), it creates a Linux bridge on each cluster node. Each bridge is independent — there is no connectivity between bridges on different nodes. This is the simplest networking mode and works without any additional infrastructure.

Bridge topology

flowchart TD
    subgraph node1["Node 1 · net-node-01"]
        c1["c1 · .202"] & c2["c2 · .40"] --- br1("incusbr0<br/>10.0.0.1/24")
        br1 --> nat1["NAT · 192.168.1.209"]
    end

    subgraph node2["Node 2 · net-node-02"]
        c3["c3 · .52"] --- br2("incusbr0<br/>10.0.0.1/24")
        br2 --> nat2["NAT · 192.168.1.150"]
    end

    nat1 & nat2 --- lan(("LAN<br/>192.168.1.0/24"))

    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef node fill:#009E73,color:#fff,stroke:#007a5e

    class c1,c2,c3 instance
    class br1,br2,lan network
    class nat1,nat2 node

    style node1 fill:#e6f5f0,stroke:#009E73
    style node2 fill:#e6f5f0,stroke:#009E73

Key point: Each node has its own bridge with the same subnet (10.0.0.1/24). The bridges are NOT connected to each other.

Test results

Same-node communication: PASS

Containers on the same node share the same bridge (same L2 domain):

# test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01
$ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40
64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms
# 0% packet loss, ~0.03ms latency

NAT to internet: PASS

The bridge has ipv4.nat=true, so containers can reach the internet:

$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms
# 0% packet loss, ~10ms to Cloudflare DNS

Cross-node communication: FAIL

Containers on different nodes CANNOT reach each other:

# test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52)
$ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52
From 10.0.0.202 icmp_seq=1 Destination Host Unreachable
# 100% packet loss

When to use bridge networking

Single-node setups (development, testing)
Workloads that don't need cross-node communication
Simple NAT-to-internet access
As an uplink for OVN networks (see Part 2)

Bridge network configuration options

# View current config
incus network show net-node-01:incusbr0

# Key settings
ipv4.address      # Bridge IP and subnet (e.g., 10.0.0.1/24)
ipv4.nat          # Enable NAT for internet access (true/false)
ipv4.dhcp         # Enable DHCP server (true/false, default true)
ipv4.dhcp.ranges  # DHCP range (e.g., 10.0.0.100-10.0.0.200)
dns.domain        # DNS domain for instances (e.g., incus)
dns.mode          # DNS mode: managed, dynamic, or none
ipv6.address      # IPv6 address (set to "none" to disable)

Part 2: OVN Overlay Networking

Why OVN?

Bridge networking is node-local. To enable cross-node communication without routing hacks, you need an overlay network. OVN (Open Virtual Network) provides:

Cross-node L2 connectivity via Geneve tunnels
Network isolation (multiple independent virtual networks)
Distributed routing (no single point of failure)
Network ACLs (firewall rules at the network level)
Load balancers (built-in L4 load balancing)
Network forwards (port forwarding from external IPs)
DNS (automatic hostname resolution between instances)

OVN architecture on IncusOS

IncusOS includes OVN client components only:

ovn-controller (data plane)
ovs-vswitchd (Open vSwitch)

The OVN control plane must run separately:

ovn-northd (translates logical to physical flows)
ovsdb-server (northbound + southbound databases)

Options for deploying the control plane:

As a container on the Incus cluster itself (used in this guide)
On an external host

OVN topology

flowchart TD
    subgraph cp["OVN Control Plane"]
        ovnc["ovn-central<br/>NB :6641 · SB :6642"]
    end

    subgraph n1["Node 1 · net-node-01"]
        ctrl1["ovn-controller"] ~~~ ls1
        c1["c1 · .2"] & c2["c2 · .3"] --- ls1("logical switch<br/>10.10.10.0/24")
    end

    subgraph n2["Node 2 · net-node-02"]
        ctrl2["ovn-controller"] ~~~ ls2
        c3["c3 · .4"] --- ls2("logical switch<br/>10.10.10.0/24")
    end

    subgraph n3["Node 3 · net-node-03"]
        ctrl3["ovn-controller"] ~~~ ls3
        c4["c4 · .5"] --- ls3("logical switch<br/>10.10.10.0/24")
    end

    ovnc -.->|"proxy devices"| ctrl1 & ctrl2 & ctrl3

    ls1 <-->|Geneve| ls2
    ls2 <-->|Geneve| ls3
    ls1 <-->|Geneve| ls3

    n1 & n2 & n3 --- lan(("LAN<br/>192.168.1.0/24"))

    classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef node fill:#009E73,color:#fff,stroke:#007a5e

    class ovnc mgmt
    class ctrl1,ctrl2,ctrl3 node
    class c1,c2,c3,c4 instance
    class ls1,ls2,ls3,lan network

    style cp fill:#f5e6f0,stroke:#CC79A7
    style n1 fill:#e6f5f0,stroke:#009E73
    style n2 fill:#e6f5f0,stroke:#009E73
    style n3 fill:#e6f5f0,stroke:#009E73

Key difference from bridge: All instances share a single logical switch connected via Geneve tunnels. Cross-node traffic is encapsulated and sent over the LAN between nodes.

Step 1: Deploy OVN control plane

Launch a container to run the OVN central services:

# Launch OVN central container
incus launch images:debian/12 net-node-01:ovn-central --target net-node-01

# Install OVN central
incus exec net-node-01:ovn-central -- apt-get update -qq
incus exec net-node-01:ovn-central -- apt-get install -y ovn-central

# Configure listening on all interfaces
incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641
incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642

# Verify
incus exec net-node-01:ovn-central -- ovn-nbctl get-connection  # ptcp:6641
incus exec net-node-01:ovn-central -- ovn-sbctl get-connection  # ptcp:6642

The container runs on incusbr0 (node-local bridge at 10.0.0.x). Other nodes can't reach this IP directly. Use proxy devices to expose the ports on the host IP:

# Forward NB and SB ports from host to container
incus config device add net-node-01:ovn-central ovn-nb proxy \
    listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641
incus config device add net-node-01:ovn-central ovn-sb proxy \
    listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642

Now all nodes can reach the OVN databases via tcp:192.168.1.209:6641 and tcp:192.168.1.209:6642.

Step 2: Enable OVN on IncusOS nodes

Critical: The OVN client services on IncusOS are disabled by default. You MUST enable them on every node before configuring Incus.

Without this step, incus config set network.ovn.northbound_connection fails:

Error: failed to notify peer: Failed to connect to OVS:
  failed to connect to unix:/run/openvswitch/db.sock: no such file or directory

Enable OVN via the IncusOS REST API on each node:

# Enable OVN on each node (set database to SOUTHBOUND port 6642)
incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.1.209:6642",
    "enabled": true,
    "tunnel_address": "192.168.1.209",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.1.209:6642",
    "enabled": true,
    "tunnel_address": "192.168.1.150",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

# Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169)
# with their respective tunnel_address values

Key fields:

database: OVN southbound DB connection (port 6642, NOT 6641)
tunnel_address: this node's LAN IP for Geneve tunnel encapsulation
tunnel_protocol: geneve (default and recommended)
enabled: true to start ovsdb-server, ovs-vswitchd, and ovn-controller

Wait ~10 seconds for services to start, then verify:

incus query net-node-01:/os/1.0/services/ovn
# Should show enabled: true

Step 3: Configure Incus OVN connection

Tell Incus where the OVN northbound database is:

incus config set net-node-01: network.ovn.northbound_connection \
    tcp:192.168.1.209:6641

This is a cluster-wide setting that propagates to all members. It WILL FAIL if any member doesn't have OVS running (Step 2 not complete).

Verify all chassis are registered:

$ incus exec net-node-01:ovn-central -- ovn-sbctl show
Chassis "c1a92a50-..."
    hostname: net-node-01
    Encap geneve
        ip: "192.168.1.209"
Chassis "f4fb8f29-..."
    hostname: net-node-02
    Encap geneve
        ip: "192.168.1.150"
Chassis "88f8cd95-..."
    hostname: net-node-03
    Encap geneve
        ip: "192.168.1.13"
Chassis "6be485be-..."
    hostname: net-node-04
    Encap geneve
        ip: "192.168.1.169"

Step 4: Assign ovn-chassis roles

Designate which nodes participate in OVN gateway HA:

for node in net-node-01 net-node-02 net-node-03 net-node-04; do
    incus cluster role add net-node-01:"$node" ovn-chassis
done

Step 5: Create physical uplink network

OVN networks need an uplink for external/LAN connectivity:

# Per-member definition (two-step cluster pattern)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
    incus network create net-node-01:UPLINK --type=physical \
        parent=ens18 --target "$node"
done

# Finalize with external IP range and gateway
incus network create net-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \
    ipv4.gateway=192.168.1.1/24 \
    dns.nameservers=192.168.1.1

parent=ens18: the node's physical NIC. Note: newer IncusOS versions name this interface mgmt. If ens18 fails with "Parent interface not found", use parent=mgmt instead. Check with: incus query REMOTE:/os/1.0/system/network
ipv4.ovn.ranges: pool of LAN IPs for OVN router external addresses
ipv4.gateway: LAN gateway in CIDR format

Step 6: Create OVN network

incus network create net-node-01:ovn-net1 --type=ovn \
    network=UPLINK \
    ipv4.address=10.10.10.1/24 \
    ipv4.nat=true \
    ipv6.address=none

Incus automatically creates:

Logical router with external IP from the uplink range (e.g., 192.168.1.230)
Internal logical switch (10.10.10.0/24)
External logical switch connected to the physical uplink
SNAT rule: 10.10.10.0/24 → 192.168.1.230

Part 3: OVN Test Results

Cross-node communication: PASS

The fundamental OVN benefit — instances on different nodes can communicate:

# test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3)
$ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms
# 0% packet loss, ~0.1-0.8ms via Geneve tunnel

Compare with bridge networking (Part 1) where cross-node ping FAILED.

Full mesh connectivity: PASS

All 12 directional paths between 4 containers on 4 different nodes:

test-c1 -> test-c2: PASS (0.672ms)    test-c2 -> test-c1: PASS (0.289ms)
test-c1 -> test-c3: PASS (0.525ms)    test-c3 -> test-c1: PASS (0.335ms)
test-c1 -> test-c4: PASS (0.709ms)    test-c4 -> test-c1: PASS (0.412ms)
test-c2 -> test-c3: PASS (0.539ms)    test-c3 -> test-c2: PASS (0.344ms)
test-c2 -> test-c4: PASS (0.730ms)    test-c4 -> test-c2: PASS (0.366ms)
test-c3 -> test-c4: PASS (0.704ms)    test-c4 -> test-c3: PASS (0.356ms)

Internet access via NAT: PASS

$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms
# NAT through OVN router -> physical uplink -> LAN -> internet

LAN access via physical uplink: PASS

OVN instances can reach any host on the LAN:

# LAN gateway
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms

# Proxmox host
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29
64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms

# Cluster nodes
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13
64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms

Part 4: Network Isolation (Micro-Segmentation)

Multiple OVN networks

Create a second OVN network with a different subnet:

incus network create net-node-01:ovn-net2 --type=ovn \
    network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none

Each OVN network gets its own external IP from the uplink range (ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231).

Isolation test results

Same network, cross-node:
  test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms)

Different network, same node:
  test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss)

Different network, different node:
  test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss)

Both networks have independent internet access:
  test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms)

Each OVN network is fully isolated. Instances on different networks cannot communicate, even when running on the same physical node.

Part 5: Network ACLs

Basic ACL: block ICMP

# Create ACL
incus network acl create net-node-01:block-ping
incus network acl rule add net-node-01:block-ping ingress \
    action=drop protocol=icmp4

# Apply to an instance's NIC
incus config device set net-node-01:test-c1 eth0 security.acls=block-ping

Result: All inbound ICMP to test-c1 is dropped, including echo replies. This means ping FROM test-c1 also fails (reply packets are dropped by the ingress rule).

Targeted ACL: block specific source

# Create ACL with default allow + specific drop
incus network acl create net-node-01:allow-except-c2
incus network acl rule add net-node-01:allow-except-c2 ingress action=allow
incus network acl rule add net-node-01:allow-except-c2 ingress \
    action=drop source=10.10.10.3/32

# Apply to test-c1
incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2

Results:

test-c3 (allowed)  -> test-c1: PASS (0.514ms)
test-c4 (allowed)  -> test-c1: PASS (0.786ms)
test-c2 (blocked)  -> test-c1: BLOCKED (100% loss)

Remove ACL

incus config device set net-node-01:test-c1 eth0 security.acls=""
incus network acl delete net-node-01:allow-except-c2

Part 6: Network Peering

Connect two previously isolated OVN networks:

# Create mutual peering (both directions required)
incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2
incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1

Before peering

# test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss)

After peering

# test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms)
# test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms)

Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers.

Remove peering

incus network peer delete net-node-01:ovn-net1 peer-to-net2
incus network peer delete net-node-01:ovn-net2 peer-to-net1

Part 7: Load Balancers

OVN provides built-in L4 load balancing with LAN-routable virtual IPs.

Create load balancer

# Create LB with a listen address from the uplink range
incus network load-balancer create net-node-01:ovn-net1 192.168.1.232

# Add backends
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
    backend-c1 10.10.10.2 80
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
    backend-c3 10.10.10.4 80

# Map frontend port to backends
incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \
    tcp 80 backend-c1,backend-c3

Test results

# 6 requests to the load balancer VIP
Request 1: Hello from test-c1 (node-01)
Request 2: Hello from test-c1 (node-01)
Request 3: Hello from test-c1 (node-01)
Request 4: Hello from test-c3 (node-03)
Request 5: Hello from test-c3 (node-03)
Request 6: Hello from test-c1 (node-01)

Traffic is distributed across both backends. OVN uses connection-based hashing (not strict round-robin).

Clean up

incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232

Part 8: Network Forwards (Port Forwarding)

Expose internal services on LAN-routable IPs without load balancing.

Create forward

# Allocate a LAN IP for forwarding
incus network forward create net-node-01:ovn-net1 192.168.1.233

# Forward external port 8080 to internal instance port 80
incus network forward port add net-node-01:ovn-net1 192.168.1.233 \
    tcp 8080 10.10.10.2 80

Test results

# From LAN (dev machine)
$ curl http://192.168.1.233:8080/
Hello from test-c1 (node-01)

# From another OVN network
$ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/
Hello from test-c1 (node-01)

Clean up

incus network forward delete net-node-01:ovn-net1 192.168.1.233

Part 9: DNS Resolution

OVN networks provide automatic DNS for instances within the same network.

Configuration

Instances get search incus in /etc/resolv.conf via DHCP. DNS queries for instance hostnames are resolved by the OVN DHCP server.

Test results

# Same network: hostname resolution works
$ incus exec net-node-01:test-c2 -- python3 -c "
import socket
print(socket.gethostbyname('test-c1'))   # 10.10.10.2
print(socket.gethostbyname('test-c3'))   # 10.10.10.4
print(socket.gethostbyname('test-c4'))   # 10.10.10.5
"

# Cross-network: resolution fails (expected)
$ incus exec net-node-01:test-c5 -- python3 -c "
import socket
socket.gethostbyname('test-c1')   # Raises: Name or service not known
"

# Ping by hostname works within the same OVN network
$ incus exec net-node-01:test-c2 -- ping -c 1 test-c1
PING test-c1 (10.10.10.2) ...
64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms

DNS is per-network — instances on ovn-net1 can resolve each other but NOT instances on ovn-net2.

Part 10: Final Network State

After all setup:

+----------+----------+---------+---------------+-------------+---------+
|   NAME   |   TYPE   | MANAGED |     IPV4      | DESCRIPTION |  STATE  |
+----------+----------+---------+---------------+-------------+---------+
| UPLINK   | physical | YES     |               |             | CREATED |
| incusbr0 | bridge   | YES     | 10.0.0.1/24   |             | CREATED |
| ovn-net1 | ovn      | YES     | 10.10.10.1/24 |             | CREATED |
| ovn-net2 | ovn      | YES     | 10.10.20.1/24 |             | CREATED |
+----------+----------+---------+---------------+-------------+---------+

Network summary

Network	Type	Subnet	External IP	Purpose
incusbr0	bridge	10.0.0.1/24	NAT via ens18	Default, node-local
UPLINK	physical	-	LAN bridge	OVN external gateway
ovn-net1	ovn	10.10.10.1/24	192.168.1.230	Cross-node workloads
ovn-net2	ovn	10.10.20.1/24	192.168.1.231	Isolated workloads

Deployment Notes

OC-provisioned ISO gotchas

Stale ISO on Proxmox storage: When using incusos-proxmox --iso, the script checks if an ISO with the same filename already exists on Proxmox storage. If found, it skips the upload. This means a stale ISO from a previous OC server (with a different IP baked in) will be reused, causing all nodes to fail with "no route to host" errors when trying to reach the old OC server for updates.

Fix: Always delete the old ISO from Proxmox before deploying with a new OC server:

# Check for stale ISOs
ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso"
# Delete if present
ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso"

The --cleanup-all --deep command deletes ISOs matching IncusOS_*.iso (underscore pattern) but NOT IncusOS-oc.iso (custom hyphenated name).

Symptom on console (verified via screenshots):

ERROR Failed to check for Secure Boot key updates
  err=http request timed out after five seconds:
  Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1":
  dial tcp OLD_IP:8443: connect: no route to host
  provider=operations-center

OC update prerequisites for clustering

OC v0.3.0 requires all nodes to show needs update: false before provisioning cluster add will work. Even when nodes are running the latest IncusOS version, OC may report "update pending" indefinitely.

Workaround: Form the cluster manually using incus cluster enable and incus cluster join instead of operations-center provisioning cluster add.

Cluster resource creation with apply_defaults: false

When nodes are deployed with apply_defaults: false (recommended for OC-managed nodes and cluster joining nodes), no storage pool, network, or profile devices are created. After cluster formation, create them manually using the two-step pattern:

Define per-member config with --target <member>
Finalize cluster-wide with the creation command (no --target)

This is required because cluster-specific settings like source and zfs.pool_name must be defined per-member.

IncusOS OVN service activation

The OVN client services (ovsdb-server, ovs-vswitchd, ovn-controller) on IncusOS are disabled by default. They must be explicitly enabled via the IncusOS REST API (/os/1.0/services/ovn) before Incus can configure OVN networking.

Setting network.ovn.northbound_connection before enabling OVN services on all nodes will fail with:

Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock

Enable OVN on each node with the correct southbound database address and tunnel IP before setting the Incus northbound connection.

OVN control plane in a container

When running the OVN control plane as a container on incusbr0 (bridge network), the container is only directly reachable from the node it runs on. Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the host's LAN IP so all cluster nodes can reach it.

24 KiB Raw Blame History Unescape Escape

Incus Networking Guide — Bridge, OVN, and LAN Integration

Prerequisites

Lab configuration

Cluster formation

Part 1: Bridge Networking (Default)

What is bridge networking?

Bridge topology

Test results

Same-node communication: PASS

NAT to internet: PASS

Cross-node communication: FAIL

When to use bridge networking

Bridge network configuration options

Part 2: OVN Overlay Networking

Why OVN?

OVN architecture on IncusOS

OVN topology

Step 1: Deploy OVN control plane

Step 2: Enable OVN on IncusOS nodes

Step 3: Configure Incus OVN connection

Step 4: Assign ovn-chassis roles

Step 5: Create physical uplink network

Step 6: Create OVN network

Part 3: OVN Test Results

Cross-node communication: PASS

Full mesh connectivity: PASS

Internet access via NAT: PASS

LAN access via physical uplink: PASS

Part 4: Network Isolation (Micro-Segmentation)

Multiple OVN networks

Isolation test results

Part 5: Network ACLs

Basic ACL: block ICMP

Targeted ACL: block specific source

Remove ACL

Part 6: Network Peering

Before peering

After peering

Remove peering

Part 7: Load Balancers

Create load balancer

Test results

Clean up

Part 8: Network Forwards (Port Forwarding)

Create forward

Test results

Clean up

Part 9: DNS Resolution

Configuration

Test results

Part 10: Final Network State

Network summary

Deployment Notes

OC-provisioned ISO gotchas

OC update prerequisites for clustering

Cluster resource creation with apply_defaults: false

IncusOS OVN service activation

OVN control plane in a container

24 KiB

Raw Blame History