maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/ovn-deep-dive.md

22 KiB
Raw Permalink Blame History

OVN Deep-Dive: Incus Cluster Networking Internals

How Incus translates its network abstractions into OVN logical objects, OVS physical flows, and Geneve tunnels. Based on a live 3-node IncusOS cluster running OVS 3.6.1 with an OVN overlay network.

Lab Topology

flowchart TD
    lan(("LAN<br/>192.168.100.0/22"))

    n1["oc-node-01 · .140<br/>Geneve endpoint"]
    n2["oc-node-02 · .141<br/>Geneve endpoint"]
    n3["oc-node-03 · .142<br/>Geneve + gateway"]

    uplink("UPLINK<br/>OVN range .103.200-210")
    netprod("net-prod<br/>10.10.10.0/24")

    lan --- n1 & n2 & n3
    lan --- uplink --> netprod

    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 node
    class lan,uplink,netprod network

Instances on net-prod:

Instance IP Node Role
ffsdn-haproxy-52-01 10.10.10.50 oc-node-01 HAProxy LB
ffsdn-haproxy-52-02 10.10.10.51 oc-node-02 HAProxy LB
nginx-lb-01 10.10.10.60 oc-node-01 Nginx backend
nginx-lb-02 10.10.10.61 oc-node-02 Nginx backend
nginx-lb-03 10.10.10.62 oc-node-03 Nginx backend
test-app-manual-web-tier-app-1 10.10.10.4 oc-node-03 App server
test-app-manual-web-tier-web-1 10.10.10.2 oc-node-01 Web server
test-app-manual-web-tier-web-2 10.10.10.3 oc-node-02 Web server

OVN Northbound Database (Logical Layer)

The NB database describes the desired network topology. Incus creates and manages all objects here — users never touch OVN directly.

Incus → OVN Naming Convention

Incus uses a consistent naming scheme. For a network named net-prod with internal ID net8:

Incus Concept OVN Object OVN Name
OVN network Logical Switch (internal) incus-net8-ls-int
OVN uplink Logical Switch (external) incus-net8-ls-ext
OVN gateway Logical Router incus-net8-lr
Instance NIC Logical Switch Port incus-net8-instance-<UUID>-eth0
Router↔int switch Router Port + Switch Port incus-net8-lr-lrp-int / incus-net8-ls-int-lsp-router
Router↔ext switch Router Port + Switch Port incus-net8-lr-lrp-ext / incus-net8-ls-ext-lsp-router
Provider connection Localnet Port incus-net8-ls-ext-lsp-provider
Network forward LB Load Balancer incus-net8-lb-<VIP>-<proto>

The numeric net8 suffix is an internal Incus identifier for the network. It increments as networks are created.

Logical Switches

Internal switch (incus-net8-ls-int): The overlay network where all instances connect. Subnet 10.10.10.0/24, IPv6 fd42:842b:1e5f:b27::/64.

incus-net8-ls-int
├── 8 instance ports (one per container on net-prod)
├── 1 router port (incus-net8-ls-int-lsp-router)
├── 1 load balancer attached (VIP 192.168.103.200:80)
├── 15 ACL rules (Incus baseline security)
├── 8 DNS records (forward + reverse per instance)
└── DHCP options (IPv4 lease=3600, MTU=1442, DNS=192.168.100.1)

Key config in other_config:

  • subnet=10.10.10.0/24 — Incus IPAM range
  • exclude_ips=10.10.10.1 10.10.10.254 ... — reserved IPs (gateway, broadcast, static assignments)
  • ipv6_prefix=fd42:842b:1e5f:b27::/64 — SLAAC prefix

External switch (incus-net8-ls-ext): Bridges the logical router to the physical UPLINK network.

incus-net8-ls-ext
├── 1 localnet port (incus-net8-ls-ext-lsp-provider)
│   └── options: network_name=UPLINK
└── 1 router port (incus-net8-ls-ext-lsp-router)
    └── nat_addresses: 10:66:6a:d4:30:c7 192.168.103.200
        is_chassis_resident("cr-incus-net8-lr-lrp-ext")

The localnet port type is special — it tells OVN to bridge this logical switch to a physical network via ovn-bridge-mappings on each chassis. The mapping UPLINK:incusovn7 connects it to the OVS provider bridge.

Logical Router

Router (incus-net8-lr): Connects internal overlay to external UPLINK.

incus-net8-lr
├── lrp-int: 10.10.10.1/24, fd42:842b:1e5f:b27::1/64
│   MAC: 10:66:6a:d4:30:c7, MTU: 1442
│   IPv6 RA: periodic, DHCPv6 stateless, DNSSL=incus
│
├── lrp-ext: 192.168.103.200/22
│   MAC: 10:66:6a:d4:30:c7, MTU: 1442
│   HA Chassis Group: incus-net8 (gateway on oc-node-03)
│
├── NAT:
│   └── SNAT: 10.10.10.0/24 → 192.168.103.200 (stateful)
│
├── Static Routes:
│   ├── 0.0.0.0/0 → 192.168.100.1 via lrp-ext  (default gateway)
│   └── 192.168.103.200/32 → 10.10.10.1          (hairpin VIP route)
│
├── Load Balancer: (same LB as on ls-int)
│   └── 192.168.103.200:80 → 10.10.10.50:80, 10.10.10.51:80
│
├── Policies:
│   ├── priority 600: allow ip4.src == $incus_net8_routes_ip4
│   ├── priority 600: allow ip6.src == $incus_net8_routes_ip6
│   └── priority 500: drop (inport == lrp-int)  ← deny-by-default
│
└── Options:
    ├── always_learn_from_arp_request=false
    └── dynamic_neigh_routers=true

Router policies explained: The priority-500 drop rule blocks all traffic from the internal network by default. The priority-600 allow rules create exceptions for the network's own IP ranges. This prevents instances from spoofing source IPs outside their assigned ranges.

Hairpin VIP route: The 192.168.103.200/32 → 10.10.10.1 route handles the case where an instance on net-prod accesses the VIP from inside the overlay. Without this, the router would try to route the VIP out the external port, but the packet originated internally.

Load Balancer

Name:     incus-net8-lb-192.168.103.200-tcp
Protocol: tcp
VIPs:     192.168.103.200:80 → 10.10.10.50:80, 10.10.10.51:80

Incus creates this LB from incus network forward or from Aether's HAProxy deployment. The LB is attached to both the logical switch (ls-int) and the logical router (lr). This dual attachment ensures the LB intercepts traffic regardless of where it enters:

  • On the router: Catches external traffic (LAN → VIP)
  • On the switch: Catches internal traffic (instance → VIP)

DHCP and DNS

DHCP Options (IPv4):

  • Server: 10.10.10.1, MAC: 10:66:6a:d4:30:c7
  • Lease time: 3600s, MTU: 1442 (Geneve overhead: 1500 - 58 = 1442)
  • DNS: 192.168.100.1 (upstream resolver from UPLINK config)
  • Domain: incus, search list: incus

DHCP Options (IPv6):

  • Stateless DHCPv6 (SLAAC for addresses, DHCPv6 for DNS)
  • DNS: fd42:842b:1e5f:b27::1

DNS Records: Incus creates forward and reverse DNS entries for every instance. The DNS server runs inside the OVN router (lrp-int port).

ffsdn-haproxy-52-01.incus → 10.10.10.50, fd42:842b:1e5f:b27:1266:6aff:fe67:3482
ffsdn-haproxy-52-02.incus → 10.10.10.51, fd42:842b:1e5f:b27:1266:6aff:fe5f:418f
nginx-lb-01.incus         → 10.10.10.60, fd42:842b:1e5f:b27:1266:6aff:fe2a:da10
nginx-lb-02.incus         → 10.10.10.61, fd42:842b:1e5f:b27:1266:6aff:fe69:888e
nginx-lb-03.incus         → 10.10.10.62, fd42:842b:1e5f:b27:1266:6aff:fe90:4ab8

ACLs (Baseline Security)

Incus creates a default ACL set on the internal switch. These are not user-configurable — they form the baseline security policy:

Priority Direction Match Action Purpose
200 to-lport arp || nd allow ARP/ND always allowed
200 to-lport icmp4.type == {3,11,12} && ip.ttl == 255 allow ICMP errors
200 to-lport igmp && ip.ttl == 1 && ip4.mcast allow IGMP
200 to-lport Router port ping echo reply allow Router ping
200 to-lport DHCP to router port allow DHCP relay
200 to-lport DNS to router port allow DNS queries
200 to-lport nd_ra from router allow IPv6 RA
200 to-lport nd_rs to router allow IPv6 RS
200 to-lport tcp.flags == 0x014 allow TCP RST+ACK

All ACLs are to-lport (ingress to the logical port), which means they filter traffic arriving at a port. There are no explicit from-lport rules — the default for egress is allow.

OVN Southbound Database (Physical Layer)

The SB database maps logical topology to physical infrastructure. It's computed by ovn-northd from the NB database and consumed by ovn-controller on each chassis.

Chassis

Each Incus cluster member registers as a chassis:

Chassis Hostname Geneve IP OVS UUID
b840b5b2-... oc-node-01 192.168.102.140 298b76a9-...
4652b51f-... oc-node-02 192.168.102.141 5ee7906c-...
3f7400f9-... oc-node-03 192.168.102.142 fd0c3d36-...

All chassis use Geneve encapsulation with checksum enabled (csum=true). Each chassis advertises ovn-bridge-mappings=UPLINK:incusovn7.

Key other_config values:

  • datapath-type=system — kernel datapath (not DPDK)
  • ovn-bridge-mappings=UPLINK:incusovn7 — maps logical "UPLINK" network to OVS bridge
  • ovn-monitor-all=false — each chassis only gets flows relevant to its local ports
  • ct-commit-nat-v2=true, ct-next-zone=true — modern conntrack features

Datapath Bindings (Tunnel Keys)

Each logical switch/router gets a unique tunnel key used inside Geneve:

Tunnel Key Datapath OVN Object
1 incus-net8-lr Logical Router
2 incus-net8-ls-ext External Switch
3 incus-net8-ls-int Internal Switch

When a packet traverses a Geneve tunnel, the tunnel key (VNI) identifies which datapath it belongs to.

Port Bindings

Port bindings map logical ports to physical chassis:

Logical Port Type Chassis Tunnel Key
Instance ports (8 total) VIF Respective chassis 2-10
cr-incus-net8-lr-lrp-ext chassisredirect oc-node-03 2
incus-net8-lr-lrp-ext patch (distributed) 1
incus-net8-lr-lrp-int patch (distributed) 3
incus-net8-ls-ext-lsp-provider localnet (all chassis) 2
incus-net8-ls-ext-lsp-router patch (distributed) 1

Key port types:

  • VIF (Virtual Interface): Regular instance ports, bound to a specific chassis. Each has a MAC+IP pair for the instance.
  • patch: Connects two OVN datapaths (switch↔router). Distributed — processed locally on every chassis.
  • chassisredirect: The gateway port. Centralizes external-facing traffic on a single chassis for SNAT/DNAT. Bound to oc-node-03.
  • localnet: Maps to a physical network. Present on every chassis via ovn-bridge-mappings.

HA Chassis Group (Gateway Failover)

The gateway port uses an HA chassis group for failover:

HA Chassis Group: incus-net8
├── oc-node-03  priority 22372  ← active gateway
├── oc-node-02  priority 18011  ← first failover
└── oc-node-01  priority 12213  ← second failover

Incus assigns random-looking priorities (likely based on a hash). If oc-node-03 goes down, the cr-incus-net8-lr-lrp-ext binding migrates to oc-node-02, then oc-node-01 if needed. BFD (Bidirectional Forwarding Detection) between chassis enables fast failure detection.

Logical Flows (Compiled Pipeline)

The lflow pipeline is the compiled form of all NB rules. For ls-int ingress:

Table Name Key Rules
0 ls_in_check_port_sec Drop multicast src, VLAN tagged
1 ls_in_apply_port_sec Enforce port security
4 ls_in_pre_acl Skip ACL for router port traffic
5 ls_in_pre_lb Pre-process for LB (set reg0[2] for CT)
6 ls_in_pre_stateful LB intercept: dst=192.168.103.200:80 → ct_lb_mark
7 ls_in_acl_hint Compute ACL hints from conntrack state
8 ls_in_acl Apply ACLs, track connections

The critical LB flow in table 6:

priority=120, match=(reg0[2] == 1 && ip4.dst == 192.168.103.200 && tcp.dst == 80)
action=(reg1 = 192.168.103.200; reg2[0..15] = 80; ct_lb_mark;)

This intercepts packets to the VIP and sends them through conntrack load balancing, which selects a backend and rewrites the destination.

OVS Physical Layer

Each IncusOS node runs OVS 3.6.1 with two bridges.

Bridge Architecture

flowchart LR
    subgraph provider["incusovn7 · provider bridge"]
        nic["physical NIC"]
        intport["internal port"]
        patch1["patch to br-int"]
    end

    subgraph integration["br-int · integration bridge"]
        veth["instance veth ports"]
        ovntun["Geneve tunnels"]
        patch2["patch to incusovn7"]
    end

    patch1 <-->|"patch port"| patch2

    classDef prov fill:#0072B2,color:#fff,stroke:#005a8e
    classDef integ fill:#009E73,color:#fff,stroke:#007a5e

    class nic,intport,patch1 prov
    class veth,ovntun,patch2 integ

    style provider fill:#e0eef8,stroke:#0072B2
    style integration fill:#e6f5f0,stroke:#009E73

br-int (integration bridge):

  • fail_mode: secure — drops all traffic if OVN controller disconnects
  • Contains all instance veth ports, Geneve tunnel ports, and the patch port to the provider bridge
  • All OVN logical processing (ACLs, NAT, LB, routing) happens here via OpenFlow rules installed by ovn-controller

incusovn7 (provider bridge):

  • Named after the Incus-managed OVS bridge (incusovn + network ID)
  • Contains the physical NIC port, an internal port, and a patch port to br-int
  • Single OpenFlow rule: priority=0 actions=NORMAL (standard L2 switching)
  • This bridge is the on-ramp/off-ramp between OVN and the physical network

Geneve Tunnels (Full Mesh)

Every pair of chassis has a Geneve tunnel with BFD health monitoring:

graph LR
    n1(("oc-node-01<br/>.140"))
    n2(("oc-node-02<br/>.141"))
    n3(("oc-node-03<br/>.142"))

    n1 <-->|"Geneve 6081"| n2
    n2 <-->|"Geneve 6081"| n3
    n1 <-->|"Geneve 6081"| n3

    classDef chassis fill:#009E73,color:#fff,stroke:#007a5e

    class n1,n2,n3 chassis
Source Destination OVS Port Name BFD State
.140 (node-01) .141 (node-02) ovn-4652b5-0 forwarding=true
.140 (node-01) .142 (node-03) ovn-3f7400-0 forwarding=true
.141 (node-02) .140 (node-01) ovn-b840b5-0 forwarding=true
.141 (node-02) .142 (node-03) ovn-3f7400-0 forwarding=true
.142 (node-03) .140 (node-01) ovn-b840b5-0 forwarding=true
.142 (node-03) .141 (node-02) ovn-4652b5-0 forwarding=true

Tunnel port names use the first 6 hex chars of the remote chassis name. Options: key=flow (tunnel key set per-packet from OVN datapath), csum=true.

Veth Port Mapping

Each instance on net-prod gets a veth pair: one end in the container's network namespace, the other plugged into br-int. The OVS external_ids field links the veth to its OVN logical port.

oc-node-01:

veth OVN Port (iface-id) Instance OVS ofport
veth8bd9abf3 ...c446bd6a...-eth0 nginx-lb-01 (10.10.10.60) 8
vethfc99b1ec ...b90abddd...-eth0 test-web-tier-web-1 (10.10.10.2) 5
veth07cf17cf ...0880f911...-eth0 ffsdn-haproxy-52-01 (10.10.10.50) 9

oc-node-02:

veth OVN Port (iface-id) Instance
veth3352cb4f ...a2cad635...-eth0 ffsdn-haproxy-52-02 (10.10.10.51)
vethc0b4d30a ...07270515...-eth0 nginx-lb-02 (10.10.10.61)
vethd2307cca ...b2cbf869...-eth0 test-web-tier-web-2 (10.10.10.3)

oc-node-03:

veth OVN Port (iface-id) Instance
veth9306f6ef ...292ce9ce...-eth0 nginx-lb-03 (10.10.10.62)
veth4701597a ...1f056d1a...-eth0 test-web-tier-app-1 (10.10.10.4)

Packet Trace: LAN → VIP → HAProxy → Nginx → Return

Complete path for an HTTP request from a LAN client to http://192.168.103.200/ (the HAProxy VIP).

1. Client → Gateway Chassis

Client (192.168.1.x) sends TCP SYN to 192.168.103.200:80
→ LAN switch forwards to oc-node-03 (gateway chassis)
   192.168.103.200 is announced by oc-node-03 via the
   cr-incus-net8-lr-lrp-ext chassisredirect port
→ Packet enters physical NIC → incusovn7 bridge
→ OVS NORMAL action → patch port → br-int

Why oc-node-03? The external router port (lrp-ext) uses HA chassis scheduling, and oc-node-03 has the highest priority (22372). OVN makes oc-node-03's OVS respond to ARP for 192.168.103.200, directing all external traffic to this node.

2. br-int → OVN Router (External Processing)

br-int receives packet on patch port
→ OpenFlow table 0: classify as localnet traffic (metadata=0x2, ls-ext)
→ Pipeline: ls-ext ingress → router pipeline
→ Router receives on lrp-ext (192.168.103.200/22)

3. OVN Load Balancer DNAT

Router detects dst=192.168.103.200:80 matches LB
→ ct_lb_mark: conntrack creates new entry
→ DNAT: rewrite dst to 10.10.10.50:80 or 10.10.10.51:80
   (round-robin selection, conntrack-aware)
→ Packet now has dst=10.10.10.50:80 (say, haproxy-01)

The LB is processed in the router pipeline because it's attached to both the router and the internal switch. For external traffic, the router processes it first.

4. Router → Internal Switch

Router forwards via lrp-int (10.10.10.1)
→ Enters ls-int pipeline
→ ACL check (baseline allows established connections)
→ Destination lookup: 10.10.10.50 → ffsdn-haproxy-52-01
→ Port binding: haproxy-01 is on oc-node-01

5. Geneve Tunnel (Cross-Chassis)

Since the gateway is on oc-node-03 but the destination (haproxy-01) is on oc-node-01:

br-int on oc-node-03:
→ Output action: tunnel to oc-node-01
→ Geneve encapsulate:
   - Outer: src=192.168.102.142, dst=192.168.102.140, UDP:6081
   - VNI (tunnel key): 3 (ls-int datapath)
   - TUN_METADATA0: encodes reg14 (destination port) + reg15
→ Physical NIC sends to LAN

oc-node-01 receives Geneve packet:
→ br-int decapsulates
→ Restores metadata from tunnel headers
→ Delivers to veth07cf17cf (haproxy-01's veth, ofport 9)
→ Packet enters container's network namespace

6. HAProxy Processing

HAProxy receives HTTP request on 10.10.10.50:80
→ HAProxy selects backend: nginx-lb-01 (10.10.10.60)
→ Opens new TCP connection to 10.10.10.60:80
→ Proxies request

7. HAProxy → Nginx (Same or Different Chassis)

If nginx-lb-01 is on the same node (oc-node-01):

→ br-int local delivery (no tunnel needed)
→ veth8bd9abf3 (nginx-lb-01, ofport 8)

If HAProxy chose nginx-lb-02 (oc-node-02) or nginx-lb-03 (oc-node-03):

→ Geneve tunnel to remote node
→ Same encap/decap as step 5

8. Return Path

Nginx response → HAProxy (reverse of step 7)
→ HAProxy response → OVN (enters ls-int as src=10.10.10.50)
→ ls-int → router (dst is LAN client, matches default route)
→ Router SNAT: src 10.10.10.50 → 192.168.103.200
→ lrp-ext → ls-ext → provider bridge → physical NIC → LAN
→ Client receives response from 192.168.103.200

Important: The return path from HAProxy to the LAN client goes through the gateway chassis (oc-node-03) because SNAT is centralized there. If HAProxy is on oc-node-01, the reply tunnels to oc-node-03 for SNAT, then exits to the LAN.

MTU: Why 1442?

Standard Ethernet MTU is 1500 bytes. Geneve adds 58 bytes of overhead:

Outer Ethernet:  14 bytes
Outer IP:        20 bytes
Outer UDP:        8 bytes
Geneve header:   16 bytes (8 base + 8 metadata)
                ─────────
Total overhead:  58 bytes
Inner MTU:     1442 bytes (1500 - 58)

Incus sets bridge.mtu=1442 on net-prod and propagates this via:

  • DHCP option: mtu=1442
  • Router port option: gateway_mtu=1442
  • IPv6 RA: mtu=1442

Inspection Tools

Using ovn-inspect

The incusos/helpers/ovn-inspect script provides structured inspection:

# NB database (logical topology)
incusos/helpers/ovn-inspect --nb

# SB database (physical bindings)
incusos/helpers/ovn-inspect --sb

# OVS on each node (deploys temp containers, cleans up after)
incusos/helpers/ovn-inspect --ovs

# Everything
incusos/helpers/ovn-inspect --full

# Trace a VIP's packet path
incusos/helpers/ovn-inspect --trace 192.168.103.200

# Dry run (show commands without executing)
incusos/helpers/ovn-inspect --nb --dry-run

Manual Commands

All NB/SB commands go through the ovn-central container:

# NB: Logical topology
incus exec oc-node-01:ovn-central -- ovn-nbctl ls-list
incus exec oc-node-01:ovn-central -- ovn-nbctl lr-list
incus exec oc-node-01:ovn-central -- ovn-nbctl lsp-list incus-net8-ls-int
incus exec oc-node-01:ovn-central -- ovn-nbctl lr-nat-list incus-net8-lr
incus exec oc-node-01:ovn-central -- ovn-nbctl lb-list
incus exec oc-node-01:ovn-central -- ovn-nbctl acl-list incus-net8-ls-int

# SB: Physical bindings
incus exec oc-node-01:ovn-central -- ovn-sbctl show
incus exec oc-node-01:ovn-central -- ovn-sbctl list Chassis
incus exec oc-node-01:ovn-central -- ovn-sbctl list Port_Binding
incus exec oc-node-01:ovn-central -- ovn-sbctl lflow-list incus-net8-ls-int

# OVS: Physical layer (requires privileged container with /run/openvswitch)
ovs-vsctl show
ovs-ofctl dump-flows br-int
ovs-vsctl get Interface <veth> external_ids

Incus CLI Cross-Reference

# See OVN network config
incus network show oc-node-01:net-prod --target oc-node-01

# See UPLINK config
incus network show oc-node-01:UPLINK --target oc-node-01

# See which instances use net-prod
incus list oc-node-01: -f csv -c n4l | grep "10.10.10"

# Network forwards (LB VIPs managed via Incus)
incus network forward list oc-node-01:net-prod

Key Architectural Insights

  1. Incus fully manages OVN: Users never interact with OVN directly. incus network create, incus network forward, and instance NIC configs translate to OVN objects automatically.

  2. Single gateway chassis: All external traffic (SNAT, DNAT, LB for external VIPs) is centralized on one node. This is a potential bottleneck but simplifies state management. HA failover handles node failures.

  3. Distributed routing for internal traffic: East-west traffic between instances on the same switch is fully distributed. No traffic goes through the gateway unless it needs SNAT/DNAT.

  4. LB is in OVN, not HAProxy: The VIP load balancing between HAProxy instances is done by OVN's built-in L4 LB (conntrack-based). HAProxy then does L7 load balancing to nginx backends. This is a two-tier LB architecture.

  5. BFD for fast failover: All Geneve tunnels have BFD enabled, which detects chassis failures in ~3×detection-interval (typically <1s), much faster than relying on OVN cluster heartbeats.

  6. MTU must be consistent: The 1442 byte MTU is critical. If any path (physical switch, hypervisor NIC) has MTU < 1500, Geneve encapsulated packets will be fragmented or dropped.