Add VLAN 69 support, OVN networking guide, and lab-networking config
VLAN tagging: incusos-proxmox now reads optional `vlan` key from proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec in both SSH and API creation paths. Tested manually on VMID 850 — VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is Proxmox-level only; IncusOS and Incus instances are unaware of it. Networking guide: comprehensive tutorial covering bridge networking, OVN overlay setup on IncusOS (services disabled by default, must enable via REST API), cross-node connectivity, isolation, ACLs, peering, load balancers, forwards, and DNS. All scenarios tested on a 4-node OC-managed cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
2e2447de06
commit
9e3201393e
56
CLAUDE.md
56
CLAUDE.md
|
|
@ -28,6 +28,7 @@ incu-contrib/
|
|||
└── notes/ # Research notes and reference material
|
||||
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
||||
├── operations-center-guide.md # Operations Center provisioning & management
|
||||
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
||||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
||||
└── utm-support.md # UTM support design document (future)
|
||||
```
|
||||
|
|
@ -117,6 +118,13 @@ incu-contrib/
|
|||
- `balloon=0` -- IncusOS manages memory internally
|
||||
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
||||
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
||||
- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag
|
||||
to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM
|
||||
on the tagged VLAN instead of the native/untagged network. The current lab
|
||||
uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag,
|
||||
VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the
|
||||
Proxmox VM level only — IncusOS and Incus instances inside the VM are
|
||||
unaware of it. The setting is optional: omit `vlan:` for untagged access.
|
||||
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
||||
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
||||
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
||||
|
|
@ -744,6 +752,54 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
|||
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
||||
not "sudo apt install incus".
|
||||
|
||||
### Incus networking (OVN)
|
||||
|
||||
- **Bridge networks are node-local**: each cluster member has its own
|
||||
independent bridge. Instances on the same bridge (same node) can
|
||||
communicate; cross-node instances CANNOT. Each bridge has the same subnet
|
||||
(e.g., 10.0.0.1/24) but they are separate L2 domains.
|
||||
- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes.
|
||||
Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
|
||||
+ client services + physical uplink network.
|
||||
- **IncusOS OVN services are disabled by default**: must be enabled via the
|
||||
IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring
|
||||
Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection`
|
||||
fails with `db.sock not found`.
|
||||
- **OVN service enable API call**:
|
||||
```bash
|
||||
incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
|
||||
"config": {
|
||||
"database": "tcp:<SB_HOST>:6642",
|
||||
"enabled": true,
|
||||
"tunnel_address": "<THIS_NODE_LAN_IP>",
|
||||
"tunnel_protocol": "geneve"
|
||||
},
|
||||
"state": {}
|
||||
}'
|
||||
```
|
||||
`database` is the **southbound** DB (port 6642), NOT northbound (6641).
|
||||
- **OVN control plane as container**: deploy `ovn-central` package in a
|
||||
Debian container on the cluster. Use proxy devices to expose NB (6641)
|
||||
and SB (6642) ports on the host's LAN IP so all nodes can reach it.
|
||||
- **Setup sequence** (order matters):
|
||||
1. Deploy OVN control plane container
|
||||
2. Enable OVN services on ALL IncusOS nodes
|
||||
3. `incus config set network.ovn.northbound_connection tcp:<host>:6641`
|
||||
4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes)
|
||||
5. Create physical uplink network (two-step cluster pattern)
|
||||
6. Create OVN network with `--type=ovn network=UPLINK`
|
||||
- **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC),
|
||||
`ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses,
|
||||
`ipv4.gateway` is the LAN gateway in CIDR format.
|
||||
- **OVN network isolation**: multiple OVN networks are fully isolated.
|
||||
Instances on different networks cannot communicate, even on the same node.
|
||||
Network peering (`incus network peer create`) enables cross-network routing.
|
||||
- **OVN features tested**: cross-node connectivity, network isolation,
|
||||
ACLs (per-source blocking), network peering, L4 load balancers (connection-
|
||||
based hashing, not round-robin), network forwards (port forwarding to LAN
|
||||
IPs), DNS resolution (per-network, hostname.incus domain).
|
||||
- See `notes/networking-guide.md` for full tutorial with test results.
|
||||
|
||||
### Migration into Incus
|
||||
|
||||
- **`incus-migrate`**: official tool for importing disk images, running
|
||||
|
|
|
|||
|
|
@ -389,6 +389,7 @@ node: pve
|
|||
storage: local-lvm
|
||||
iso_storage: local
|
||||
bridge: vmbr0
|
||||
vlan: 69
|
||||
pool: IncusLab
|
||||
ssh_user: root
|
||||
```
|
||||
|
|
|
|||
|
|
@ -0,0 +1,42 @@
|
|||
# lab-networking.yaml - 4-node cluster for networking exploration
|
||||
#
|
||||
# Deploys 4 Incus nodes using an OC-provisioned ISO for a networking
|
||||
# tutorial covering bridge, OVN overlay, ACLs, load balancers, and
|
||||
# LAN integration.
|
||||
#
|
||||
# Usage:
|
||||
# # First: deploy OC server with lab-oc-deploy.yaml
|
||||
# # Then: create OC provisioning token + download ISO
|
||||
# # Then deploy with --iso:
|
||||
# incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run examples/lab-networking.yaml
|
||||
# incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-networking.yaml
|
||||
# incusos-proxmox --status examples/lab-networking.yaml
|
||||
# incusos-proxmox --cleanup --deep examples/lab-networking.yaml
|
||||
#
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
#
|
||||
# Resource budget: 4 nodes x 8 GiB = 32 GiB RAM
|
||||
# Plus OC server (4 GiB) = 36 GiB total
|
||||
|
||||
defaults:
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 50
|
||||
start_vmid: 400
|
||||
|
||||
vms:
|
||||
- name: net-node-01
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
|
||||
- name: net-node-02
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
|
||||
- name: net-node-03
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
|
||||
- name: net-node-04
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
|
|
@ -166,6 +166,7 @@ ${BOLD}PROXMOX CONNECTION CONFIG${RESET}
|
|||
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||
iso_storage: local # ISO storage (default: local)
|
||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||
vlan: 69 # VLAN tag for VM NIC (optional)
|
||||
pool: IncusLab # Resource pool (optional)
|
||||
ssh_user: root # SSH user (default: root)
|
||||
|
||||
|
|
@ -1294,6 +1295,7 @@ PVE_NODE=""
|
|||
PVE_STORAGE=""
|
||||
PVE_ISO_STORAGE=""
|
||||
PVE_BRIDGE=""
|
||||
PVE_VLAN=""
|
||||
PVE_METHOD=""
|
||||
PVE_SSH_USER=""
|
||||
PVE_API_TOKEN_ID=""
|
||||
|
|
@ -1393,6 +1395,7 @@ json.dump(result, sys.stdout, indent=2)
|
|||
val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||
val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
|
||||
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||
|
|
@ -1415,6 +1418,7 @@ load_config() {
|
|||
val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||
|
|
@ -1852,7 +1856,9 @@ pve_create_vm() {
|
|||
cmd+=" --scsi0 ${PVE_STORAGE}:${disk_gib},iothread=1"
|
||||
cmd+=" --ide2 ${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom"
|
||||
cmd+=" --ide3 ${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom"
|
||||
cmd+=" --net0 virtio,bridge=${PVE_BRIDGE}"
|
||||
local net0_spec="virtio,bridge=${PVE_BRIDGE}"
|
||||
[[ -n "$PVE_VLAN" ]] && net0_spec+=",tag=${PVE_VLAN}"
|
||||
cmd+=" --net0 ${net0_spec}"
|
||||
cmd+=" --boot order=ide2\\;scsi0"
|
||||
cmd+=" --agent 1"
|
||||
if [[ -n "$PVE_POOL" ]]; then
|
||||
|
|
@ -1892,10 +1898,14 @@ payload = {
|
|||
'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1',
|
||||
'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom',
|
||||
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom',
|
||||
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
||||
'boot': 'order=ide2;scsi0',
|
||||
'agent': 1,
|
||||
}
|
||||
net0 = 'virtio,bridge=${PVE_BRIDGE}'
|
||||
vlan = '${PVE_VLAN}'
|
||||
if vlan:
|
||||
net0 += ',tag=' + vlan
|
||||
payload['net0'] = net0
|
||||
pool = '${PVE_POOL}'
|
||||
if pool:
|
||||
payload['pool'] = pool
|
||||
|
|
@ -2866,7 +2876,11 @@ print_plan() {
|
|||
echo -e " Proxmox host: ${CYAN}${PVE_HOST}${RESET} (${PVE_METHOD})"
|
||||
echo -e " Node: ${PVE_NODE}"
|
||||
echo -e " Storage: ${PVE_STORAGE} (disk), ${PVE_ISO_STORAGE} (iso)"
|
||||
echo -e " Bridge: ${PVE_BRIDGE}"
|
||||
if [[ -n "$PVE_VLAN" ]]; then
|
||||
echo -e " Bridge: ${PVE_BRIDGE} (VLAN ${PVE_VLAN})"
|
||||
else
|
||||
echo -e " Bridge: ${PVE_BRIDGE}"
|
||||
fi
|
||||
if [[ -n "$PVE_POOL" ]]; then
|
||||
echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -0,0 +1,799 @@
|
|||
# Incus Networking Guide — Bridge, OVN, and LAN Integration
|
||||
|
||||
A hands-on tutorial covering Incus networking from basic bridge networking
|
||||
to OVN overlay networks, tested on a 4-node IncusOS cluster managed by
|
||||
Operations Center on Proxmox VE.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- 4-node IncusOS cluster (this guide uses `net-node-01` through `net-node-04`)
|
||||
- Operations Center server (for OC-managed deployment)
|
||||
- Proxmox VE host with `incusos-proxmox` deployment tool
|
||||
- `incus` client configured with remotes for all cluster nodes
|
||||
|
||||
### Lab configuration
|
||||
|
||||
```yaml
|
||||
# incusos/examples/lab-networking.yaml
|
||||
defaults:
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 50
|
||||
start_vmid: 400
|
||||
|
||||
vms:
|
||||
- name: net-node-01
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
- name: net-node-02
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
- name: net-node-03
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
- name: net-node-04
|
||||
app: incus
|
||||
apply_defaults: false
|
||||
```
|
||||
|
||||
**Resource budget:** OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total.
|
||||
|
||||
### Cluster formation
|
||||
|
||||
Nodes were deployed with `apply_defaults: false`, so no default storage pool,
|
||||
network, or profile devices exist after installation. After forming the cluster
|
||||
manually (OC's `cluster add` requires "needs update: false" which was blocked
|
||||
by a v0.3.0 tracking issue), we created these resources:
|
||||
|
||||
```bash
|
||||
# Storage pool (per-member source, then cluster-wide finalize)
|
||||
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||
incus storage create net-node-01:local zfs \
|
||||
source=local/incus zfs.pool_name=local/incus --target "$node"
|
||||
done
|
||||
incus storage create net-node-01:local zfs
|
||||
|
||||
# Bridge network (per-member pending, then finalize with config)
|
||||
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||
incus network create net-node-01:incusbr0 --type bridge --target "$node"
|
||||
done
|
||||
incus network create net-node-01:incusbr0 --type bridge \
|
||||
ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none
|
||||
|
||||
# Default profile
|
||||
incus profile device add net-node-01:default root disk path=/ pool=local
|
||||
incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Bridge Networking (Default)
|
||||
|
||||
### What is bridge networking?
|
||||
|
||||
When Incus creates a bridge network (type `bridge`), it creates a Linux bridge
|
||||
on each cluster node. Each bridge is **independent** — there is no connectivity
|
||||
between bridges on different nodes. This is the simplest networking mode and
|
||||
works without any additional infrastructure.
|
||||
|
||||
### Bridge topology
|
||||
|
||||
```
|
||||
Node 1 (net-node-01) Node 2 (net-node-02)
|
||||
┌─────────────────────┐ ┌─────────────────────┐
|
||||
│ incusbr0 │ │ incusbr0 │
|
||||
│ 10.0.0.1/24 │ │ 10.0.0.1/24 │
|
||||
│ ┌───┐ ┌───┐ │ │ ┌───┐ │
|
||||
│ │c1 │ │c2 │ │ │ │c3 │ │
|
||||
│ │.202│ │.40│ │ │ │.52│ │
|
||||
│ └───┘ └───┘ │ │ └───┘ │
|
||||
│ │ │ │ │ │ │
|
||||
│ ──┴──────┴── │ │ ──┴── │
|
||||
│ (bridge) │ │ (bridge) │
|
||||
│ │ │ │ │ │
|
||||
│ NAT ──> ens18 │ │ NAT ──> ens18 │
|
||||
│ 192.168.1.209 │ │ 192.168.1.150 │
|
||||
└─────────────────────┘ └─────────────────────┘
|
||||
│ │
|
||||
────┴──────────────────────────────────┴────
|
||||
LAN (192.168.1.0/24)
|
||||
```
|
||||
|
||||
**Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24).
|
||||
The bridges are NOT connected to each other.
|
||||
|
||||
### Test results
|
||||
|
||||
#### Same-node communication: PASS
|
||||
|
||||
Containers on the same node share the same bridge (same L2 domain):
|
||||
|
||||
```bash
|
||||
# test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01
|
||||
$ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40
|
||||
64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms
|
||||
# 0% packet loss, ~0.03ms latency
|
||||
```
|
||||
|
||||
#### NAT to internet: PASS
|
||||
|
||||
The bridge has `ipv4.nat=true`, so containers can reach the internet:
|
||||
|
||||
```bash
|
||||
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
|
||||
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms
|
||||
# 0% packet loss, ~10ms to Cloudflare DNS
|
||||
```
|
||||
|
||||
#### Cross-node communication: FAIL
|
||||
|
||||
Containers on different nodes CANNOT reach each other:
|
||||
|
||||
```bash
|
||||
# test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52)
|
||||
$ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52
|
||||
From 10.0.0.202 icmp_seq=1 Destination Host Unreachable
|
||||
# 100% packet loss
|
||||
```
|
||||
|
||||
### When to use bridge networking
|
||||
|
||||
- Single-node setups (development, testing)
|
||||
- Workloads that don't need cross-node communication
|
||||
- Simple NAT-to-internet access
|
||||
- As an uplink for OVN networks (see Part 2)
|
||||
|
||||
### Bridge network configuration options
|
||||
|
||||
```bash
|
||||
# View current config
|
||||
incus network show net-node-01:incusbr0
|
||||
|
||||
# Key settings
|
||||
ipv4.address # Bridge IP and subnet (e.g., 10.0.0.1/24)
|
||||
ipv4.nat # Enable NAT for internet access (true/false)
|
||||
ipv4.dhcp # Enable DHCP server (true/false, default true)
|
||||
ipv4.dhcp.ranges # DHCP range (e.g., 10.0.0.100-10.0.0.200)
|
||||
dns.domain # DNS domain for instances (e.g., incus)
|
||||
dns.mode # DNS mode: managed, dynamic, or none
|
||||
ipv6.address # IPv6 address (set to "none" to disable)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 2: OVN Overlay Networking
|
||||
|
||||
### Why OVN?
|
||||
|
||||
Bridge networking is node-local. To enable cross-node communication without
|
||||
routing hacks, you need an overlay network. OVN (Open Virtual Network)
|
||||
provides:
|
||||
|
||||
- **Cross-node L2 connectivity** via Geneve tunnels
|
||||
- **Network isolation** (multiple independent virtual networks)
|
||||
- **Distributed routing** (no single point of failure)
|
||||
- **Network ACLs** (firewall rules at the network level)
|
||||
- **Load balancers** (built-in L4 load balancing)
|
||||
- **Network forwards** (port forwarding from external IPs)
|
||||
- **DNS** (automatic hostname resolution between instances)
|
||||
|
||||
### OVN architecture on IncusOS
|
||||
|
||||
IncusOS includes OVN **client** components only:
|
||||
- `ovn-controller` (data plane)
|
||||
- `ovs-vswitchd` (Open vSwitch)
|
||||
|
||||
The OVN **control plane** must run separately:
|
||||
- `ovn-northd` (translates logical to physical flows)
|
||||
- `ovsdb-server` (northbound + southbound databases)
|
||||
|
||||
Options for deploying the control plane:
|
||||
1. As a container on the Incus cluster itself (used in this guide)
|
||||
2. On an external host
|
||||
|
||||
### OVN topology
|
||||
|
||||
```
|
||||
┌──────────────────────────────────┐
|
||||
│ OVN Control Plane │
|
||||
│ (container: ovn-central) │
|
||||
│ ovn-northd + ovsdb-server │
|
||||
│ NB: tcp:192.168.1.209:6641 │
|
||||
│ SB: tcp:192.168.1.209:6642 │
|
||||
└──────────┬───────────────────────┘
|
||||
│ (proxy devices)
|
||||
┌─────────────────────────┼─────────────────────────┐
|
||||
│ │ │
|
||||
Node 1 (net-node-01) Node 2 (net-node-02) Node 3 (net-node-03)
|
||||
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
|
||||
│ ovn-controller │ │ ovn-controller │ │ ovn-controller │
|
||||
│ ovs-vswitchd │ │ ovs-vswitchd │ │ ovs-vswitchd │
|
||||
│ │ │ │ │ │
|
||||
│ ┌───┐ ┌───┐ │ │ ┌───┐ │ │ ┌───┐ │
|
||||
│ │c1 │ │c2 │ │ │ │c3 │ │ │ │c4 │ │
|
||||
│ │.2 │ │.3 │ │ │ │.4 │ │ │ │.5 │ │
|
||||
│ └─┬─┘ └─┬─┘ │ │ └─┬─┘ │ │ └─┬─┘ │
|
||||
│ └──┬───┘ │ │ │ │ │ │ │
|
||||
│ OVN logical │ │ OVN logical │ │ OVN logical │
|
||||
│ switch │ │ switch │ │ switch │
|
||||
│ 10.10.10.0/24 │ │ 10.10.10.0/24 │ │ 10.10.10.0/24 │
|
||||
│ │ │ │ │ │ │ │ │
|
||||
│ Geneve tunnel ──┼──┼── Geneve tunnel ──┼──┼── Geneve tunnel │
|
||||
│ 192.168.1.209 │ │ 192.168.1.150 │ │ 192.168.1.13 │
|
||||
└───────────────────┘ └───────────────────┘ └───────────────────┘
|
||||
│ │ │
|
||||
────┴──────────────────────┴──────────────────────┴────
|
||||
LAN (192.168.1.0/24)
|
||||
```
|
||||
|
||||
**Key difference from bridge:** All instances share a single logical switch
|
||||
connected via Geneve tunnels. Cross-node traffic is encapsulated and sent
|
||||
over the LAN between nodes.
|
||||
|
||||
### Step 1: Deploy OVN control plane
|
||||
|
||||
Launch a container to run the OVN central services:
|
||||
|
||||
```bash
|
||||
# Launch OVN central container
|
||||
incus launch images:debian/12 net-node-01:ovn-central --target net-node-01
|
||||
|
||||
# Install OVN central
|
||||
incus exec net-node-01:ovn-central -- apt-get update -qq
|
||||
incus exec net-node-01:ovn-central -- apt-get install -y ovn-central
|
||||
|
||||
# Configure listening on all interfaces
|
||||
incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641
|
||||
incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642
|
||||
|
||||
# Verify
|
||||
incus exec net-node-01:ovn-central -- ovn-nbctl get-connection # ptcp:6641
|
||||
incus exec net-node-01:ovn-central -- ovn-sbctl get-connection # ptcp:6642
|
||||
```
|
||||
|
||||
The container runs on `incusbr0` (node-local bridge at 10.0.0.x). Other nodes
|
||||
can't reach this IP directly. Use proxy devices to expose the ports on the
|
||||
host IP:
|
||||
|
||||
```bash
|
||||
# Forward NB and SB ports from host to container
|
||||
incus config device add net-node-01:ovn-central ovn-nb proxy \
|
||||
listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641
|
||||
incus config device add net-node-01:ovn-central ovn-sb proxy \
|
||||
listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642
|
||||
```
|
||||
|
||||
Now all nodes can reach the OVN databases via `tcp:192.168.1.209:6641` and
|
||||
`tcp:192.168.1.209:6642`.
|
||||
|
||||
### Step 2: Enable OVN on IncusOS nodes
|
||||
|
||||
**Critical:** The OVN client services on IncusOS are disabled by default.
|
||||
You MUST enable them on every node before configuring Incus.
|
||||
|
||||
Without this step, `incus config set network.ovn.northbound_connection` fails:
|
||||
```
|
||||
Error: failed to notify peer: Failed to connect to OVS:
|
||||
failed to connect to unix:/run/openvswitch/db.sock: no such file or directory
|
||||
```
|
||||
|
||||
Enable OVN via the IncusOS REST API on each node:
|
||||
|
||||
```bash
|
||||
# Enable OVN on each node (set database to SOUTHBOUND port 6642)
|
||||
incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{
|
||||
"config": {
|
||||
"database": "tcp:192.168.1.209:6642",
|
||||
"enabled": true,
|
||||
"tunnel_address": "192.168.1.209",
|
||||
"tunnel_protocol": "geneve"
|
||||
},
|
||||
"state": {}
|
||||
}'
|
||||
|
||||
incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{
|
||||
"config": {
|
||||
"database": "tcp:192.168.1.209:6642",
|
||||
"enabled": true,
|
||||
"tunnel_address": "192.168.1.150",
|
||||
"tunnel_protocol": "geneve"
|
||||
},
|
||||
"state": {}
|
||||
}'
|
||||
|
||||
# Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169)
|
||||
# with their respective tunnel_address values
|
||||
```
|
||||
|
||||
Key fields:
|
||||
- `database`: OVN **southbound** DB connection (port 6642, NOT 6641)
|
||||
- `tunnel_address`: this node's LAN IP for Geneve tunnel encapsulation
|
||||
- `tunnel_protocol`: `geneve` (default and recommended)
|
||||
- `enabled`: `true` to start `ovsdb-server`, `ovs-vswitchd`, and `ovn-controller`
|
||||
|
||||
Wait ~10 seconds for services to start, then verify:
|
||||
|
||||
```bash
|
||||
incus query net-node-01:/os/1.0/services/ovn
|
||||
# Should show enabled: true
|
||||
```
|
||||
|
||||
### Step 3: Configure Incus OVN connection
|
||||
|
||||
Tell Incus where the OVN **northbound** database is:
|
||||
|
||||
```bash
|
||||
incus config set net-node-01: network.ovn.northbound_connection \
|
||||
tcp:192.168.1.209:6641
|
||||
```
|
||||
|
||||
This is a cluster-wide setting that propagates to all members. It WILL FAIL
|
||||
if any member doesn't have OVS running (Step 2 not complete).
|
||||
|
||||
Verify all chassis are registered:
|
||||
|
||||
```bash
|
||||
$ incus exec net-node-01:ovn-central -- ovn-sbctl show
|
||||
Chassis "c1a92a50-..."
|
||||
hostname: net-node-01
|
||||
Encap geneve
|
||||
ip: "192.168.1.209"
|
||||
Chassis "f4fb8f29-..."
|
||||
hostname: net-node-02
|
||||
Encap geneve
|
||||
ip: "192.168.1.150"
|
||||
Chassis "88f8cd95-..."
|
||||
hostname: net-node-03
|
||||
Encap geneve
|
||||
ip: "192.168.1.13"
|
||||
Chassis "6be485be-..."
|
||||
hostname: net-node-04
|
||||
Encap geneve
|
||||
ip: "192.168.1.169"
|
||||
```
|
||||
|
||||
### Step 4: Assign ovn-chassis roles
|
||||
|
||||
Designate which nodes participate in OVN gateway HA:
|
||||
|
||||
```bash
|
||||
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||
incus cluster role add net-node-01:"$node" ovn-chassis
|
||||
done
|
||||
```
|
||||
|
||||
### Step 5: Create physical uplink network
|
||||
|
||||
OVN networks need an uplink for external/LAN connectivity:
|
||||
|
||||
```bash
|
||||
# Per-member definition (two-step cluster pattern)
|
||||
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||
incus network create net-node-01:UPLINK --type=physical \
|
||||
parent=ens18 --target "$node"
|
||||
done
|
||||
|
||||
# Finalize with external IP range and gateway
|
||||
incus network create net-node-01:UPLINK --type=physical \
|
||||
ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \
|
||||
ipv4.gateway=192.168.1.1/24 \
|
||||
dns.nameservers=192.168.1.1
|
||||
```
|
||||
|
||||
- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox)
|
||||
- `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses
|
||||
- `ipv4.gateway`: LAN gateway in CIDR format
|
||||
|
||||
### Step 6: Create OVN network
|
||||
|
||||
```bash
|
||||
incus network create net-node-01:ovn-net1 --type=ovn \
|
||||
network=UPLINK \
|
||||
ipv4.address=10.10.10.1/24 \
|
||||
ipv4.nat=true \
|
||||
ipv6.address=none
|
||||
```
|
||||
|
||||
Incus automatically creates:
|
||||
- **Logical router** with external IP from the uplink range (e.g., 192.168.1.230)
|
||||
- **Internal logical switch** (10.10.10.0/24)
|
||||
- **External logical switch** connected to the physical uplink
|
||||
- **SNAT rule**: 10.10.10.0/24 → 192.168.1.230
|
||||
|
||||
---
|
||||
|
||||
## Part 3: OVN Test Results
|
||||
|
||||
### Cross-node communication: PASS
|
||||
|
||||
The fundamental OVN benefit — instances on different nodes can communicate:
|
||||
|
||||
```bash
|
||||
# test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3)
|
||||
$ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3
|
||||
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms
|
||||
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms
|
||||
# 0% packet loss, ~0.1-0.8ms via Geneve tunnel
|
||||
```
|
||||
|
||||
Compare with bridge networking (Part 1) where cross-node ping FAILED.
|
||||
|
||||
### Full mesh connectivity: PASS
|
||||
|
||||
All 12 directional paths between 4 containers on 4 different nodes:
|
||||
|
||||
```
|
||||
test-c1 -> test-c2: PASS (0.672ms) test-c2 -> test-c1: PASS (0.289ms)
|
||||
test-c1 -> test-c3: PASS (0.525ms) test-c3 -> test-c1: PASS (0.335ms)
|
||||
test-c1 -> test-c4: PASS (0.709ms) test-c4 -> test-c1: PASS (0.412ms)
|
||||
test-c2 -> test-c3: PASS (0.539ms) test-c3 -> test-c2: PASS (0.344ms)
|
||||
test-c2 -> test-c4: PASS (0.730ms) test-c4 -> test-c2: PASS (0.366ms)
|
||||
test-c3 -> test-c4: PASS (0.704ms) test-c4 -> test-c3: PASS (0.356ms)
|
||||
```
|
||||
|
||||
### Internet access via NAT: PASS
|
||||
|
||||
```bash
|
||||
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
|
||||
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms
|
||||
# NAT through OVN router -> physical uplink -> LAN -> internet
|
||||
```
|
||||
|
||||
### LAN access via physical uplink: PASS
|
||||
|
||||
OVN instances can reach any host on the LAN:
|
||||
|
||||
```bash
|
||||
# LAN gateway
|
||||
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1
|
||||
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms
|
||||
|
||||
# Proxmox host
|
||||
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29
|
||||
64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms
|
||||
|
||||
# Cluster nodes
|
||||
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13
|
||||
64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 4: Network Isolation (Micro-Segmentation)
|
||||
|
||||
### Multiple OVN networks
|
||||
|
||||
Create a second OVN network with a different subnet:
|
||||
|
||||
```bash
|
||||
incus network create net-node-01:ovn-net2 --type=ovn \
|
||||
network=UPLINK \
|
||||
ipv4.address=10.10.20.1/24 \
|
||||
ipv4.nat=true \
|
||||
ipv6.address=none
|
||||
```
|
||||
|
||||
Each OVN network gets its own external IP from the uplink range
|
||||
(ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231).
|
||||
|
||||
### Isolation test results
|
||||
|
||||
```
|
||||
Same network, cross-node:
|
||||
test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms)
|
||||
|
||||
Different network, same node:
|
||||
test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss)
|
||||
|
||||
Different network, different node:
|
||||
test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss)
|
||||
|
||||
Both networks have independent internet access:
|
||||
test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms)
|
||||
```
|
||||
|
||||
**Each OVN network is fully isolated.** Instances on different networks cannot
|
||||
communicate, even when running on the same physical node.
|
||||
|
||||
---
|
||||
|
||||
## Part 5: Network ACLs
|
||||
|
||||
### Basic ACL: block ICMP
|
||||
|
||||
```bash
|
||||
# Create ACL
|
||||
incus network acl create net-node-01:block-ping
|
||||
incus network acl rule add net-node-01:block-ping ingress \
|
||||
action=drop protocol=icmp4
|
||||
|
||||
# Apply to an instance's NIC
|
||||
incus config device set net-node-01:test-c1 eth0 security.acls=block-ping
|
||||
```
|
||||
|
||||
**Result:** All inbound ICMP to test-c1 is dropped, including echo replies.
|
||||
This means ping FROM test-c1 also fails (reply packets are dropped by the
|
||||
ingress rule).
|
||||
|
||||
### Targeted ACL: block specific source
|
||||
|
||||
```bash
|
||||
# Create ACL with default allow + specific drop
|
||||
incus network acl create net-node-01:allow-except-c2
|
||||
incus network acl rule add net-node-01:allow-except-c2 ingress action=allow
|
||||
incus network acl rule add net-node-01:allow-except-c2 ingress \
|
||||
action=drop source=10.10.10.3/32
|
||||
|
||||
# Apply to test-c1
|
||||
incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2
|
||||
```
|
||||
|
||||
**Results:**
|
||||
```
|
||||
test-c3 (allowed) -> test-c1: PASS (0.514ms)
|
||||
test-c4 (allowed) -> test-c1: PASS (0.786ms)
|
||||
test-c2 (blocked) -> test-c1: BLOCKED (100% loss)
|
||||
```
|
||||
|
||||
### Remove ACL
|
||||
|
||||
```bash
|
||||
incus config device set net-node-01:test-c1 eth0 security.acls=""
|
||||
incus network acl delete net-node-01:allow-except-c2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 6: Network Peering
|
||||
|
||||
Connect two previously isolated OVN networks:
|
||||
|
||||
```bash
|
||||
# Create mutual peering (both directions required)
|
||||
incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2
|
||||
incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1
|
||||
```
|
||||
|
||||
### Before peering
|
||||
|
||||
```bash
|
||||
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss)
|
||||
```
|
||||
|
||||
### After peering
|
||||
|
||||
```bash
|
||||
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms)
|
||||
# test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms)
|
||||
```
|
||||
|
||||
Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers.
|
||||
|
||||
### Remove peering
|
||||
|
||||
```bash
|
||||
incus network peer delete net-node-01:ovn-net1 peer-to-net2
|
||||
incus network peer delete net-node-01:ovn-net2 peer-to-net1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 7: Load Balancers
|
||||
|
||||
OVN provides built-in L4 load balancing with LAN-routable virtual IPs.
|
||||
|
||||
### Create load balancer
|
||||
|
||||
```bash
|
||||
# Create LB with a listen address from the uplink range
|
||||
incus network load-balancer create net-node-01:ovn-net1 192.168.1.232
|
||||
|
||||
# Add backends
|
||||
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
|
||||
backend-c1 10.10.10.2 80
|
||||
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
|
||||
backend-c3 10.10.10.4 80
|
||||
|
||||
# Map frontend port to backends
|
||||
incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \
|
||||
tcp 80 backend-c1,backend-c3
|
||||
```
|
||||
|
||||
### Test results
|
||||
|
||||
```bash
|
||||
# 6 requests to the load balancer VIP
|
||||
Request 1: Hello from test-c1 (node-01)
|
||||
Request 2: Hello from test-c1 (node-01)
|
||||
Request 3: Hello from test-c1 (node-01)
|
||||
Request 4: Hello from test-c3 (node-03)
|
||||
Request 5: Hello from test-c3 (node-03)
|
||||
Request 6: Hello from test-c1 (node-01)
|
||||
```
|
||||
|
||||
Traffic is distributed across both backends. OVN uses connection-based
|
||||
hashing (not strict round-robin).
|
||||
|
||||
### Clean up
|
||||
|
||||
```bash
|
||||
incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 8: Network Forwards (Port Forwarding)
|
||||
|
||||
Expose internal services on LAN-routable IPs without load balancing.
|
||||
|
||||
### Create forward
|
||||
|
||||
```bash
|
||||
# Allocate a LAN IP for forwarding
|
||||
incus network forward create net-node-01:ovn-net1 192.168.1.233
|
||||
|
||||
# Forward external port 8080 to internal instance port 80
|
||||
incus network forward port add net-node-01:ovn-net1 192.168.1.233 \
|
||||
tcp 8080 10.10.10.2 80
|
||||
```
|
||||
|
||||
### Test results
|
||||
|
||||
```bash
|
||||
# From LAN (dev machine)
|
||||
$ curl http://192.168.1.233:8080/
|
||||
Hello from test-c1 (node-01)
|
||||
|
||||
# From another OVN network
|
||||
$ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/
|
||||
Hello from test-c1 (node-01)
|
||||
```
|
||||
|
||||
### Clean up
|
||||
|
||||
```bash
|
||||
incus network forward delete net-node-01:ovn-net1 192.168.1.233
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 9: DNS Resolution
|
||||
|
||||
OVN networks provide automatic DNS for instances within the same network.
|
||||
|
||||
### Configuration
|
||||
|
||||
Instances get `search incus` in `/etc/resolv.conf` via DHCP. DNS queries
|
||||
for instance hostnames are resolved by the OVN DHCP server.
|
||||
|
||||
### Test results
|
||||
|
||||
```bash
|
||||
# Same network: hostname resolution works
|
||||
$ incus exec net-node-01:test-c2 -- python3 -c "
|
||||
import socket
|
||||
print(socket.gethostbyname('test-c1')) # 10.10.10.2
|
||||
print(socket.gethostbyname('test-c3')) # 10.10.10.4
|
||||
print(socket.gethostbyname('test-c4')) # 10.10.10.5
|
||||
"
|
||||
|
||||
# Cross-network: resolution fails (expected)
|
||||
$ incus exec net-node-01:test-c5 -- python3 -c "
|
||||
import socket
|
||||
socket.gethostbyname('test-c1') # Raises: Name or service not known
|
||||
"
|
||||
|
||||
# Ping by hostname works within the same OVN network
|
||||
$ incus exec net-node-01:test-c2 -- ping -c 1 test-c1
|
||||
PING test-c1 (10.10.10.2) ...
|
||||
64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms
|
||||
```
|
||||
|
||||
DNS is per-network — instances on `ovn-net1` can resolve each other but
|
||||
NOT instances on `ovn-net2`.
|
||||
|
||||
---
|
||||
|
||||
## Part 10: Final Network State
|
||||
|
||||
After all setup:
|
||||
|
||||
```
|
||||
+----------+----------+---------+---------------+-------------+---------+
|
||||
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | STATE |
|
||||
+----------+----------+---------+---------------+-------------+---------+
|
||||
| UPLINK | physical | YES | | | CREATED |
|
||||
| incusbr0 | bridge | YES | 10.0.0.1/24 | | CREATED |
|
||||
| ovn-net1 | ovn | YES | 10.10.10.1/24 | | CREATED |
|
||||
| ovn-net2 | ovn | YES | 10.10.20.1/24 | | CREATED |
|
||||
+----------+----------+---------+---------------+-------------+---------+
|
||||
```
|
||||
|
||||
### Network summary
|
||||
|
||||
| Network | Type | Subnet | External IP | Purpose |
|
||||
|----------|----------|----------------|----------------|------------------------|
|
||||
| incusbr0 | bridge | 10.0.0.1/24 | NAT via ens18 | Default, node-local |
|
||||
| UPLINK | physical | - | LAN bridge | OVN external gateway |
|
||||
| ovn-net1 | ovn | 10.10.10.1/24 | 192.168.1.230 | Cross-node workloads |
|
||||
| ovn-net2 | ovn | 10.10.20.1/24 | 192.168.1.231 | Isolated workloads |
|
||||
|
||||
---
|
||||
|
||||
## Deployment Notes
|
||||
|
||||
### OC-provisioned ISO gotchas
|
||||
|
||||
**Stale ISO on Proxmox storage**: When using `incusos-proxmox --iso`, the
|
||||
script checks if an ISO with the same filename already exists on Proxmox
|
||||
storage. If found, it skips the upload. This means a stale ISO from a
|
||||
previous OC server (with a different IP baked in) will be reused, causing
|
||||
all nodes to fail with "no route to host" errors when trying to reach
|
||||
the old OC server for updates.
|
||||
|
||||
**Fix**: Always delete the old ISO from Proxmox before deploying with a
|
||||
new OC server:
|
||||
```bash
|
||||
# Check for stale ISOs
|
||||
ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso"
|
||||
# Delete if present
|
||||
ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso"
|
||||
```
|
||||
|
||||
The `--cleanup-all --deep` command deletes ISOs matching `IncusOS_*.iso`
|
||||
(underscore pattern) but NOT `IncusOS-oc.iso` (custom hyphenated name).
|
||||
|
||||
**Symptom on console** (verified via screenshots):
|
||||
```
|
||||
ERROR Failed to check for Secure Boot key updates
|
||||
err=http request timed out after five seconds:
|
||||
Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1":
|
||||
dial tcp OLD_IP:8443: connect: no route to host
|
||||
provider=operations-center
|
||||
```
|
||||
|
||||
### OC update prerequisites for clustering
|
||||
|
||||
OC v0.3.0 requires all nodes to show `needs update: false` before
|
||||
`provisioning cluster add` will work. Even when nodes are running the
|
||||
latest IncusOS version, OC may report "update pending" indefinitely.
|
||||
|
||||
**Workaround**: Form the cluster manually using `incus cluster enable`
|
||||
and `incus cluster join` instead of `operations-center provisioning
|
||||
cluster add`.
|
||||
|
||||
### Cluster resource creation with apply_defaults: false
|
||||
|
||||
When nodes are deployed with `apply_defaults: false` (recommended for
|
||||
OC-managed nodes and cluster joining nodes), no storage pool, network,
|
||||
or profile devices are created. After cluster formation, create them
|
||||
manually using the two-step pattern:
|
||||
|
||||
1. Define per-member config with `--target <member>`
|
||||
2. Finalize cluster-wide with the creation command (no `--target`)
|
||||
|
||||
This is required because cluster-specific settings like `source` and
|
||||
`zfs.pool_name` must be defined per-member.
|
||||
|
||||
### IncusOS OVN service activation
|
||||
|
||||
The OVN client services (`ovsdb-server`, `ovs-vswitchd`, `ovn-controller`)
|
||||
on IncusOS are **disabled by default**. They must be explicitly enabled
|
||||
via the IncusOS REST API (`/os/1.0/services/ovn`) before Incus can
|
||||
configure OVN networking.
|
||||
|
||||
Setting `network.ovn.northbound_connection` before enabling OVN services
|
||||
on all nodes will fail with:
|
||||
```
|
||||
Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock
|
||||
```
|
||||
|
||||
Enable OVN on each node with the correct southbound database address and
|
||||
tunnel IP **before** setting the Incus northbound connection.
|
||||
|
||||
### OVN control plane in a container
|
||||
|
||||
When running the OVN control plane as a container on `incusbr0` (bridge
|
||||
network), the container is only directly reachable from the node it runs on.
|
||||
Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the
|
||||
host's LAN IP so all cluster nodes can reach it.
|
||||
Loading…
Reference in New Issue