Add VLAN 69 support, OVN networking guide, and lab-networking config
VLAN tagging: incusos-proxmox now reads optional `vlan` key from proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec in both SSH and API creation paths. Tested manually on VMID 850 — VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is Proxmox-level only; IncusOS and Incus instances are unaware of it. Networking guide: comprehensive tutorial covering bridge networking, OVN overlay setup on IncusOS (services disabled by default, must enable via REST API), cross-node connectivity, isolation, ACLs, peering, load balancers, forwards, and DNS. All scenarios tested on a 4-node OC-managed cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
2e2447de06
commit
9e3201393e
56
CLAUDE.md
56
CLAUDE.md
|
|
@ -28,6 +28,7 @@ incu-contrib/
|
||||||
└── notes/ # Research notes and reference material
|
└── notes/ # Research notes and reference material
|
||||||
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
||||||
├── operations-center-guide.md # Operations Center provisioning & management
|
├── operations-center-guide.md # Operations Center provisioning & management
|
||||||
|
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
||||||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
||||||
└── utm-support.md # UTM support design document (future)
|
└── utm-support.md # UTM support design document (future)
|
||||||
```
|
```
|
||||||
|
|
@ -117,6 +118,13 @@ incu-contrib/
|
||||||
- `balloon=0` -- IncusOS manages memory internally
|
- `balloon=0` -- IncusOS manages memory internally
|
||||||
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
||||||
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
||||||
|
- **VLAN tagging**: the `vlan` config key in `proxmox.yaml` adds a VLAN tag
|
||||||
|
to the VM's NIC (`net0: virtio,bridge=vmbr0,tag=69`). This places the VM
|
||||||
|
on the tagged VLAN instead of the native/untagged network. The current lab
|
||||||
|
uses VLAN 69 (Homelab VLAN, subnet 192.168.100.0/22). Without a VLAN tag,
|
||||||
|
VMs land on the native LAN (192.168.1.0/24). The VLAN tag is set at the
|
||||||
|
Proxmox VM level only — IncusOS and Incus instances inside the VM are
|
||||||
|
unaware of it. The setting is optional: omit `vlan:` for untagged access.
|
||||||
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
||||||
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
||||||
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
||||||
|
|
@ -744,6 +752,54 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||||
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
- **No hardcoded package managers**: say "install the Incus client" with a link,
|
||||||
not "sudo apt install incus".
|
not "sudo apt install incus".
|
||||||
|
|
||||||
|
### Incus networking (OVN)
|
||||||
|
|
||||||
|
- **Bridge networks are node-local**: each cluster member has its own
|
||||||
|
independent bridge. Instances on the same bridge (same node) can
|
||||||
|
communicate; cross-node instances CANNOT. Each bridge has the same subnet
|
||||||
|
(e.g., 10.0.0.1/24) but they are separate L2 domains.
|
||||||
|
- **OVN provides cross-node L2 overlay**: uses Geneve tunnels between nodes.
|
||||||
|
Sub-millisecond latency across nodes (~0.1-0.8ms). Requires control plane
|
||||||
|
+ client services + physical uplink network.
|
||||||
|
- **IncusOS OVN services are disabled by default**: must be enabled via the
|
||||||
|
IncusOS REST API (`/os/1.0/services/ovn`) on EVERY node before configuring
|
||||||
|
Incus OVN settings. Without this, `incus config set network.ovn.northbound_connection`
|
||||||
|
fails with `db.sock not found`.
|
||||||
|
- **OVN service enable API call**:
|
||||||
|
```bash
|
||||||
|
incus query <remote>:/os/1.0/services/ovn --request PUT --data '{
|
||||||
|
"config": {
|
||||||
|
"database": "tcp:<SB_HOST>:6642",
|
||||||
|
"enabled": true,
|
||||||
|
"tunnel_address": "<THIS_NODE_LAN_IP>",
|
||||||
|
"tunnel_protocol": "geneve"
|
||||||
|
},
|
||||||
|
"state": {}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
`database` is the **southbound** DB (port 6642), NOT northbound (6641).
|
||||||
|
- **OVN control plane as container**: deploy `ovn-central` package in a
|
||||||
|
Debian container on the cluster. Use proxy devices to expose NB (6641)
|
||||||
|
and SB (6642) ports on the host's LAN IP so all nodes can reach it.
|
||||||
|
- **Setup sequence** (order matters):
|
||||||
|
1. Deploy OVN control plane container
|
||||||
|
2. Enable OVN services on ALL IncusOS nodes
|
||||||
|
3. `incus config set network.ovn.northbound_connection tcp:<host>:6641`
|
||||||
|
4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes)
|
||||||
|
5. Create physical uplink network (two-step cluster pattern)
|
||||||
|
6. Create OVN network with `--type=ovn network=UPLINK`
|
||||||
|
- **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC),
|
||||||
|
`ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses,
|
||||||
|
`ipv4.gateway` is the LAN gateway in CIDR format.
|
||||||
|
- **OVN network isolation**: multiple OVN networks are fully isolated.
|
||||||
|
Instances on different networks cannot communicate, even on the same node.
|
||||||
|
Network peering (`incus network peer create`) enables cross-network routing.
|
||||||
|
- **OVN features tested**: cross-node connectivity, network isolation,
|
||||||
|
ACLs (per-source blocking), network peering, L4 load balancers (connection-
|
||||||
|
based hashing, not round-robin), network forwards (port forwarding to LAN
|
||||||
|
IPs), DNS resolution (per-network, hostname.incus domain).
|
||||||
|
- See `notes/networking-guide.md` for full tutorial with test results.
|
||||||
|
|
||||||
### Migration into Incus
|
### Migration into Incus
|
||||||
|
|
||||||
- **`incus-migrate`**: official tool for importing disk images, running
|
- **`incus-migrate`**: official tool for importing disk images, running
|
||||||
|
|
|
||||||
|
|
@ -389,6 +389,7 @@ node: pve
|
||||||
storage: local-lvm
|
storage: local-lvm
|
||||||
iso_storage: local
|
iso_storage: local
|
||||||
bridge: vmbr0
|
bridge: vmbr0
|
||||||
|
vlan: 69
|
||||||
pool: IncusLab
|
pool: IncusLab
|
||||||
ssh_user: root
|
ssh_user: root
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
# lab-networking.yaml - 4-node cluster for networking exploration
|
||||||
|
#
|
||||||
|
# Deploys 4 Incus nodes using an OC-provisioned ISO for a networking
|
||||||
|
# tutorial covering bridge, OVN overlay, ACLs, load balancers, and
|
||||||
|
# LAN integration.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# # First: deploy OC server with lab-oc-deploy.yaml
|
||||||
|
# # Then: create OC provisioning token + download ISO
|
||||||
|
# # Then deploy with --iso:
|
||||||
|
# incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run examples/lab-networking.yaml
|
||||||
|
# incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-networking.yaml
|
||||||
|
# incusos-proxmox --status examples/lab-networking.yaml
|
||||||
|
# incusos-proxmox --cleanup --deep examples/lab-networking.yaml
|
||||||
|
#
|
||||||
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
|
#
|
||||||
|
# Resource budget: 4 nodes x 8 GiB = 32 GiB RAM
|
||||||
|
# Plus OC server (4 GiB) = 36 GiB total
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
cores: 4
|
||||||
|
memory: 8192
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 400
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: net-node-01
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
|
||||||
|
- name: net-node-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
|
||||||
|
- name: net-node-03
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
|
||||||
|
- name: net-node-04
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
|
@ -166,6 +166,7 @@ ${BOLD}PROXMOX CONNECTION CONFIG${RESET}
|
||||||
storage: local-lvm # VM disk storage (default: local-lvm)
|
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||||
iso_storage: local # ISO storage (default: local)
|
iso_storage: local # ISO storage (default: local)
|
||||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||||
|
vlan: 69 # VLAN tag for VM NIC (optional)
|
||||||
pool: IncusLab # Resource pool (optional)
|
pool: IncusLab # Resource pool (optional)
|
||||||
ssh_user: root # SSH user (default: root)
|
ssh_user: root # SSH user (default: root)
|
||||||
|
|
||||||
|
|
@ -1294,6 +1295,7 @@ PVE_NODE=""
|
||||||
PVE_STORAGE=""
|
PVE_STORAGE=""
|
||||||
PVE_ISO_STORAGE=""
|
PVE_ISO_STORAGE=""
|
||||||
PVE_BRIDGE=""
|
PVE_BRIDGE=""
|
||||||
|
PVE_VLAN=""
|
||||||
PVE_METHOD=""
|
PVE_METHOD=""
|
||||||
PVE_SSH_USER=""
|
PVE_SSH_USER=""
|
||||||
PVE_API_TOKEN_ID=""
|
PVE_API_TOKEN_ID=""
|
||||||
|
|
@ -1393,6 +1395,7 @@ json.dump(result, sys.stdout, indent=2)
|
||||||
val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||||
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||||
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||||
|
val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
|
||||||
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||||
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||||
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||||
|
|
@ -1415,6 +1418,7 @@ load_config() {
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||||
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||||
|
|
@ -1852,7 +1856,9 @@ pve_create_vm() {
|
||||||
cmd+=" --scsi0 ${PVE_STORAGE}:${disk_gib},iothread=1"
|
cmd+=" --scsi0 ${PVE_STORAGE}:${disk_gib},iothread=1"
|
||||||
cmd+=" --ide2 ${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom"
|
cmd+=" --ide2 ${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom"
|
||||||
cmd+=" --ide3 ${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom"
|
cmd+=" --ide3 ${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom"
|
||||||
cmd+=" --net0 virtio,bridge=${PVE_BRIDGE}"
|
local net0_spec="virtio,bridge=${PVE_BRIDGE}"
|
||||||
|
[[ -n "$PVE_VLAN" ]] && net0_spec+=",tag=${PVE_VLAN}"
|
||||||
|
cmd+=" --net0 ${net0_spec}"
|
||||||
cmd+=" --boot order=ide2\\;scsi0"
|
cmd+=" --boot order=ide2\\;scsi0"
|
||||||
cmd+=" --agent 1"
|
cmd+=" --agent 1"
|
||||||
if [[ -n "$PVE_POOL" ]]; then
|
if [[ -n "$PVE_POOL" ]]; then
|
||||||
|
|
@ -1892,10 +1898,14 @@ payload = {
|
||||||
'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1',
|
'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1',
|
||||||
'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom',
|
'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom',
|
||||||
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom',
|
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom',
|
||||||
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
|
||||||
'boot': 'order=ide2;scsi0',
|
'boot': 'order=ide2;scsi0',
|
||||||
'agent': 1,
|
'agent': 1,
|
||||||
}
|
}
|
||||||
|
net0 = 'virtio,bridge=${PVE_BRIDGE}'
|
||||||
|
vlan = '${PVE_VLAN}'
|
||||||
|
if vlan:
|
||||||
|
net0 += ',tag=' + vlan
|
||||||
|
payload['net0'] = net0
|
||||||
pool = '${PVE_POOL}'
|
pool = '${PVE_POOL}'
|
||||||
if pool:
|
if pool:
|
||||||
payload['pool'] = pool
|
payload['pool'] = pool
|
||||||
|
|
@ -2866,7 +2876,11 @@ print_plan() {
|
||||||
echo -e " Proxmox host: ${CYAN}${PVE_HOST}${RESET} (${PVE_METHOD})"
|
echo -e " Proxmox host: ${CYAN}${PVE_HOST}${RESET} (${PVE_METHOD})"
|
||||||
echo -e " Node: ${PVE_NODE}"
|
echo -e " Node: ${PVE_NODE}"
|
||||||
echo -e " Storage: ${PVE_STORAGE} (disk), ${PVE_ISO_STORAGE} (iso)"
|
echo -e " Storage: ${PVE_STORAGE} (disk), ${PVE_ISO_STORAGE} (iso)"
|
||||||
echo -e " Bridge: ${PVE_BRIDGE}"
|
if [[ -n "$PVE_VLAN" ]]; then
|
||||||
|
echo -e " Bridge: ${PVE_BRIDGE} (VLAN ${PVE_VLAN})"
|
||||||
|
else
|
||||||
|
echo -e " Bridge: ${PVE_BRIDGE}"
|
||||||
|
fi
|
||||||
if [[ -n "$PVE_POOL" ]]; then
|
if [[ -n "$PVE_POOL" ]]; then
|
||||||
echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}"
|
echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}"
|
||||||
fi
|
fi
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,799 @@
|
||||||
|
# Incus Networking Guide — Bridge, OVN, and LAN Integration
|
||||||
|
|
||||||
|
A hands-on tutorial covering Incus networking from basic bridge networking
|
||||||
|
to OVN overlay networks, tested on a 4-node IncusOS cluster managed by
|
||||||
|
Operations Center on Proxmox VE.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- 4-node IncusOS cluster (this guide uses `net-node-01` through `net-node-04`)
|
||||||
|
- Operations Center server (for OC-managed deployment)
|
||||||
|
- Proxmox VE host with `incusos-proxmox` deployment tool
|
||||||
|
- `incus` client configured with remotes for all cluster nodes
|
||||||
|
|
||||||
|
### Lab configuration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# incusos/examples/lab-networking.yaml
|
||||||
|
defaults:
|
||||||
|
cores: 4
|
||||||
|
memory: 8192
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 400
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: net-node-01
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
- name: net-node-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
- name: net-node-03
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
- name: net-node-04
|
||||||
|
app: incus
|
||||||
|
apply_defaults: false
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resource budget:** OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total.
|
||||||
|
|
||||||
|
### Cluster formation
|
||||||
|
|
||||||
|
Nodes were deployed with `apply_defaults: false`, so no default storage pool,
|
||||||
|
network, or profile devices exist after installation. After forming the cluster
|
||||||
|
manually (OC's `cluster add` requires "needs update: false" which was blocked
|
||||||
|
by a v0.3.0 tracking issue), we created these resources:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Storage pool (per-member source, then cluster-wide finalize)
|
||||||
|
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||||
|
incus storage create net-node-01:local zfs \
|
||||||
|
source=local/incus zfs.pool_name=local/incus --target "$node"
|
||||||
|
done
|
||||||
|
incus storage create net-node-01:local zfs
|
||||||
|
|
||||||
|
# Bridge network (per-member pending, then finalize with config)
|
||||||
|
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||||
|
incus network create net-node-01:incusbr0 --type bridge --target "$node"
|
||||||
|
done
|
||||||
|
incus network create net-node-01:incusbr0 --type bridge \
|
||||||
|
ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none
|
||||||
|
|
||||||
|
# Default profile
|
||||||
|
incus profile device add net-node-01:default root disk path=/ pool=local
|
||||||
|
incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 1: Bridge Networking (Default)
|
||||||
|
|
||||||
|
### What is bridge networking?
|
||||||
|
|
||||||
|
When Incus creates a bridge network (type `bridge`), it creates a Linux bridge
|
||||||
|
on each cluster node. Each bridge is **independent** — there is no connectivity
|
||||||
|
between bridges on different nodes. This is the simplest networking mode and
|
||||||
|
works without any additional infrastructure.
|
||||||
|
|
||||||
|
### Bridge topology
|
||||||
|
|
||||||
|
```
|
||||||
|
Node 1 (net-node-01) Node 2 (net-node-02)
|
||||||
|
┌─────────────────────┐ ┌─────────────────────┐
|
||||||
|
│ incusbr0 │ │ incusbr0 │
|
||||||
|
│ 10.0.0.1/24 │ │ 10.0.0.1/24 │
|
||||||
|
│ ┌───┐ ┌───┐ │ │ ┌───┐ │
|
||||||
|
│ │c1 │ │c2 │ │ │ │c3 │ │
|
||||||
|
│ │.202│ │.40│ │ │ │.52│ │
|
||||||
|
│ └───┘ └───┘ │ │ └───┘ │
|
||||||
|
│ │ │ │ │ │ │
|
||||||
|
│ ──┴──────┴── │ │ ──┴── │
|
||||||
|
│ (bridge) │ │ (bridge) │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ NAT ──> ens18 │ │ NAT ──> ens18 │
|
||||||
|
│ 192.168.1.209 │ │ 192.168.1.150 │
|
||||||
|
└─────────────────────┘ └─────────────────────┘
|
||||||
|
│ │
|
||||||
|
────┴──────────────────────────────────┴────
|
||||||
|
LAN (192.168.1.0/24)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24).
|
||||||
|
The bridges are NOT connected to each other.
|
||||||
|
|
||||||
|
### Test results
|
||||||
|
|
||||||
|
#### Same-node communication: PASS
|
||||||
|
|
||||||
|
Containers on the same node share the same bridge (same L2 domain):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01
|
||||||
|
$ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40
|
||||||
|
64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms
|
||||||
|
# 0% packet loss, ~0.03ms latency
|
||||||
|
```
|
||||||
|
|
||||||
|
#### NAT to internet: PASS
|
||||||
|
|
||||||
|
The bridge has `ipv4.nat=true`, so containers can reach the internet:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
|
||||||
|
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms
|
||||||
|
# 0% packet loss, ~10ms to Cloudflare DNS
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Cross-node communication: FAIL
|
||||||
|
|
||||||
|
Containers on different nodes CANNOT reach each other:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52)
|
||||||
|
$ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52
|
||||||
|
From 10.0.0.202 icmp_seq=1 Destination Host Unreachable
|
||||||
|
# 100% packet loss
|
||||||
|
```
|
||||||
|
|
||||||
|
### When to use bridge networking
|
||||||
|
|
||||||
|
- Single-node setups (development, testing)
|
||||||
|
- Workloads that don't need cross-node communication
|
||||||
|
- Simple NAT-to-internet access
|
||||||
|
- As an uplink for OVN networks (see Part 2)
|
||||||
|
|
||||||
|
### Bridge network configuration options
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# View current config
|
||||||
|
incus network show net-node-01:incusbr0
|
||||||
|
|
||||||
|
# Key settings
|
||||||
|
ipv4.address # Bridge IP and subnet (e.g., 10.0.0.1/24)
|
||||||
|
ipv4.nat # Enable NAT for internet access (true/false)
|
||||||
|
ipv4.dhcp # Enable DHCP server (true/false, default true)
|
||||||
|
ipv4.dhcp.ranges # DHCP range (e.g., 10.0.0.100-10.0.0.200)
|
||||||
|
dns.domain # DNS domain for instances (e.g., incus)
|
||||||
|
dns.mode # DNS mode: managed, dynamic, or none
|
||||||
|
ipv6.address # IPv6 address (set to "none" to disable)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 2: OVN Overlay Networking
|
||||||
|
|
||||||
|
### Why OVN?
|
||||||
|
|
||||||
|
Bridge networking is node-local. To enable cross-node communication without
|
||||||
|
routing hacks, you need an overlay network. OVN (Open Virtual Network)
|
||||||
|
provides:
|
||||||
|
|
||||||
|
- **Cross-node L2 connectivity** via Geneve tunnels
|
||||||
|
- **Network isolation** (multiple independent virtual networks)
|
||||||
|
- **Distributed routing** (no single point of failure)
|
||||||
|
- **Network ACLs** (firewall rules at the network level)
|
||||||
|
- **Load balancers** (built-in L4 load balancing)
|
||||||
|
- **Network forwards** (port forwarding from external IPs)
|
||||||
|
- **DNS** (automatic hostname resolution between instances)
|
||||||
|
|
||||||
|
### OVN architecture on IncusOS
|
||||||
|
|
||||||
|
IncusOS includes OVN **client** components only:
|
||||||
|
- `ovn-controller` (data plane)
|
||||||
|
- `ovs-vswitchd` (Open vSwitch)
|
||||||
|
|
||||||
|
The OVN **control plane** must run separately:
|
||||||
|
- `ovn-northd` (translates logical to physical flows)
|
||||||
|
- `ovsdb-server` (northbound + southbound databases)
|
||||||
|
|
||||||
|
Options for deploying the control plane:
|
||||||
|
1. As a container on the Incus cluster itself (used in this guide)
|
||||||
|
2. On an external host
|
||||||
|
|
||||||
|
### OVN topology
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────┐
|
||||||
|
│ OVN Control Plane │
|
||||||
|
│ (container: ovn-central) │
|
||||||
|
│ ovn-northd + ovsdb-server │
|
||||||
|
│ NB: tcp:192.168.1.209:6641 │
|
||||||
|
│ SB: tcp:192.168.1.209:6642 │
|
||||||
|
└──────────┬───────────────────────┘
|
||||||
|
│ (proxy devices)
|
||||||
|
┌─────────────────────────┼─────────────────────────┐
|
||||||
|
│ │ │
|
||||||
|
Node 1 (net-node-01) Node 2 (net-node-02) Node 3 (net-node-03)
|
||||||
|
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
|
||||||
|
│ ovn-controller │ │ ovn-controller │ │ ovn-controller │
|
||||||
|
│ ovs-vswitchd │ │ ovs-vswitchd │ │ ovs-vswitchd │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ ┌───┐ ┌───┐ │ │ ┌───┐ │ │ ┌───┐ │
|
||||||
|
│ │c1 │ │c2 │ │ │ │c3 │ │ │ │c4 │ │
|
||||||
|
│ │.2 │ │.3 │ │ │ │.4 │ │ │ │.5 │ │
|
||||||
|
│ └─┬─┘ └─┬─┘ │ │ └─┬─┘ │ │ └─┬─┘ │
|
||||||
|
│ └──┬───┘ │ │ │ │ │ │ │
|
||||||
|
│ OVN logical │ │ OVN logical │ │ OVN logical │
|
||||||
|
│ switch │ │ switch │ │ switch │
|
||||||
|
│ 10.10.10.0/24 │ │ 10.10.10.0/24 │ │ 10.10.10.0/24 │
|
||||||
|
│ │ │ │ │ │ │ │ │
|
||||||
|
│ Geneve tunnel ──┼──┼── Geneve tunnel ──┼──┼── Geneve tunnel │
|
||||||
|
│ 192.168.1.209 │ │ 192.168.1.150 │ │ 192.168.1.13 │
|
||||||
|
└───────────────────┘ └───────────────────┘ └───────────────────┘
|
||||||
|
│ │ │
|
||||||
|
────┴──────────────────────┴──────────────────────┴────
|
||||||
|
LAN (192.168.1.0/24)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key difference from bridge:** All instances share a single logical switch
|
||||||
|
connected via Geneve tunnels. Cross-node traffic is encapsulated and sent
|
||||||
|
over the LAN between nodes.
|
||||||
|
|
||||||
|
### Step 1: Deploy OVN control plane
|
||||||
|
|
||||||
|
Launch a container to run the OVN central services:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Launch OVN central container
|
||||||
|
incus launch images:debian/12 net-node-01:ovn-central --target net-node-01
|
||||||
|
|
||||||
|
# Install OVN central
|
||||||
|
incus exec net-node-01:ovn-central -- apt-get update -qq
|
||||||
|
incus exec net-node-01:ovn-central -- apt-get install -y ovn-central
|
||||||
|
|
||||||
|
# Configure listening on all interfaces
|
||||||
|
incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641
|
||||||
|
incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
incus exec net-node-01:ovn-central -- ovn-nbctl get-connection # ptcp:6641
|
||||||
|
incus exec net-node-01:ovn-central -- ovn-sbctl get-connection # ptcp:6642
|
||||||
|
```
|
||||||
|
|
||||||
|
The container runs on `incusbr0` (node-local bridge at 10.0.0.x). Other nodes
|
||||||
|
can't reach this IP directly. Use proxy devices to expose the ports on the
|
||||||
|
host IP:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Forward NB and SB ports from host to container
|
||||||
|
incus config device add net-node-01:ovn-central ovn-nb proxy \
|
||||||
|
listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641
|
||||||
|
incus config device add net-node-01:ovn-central ovn-sb proxy \
|
||||||
|
listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642
|
||||||
|
```
|
||||||
|
|
||||||
|
Now all nodes can reach the OVN databases via `tcp:192.168.1.209:6641` and
|
||||||
|
`tcp:192.168.1.209:6642`.
|
||||||
|
|
||||||
|
### Step 2: Enable OVN on IncusOS nodes
|
||||||
|
|
||||||
|
**Critical:** The OVN client services on IncusOS are disabled by default.
|
||||||
|
You MUST enable them on every node before configuring Incus.
|
||||||
|
|
||||||
|
Without this step, `incus config set network.ovn.northbound_connection` fails:
|
||||||
|
```
|
||||||
|
Error: failed to notify peer: Failed to connect to OVS:
|
||||||
|
failed to connect to unix:/run/openvswitch/db.sock: no such file or directory
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable OVN via the IncusOS REST API on each node:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Enable OVN on each node (set database to SOUTHBOUND port 6642)
|
||||||
|
incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{
|
||||||
|
"config": {
|
||||||
|
"database": "tcp:192.168.1.209:6642",
|
||||||
|
"enabled": true,
|
||||||
|
"tunnel_address": "192.168.1.209",
|
||||||
|
"tunnel_protocol": "geneve"
|
||||||
|
},
|
||||||
|
"state": {}
|
||||||
|
}'
|
||||||
|
|
||||||
|
incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{
|
||||||
|
"config": {
|
||||||
|
"database": "tcp:192.168.1.209:6642",
|
||||||
|
"enabled": true,
|
||||||
|
"tunnel_address": "192.168.1.150",
|
||||||
|
"tunnel_protocol": "geneve"
|
||||||
|
},
|
||||||
|
"state": {}
|
||||||
|
}'
|
||||||
|
|
||||||
|
# Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169)
|
||||||
|
# with their respective tunnel_address values
|
||||||
|
```
|
||||||
|
|
||||||
|
Key fields:
|
||||||
|
- `database`: OVN **southbound** DB connection (port 6642, NOT 6641)
|
||||||
|
- `tunnel_address`: this node's LAN IP for Geneve tunnel encapsulation
|
||||||
|
- `tunnel_protocol`: `geneve` (default and recommended)
|
||||||
|
- `enabled`: `true` to start `ovsdb-server`, `ovs-vswitchd`, and `ovn-controller`
|
||||||
|
|
||||||
|
Wait ~10 seconds for services to start, then verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus query net-node-01:/os/1.0/services/ovn
|
||||||
|
# Should show enabled: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Configure Incus OVN connection
|
||||||
|
|
||||||
|
Tell Incus where the OVN **northbound** database is:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus config set net-node-01: network.ovn.northbound_connection \
|
||||||
|
tcp:192.168.1.209:6641
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a cluster-wide setting that propagates to all members. It WILL FAIL
|
||||||
|
if any member doesn't have OVS running (Step 2 not complete).
|
||||||
|
|
||||||
|
Verify all chassis are registered:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ incus exec net-node-01:ovn-central -- ovn-sbctl show
|
||||||
|
Chassis "c1a92a50-..."
|
||||||
|
hostname: net-node-01
|
||||||
|
Encap geneve
|
||||||
|
ip: "192.168.1.209"
|
||||||
|
Chassis "f4fb8f29-..."
|
||||||
|
hostname: net-node-02
|
||||||
|
Encap geneve
|
||||||
|
ip: "192.168.1.150"
|
||||||
|
Chassis "88f8cd95-..."
|
||||||
|
hostname: net-node-03
|
||||||
|
Encap geneve
|
||||||
|
ip: "192.168.1.13"
|
||||||
|
Chassis "6be485be-..."
|
||||||
|
hostname: net-node-04
|
||||||
|
Encap geneve
|
||||||
|
ip: "192.168.1.169"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Assign ovn-chassis roles
|
||||||
|
|
||||||
|
Designate which nodes participate in OVN gateway HA:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||||
|
incus cluster role add net-node-01:"$node" ovn-chassis
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5: Create physical uplink network
|
||||||
|
|
||||||
|
OVN networks need an uplink for external/LAN connectivity:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Per-member definition (two-step cluster pattern)
|
||||||
|
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
|
||||||
|
incus network create net-node-01:UPLINK --type=physical \
|
||||||
|
parent=ens18 --target "$node"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Finalize with external IP range and gateway
|
||||||
|
incus network create net-node-01:UPLINK --type=physical \
|
||||||
|
ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \
|
||||||
|
ipv4.gateway=192.168.1.1/24 \
|
||||||
|
dns.nameservers=192.168.1.1
|
||||||
|
```
|
||||||
|
|
||||||
|
- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox)
|
||||||
|
- `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses
|
||||||
|
- `ipv4.gateway`: LAN gateway in CIDR format
|
||||||
|
|
||||||
|
### Step 6: Create OVN network
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus network create net-node-01:ovn-net1 --type=ovn \
|
||||||
|
network=UPLINK \
|
||||||
|
ipv4.address=10.10.10.1/24 \
|
||||||
|
ipv4.nat=true \
|
||||||
|
ipv6.address=none
|
||||||
|
```
|
||||||
|
|
||||||
|
Incus automatically creates:
|
||||||
|
- **Logical router** with external IP from the uplink range (e.g., 192.168.1.230)
|
||||||
|
- **Internal logical switch** (10.10.10.0/24)
|
||||||
|
- **External logical switch** connected to the physical uplink
|
||||||
|
- **SNAT rule**: 10.10.10.0/24 → 192.168.1.230
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 3: OVN Test Results
|
||||||
|
|
||||||
|
### Cross-node communication: PASS
|
||||||
|
|
||||||
|
The fundamental OVN benefit — instances on different nodes can communicate:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3)
|
||||||
|
$ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3
|
||||||
|
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms
|
||||||
|
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms
|
||||||
|
# 0% packet loss, ~0.1-0.8ms via Geneve tunnel
|
||||||
|
```
|
||||||
|
|
||||||
|
Compare with bridge networking (Part 1) where cross-node ping FAILED.
|
||||||
|
|
||||||
|
### Full mesh connectivity: PASS
|
||||||
|
|
||||||
|
All 12 directional paths between 4 containers on 4 different nodes:
|
||||||
|
|
||||||
|
```
|
||||||
|
test-c1 -> test-c2: PASS (0.672ms) test-c2 -> test-c1: PASS (0.289ms)
|
||||||
|
test-c1 -> test-c3: PASS (0.525ms) test-c3 -> test-c1: PASS (0.335ms)
|
||||||
|
test-c1 -> test-c4: PASS (0.709ms) test-c4 -> test-c1: PASS (0.412ms)
|
||||||
|
test-c2 -> test-c3: PASS (0.539ms) test-c3 -> test-c2: PASS (0.344ms)
|
||||||
|
test-c2 -> test-c4: PASS (0.730ms) test-c4 -> test-c2: PASS (0.366ms)
|
||||||
|
test-c3 -> test-c4: PASS (0.704ms) test-c4 -> test-c3: PASS (0.356ms)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Internet access via NAT: PASS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
|
||||||
|
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms
|
||||||
|
# NAT through OVN router -> physical uplink -> LAN -> internet
|
||||||
|
```
|
||||||
|
|
||||||
|
### LAN access via physical uplink: PASS
|
||||||
|
|
||||||
|
OVN instances can reach any host on the LAN:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# LAN gateway
|
||||||
|
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1
|
||||||
|
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms
|
||||||
|
|
||||||
|
# Proxmox host
|
||||||
|
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29
|
||||||
|
64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms
|
||||||
|
|
||||||
|
# Cluster nodes
|
||||||
|
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13
|
||||||
|
64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 4: Network Isolation (Micro-Segmentation)
|
||||||
|
|
||||||
|
### Multiple OVN networks
|
||||||
|
|
||||||
|
Create a second OVN network with a different subnet:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus network create net-node-01:ovn-net2 --type=ovn \
|
||||||
|
network=UPLINK \
|
||||||
|
ipv4.address=10.10.20.1/24 \
|
||||||
|
ipv4.nat=true \
|
||||||
|
ipv6.address=none
|
||||||
|
```
|
||||||
|
|
||||||
|
Each OVN network gets its own external IP from the uplink range
|
||||||
|
(ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231).
|
||||||
|
|
||||||
|
### Isolation test results
|
||||||
|
|
||||||
|
```
|
||||||
|
Same network, cross-node:
|
||||||
|
test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms)
|
||||||
|
|
||||||
|
Different network, same node:
|
||||||
|
test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss)
|
||||||
|
|
||||||
|
Different network, different node:
|
||||||
|
test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss)
|
||||||
|
|
||||||
|
Both networks have independent internet access:
|
||||||
|
test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Each OVN network is fully isolated.** Instances on different networks cannot
|
||||||
|
communicate, even when running on the same physical node.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 5: Network ACLs
|
||||||
|
|
||||||
|
### Basic ACL: block ICMP
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create ACL
|
||||||
|
incus network acl create net-node-01:block-ping
|
||||||
|
incus network acl rule add net-node-01:block-ping ingress \
|
||||||
|
action=drop protocol=icmp4
|
||||||
|
|
||||||
|
# Apply to an instance's NIC
|
||||||
|
incus config device set net-node-01:test-c1 eth0 security.acls=block-ping
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result:** All inbound ICMP to test-c1 is dropped, including echo replies.
|
||||||
|
This means ping FROM test-c1 also fails (reply packets are dropped by the
|
||||||
|
ingress rule).
|
||||||
|
|
||||||
|
### Targeted ACL: block specific source
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create ACL with default allow + specific drop
|
||||||
|
incus network acl create net-node-01:allow-except-c2
|
||||||
|
incus network acl rule add net-node-01:allow-except-c2 ingress action=allow
|
||||||
|
incus network acl rule add net-node-01:allow-except-c2 ingress \
|
||||||
|
action=drop source=10.10.10.3/32
|
||||||
|
|
||||||
|
# Apply to test-c1
|
||||||
|
incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2
|
||||||
|
```
|
||||||
|
|
||||||
|
**Results:**
|
||||||
|
```
|
||||||
|
test-c3 (allowed) -> test-c1: PASS (0.514ms)
|
||||||
|
test-c4 (allowed) -> test-c1: PASS (0.786ms)
|
||||||
|
test-c2 (blocked) -> test-c1: BLOCKED (100% loss)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Remove ACL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus config device set net-node-01:test-c1 eth0 security.acls=""
|
||||||
|
incus network acl delete net-node-01:allow-except-c2
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 6: Network Peering
|
||||||
|
|
||||||
|
Connect two previously isolated OVN networks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create mutual peering (both directions required)
|
||||||
|
incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2
|
||||||
|
incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Before peering
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss)
|
||||||
|
```
|
||||||
|
|
||||||
|
### After peering
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms)
|
||||||
|
# test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms)
|
||||||
|
```
|
||||||
|
|
||||||
|
Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers.
|
||||||
|
|
||||||
|
### Remove peering
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus network peer delete net-node-01:ovn-net1 peer-to-net2
|
||||||
|
incus network peer delete net-node-01:ovn-net2 peer-to-net1
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 7: Load Balancers
|
||||||
|
|
||||||
|
OVN provides built-in L4 load balancing with LAN-routable virtual IPs.
|
||||||
|
|
||||||
|
### Create load balancer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create LB with a listen address from the uplink range
|
||||||
|
incus network load-balancer create net-node-01:ovn-net1 192.168.1.232
|
||||||
|
|
||||||
|
# Add backends
|
||||||
|
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
|
||||||
|
backend-c1 10.10.10.2 80
|
||||||
|
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
|
||||||
|
backend-c3 10.10.10.4 80
|
||||||
|
|
||||||
|
# Map frontend port to backends
|
||||||
|
incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \
|
||||||
|
tcp 80 backend-c1,backend-c3
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 6 requests to the load balancer VIP
|
||||||
|
Request 1: Hello from test-c1 (node-01)
|
||||||
|
Request 2: Hello from test-c1 (node-01)
|
||||||
|
Request 3: Hello from test-c1 (node-01)
|
||||||
|
Request 4: Hello from test-c3 (node-03)
|
||||||
|
Request 5: Hello from test-c3 (node-03)
|
||||||
|
Request 6: Hello from test-c1 (node-01)
|
||||||
|
```
|
||||||
|
|
||||||
|
Traffic is distributed across both backends. OVN uses connection-based
|
||||||
|
hashing (not strict round-robin).
|
||||||
|
|
||||||
|
### Clean up
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 8: Network Forwards (Port Forwarding)
|
||||||
|
|
||||||
|
Expose internal services on LAN-routable IPs without load balancing.
|
||||||
|
|
||||||
|
### Create forward
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Allocate a LAN IP for forwarding
|
||||||
|
incus network forward create net-node-01:ovn-net1 192.168.1.233
|
||||||
|
|
||||||
|
# Forward external port 8080 to internal instance port 80
|
||||||
|
incus network forward port add net-node-01:ovn-net1 192.168.1.233 \
|
||||||
|
tcp 8080 10.10.10.2 80
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From LAN (dev machine)
|
||||||
|
$ curl http://192.168.1.233:8080/
|
||||||
|
Hello from test-c1 (node-01)
|
||||||
|
|
||||||
|
# From another OVN network
|
||||||
|
$ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/
|
||||||
|
Hello from test-c1 (node-01)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Clean up
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus network forward delete net-node-01:ovn-net1 192.168.1.233
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 9: DNS Resolution
|
||||||
|
|
||||||
|
OVN networks provide automatic DNS for instances within the same network.
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
Instances get `search incus` in `/etc/resolv.conf` via DHCP. DNS queries
|
||||||
|
for instance hostnames are resolved by the OVN DHCP server.
|
||||||
|
|
||||||
|
### Test results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Same network: hostname resolution works
|
||||||
|
$ incus exec net-node-01:test-c2 -- python3 -c "
|
||||||
|
import socket
|
||||||
|
print(socket.gethostbyname('test-c1')) # 10.10.10.2
|
||||||
|
print(socket.gethostbyname('test-c3')) # 10.10.10.4
|
||||||
|
print(socket.gethostbyname('test-c4')) # 10.10.10.5
|
||||||
|
"
|
||||||
|
|
||||||
|
# Cross-network: resolution fails (expected)
|
||||||
|
$ incus exec net-node-01:test-c5 -- python3 -c "
|
||||||
|
import socket
|
||||||
|
socket.gethostbyname('test-c1') # Raises: Name or service not known
|
||||||
|
"
|
||||||
|
|
||||||
|
# Ping by hostname works within the same OVN network
|
||||||
|
$ incus exec net-node-01:test-c2 -- ping -c 1 test-c1
|
||||||
|
PING test-c1 (10.10.10.2) ...
|
||||||
|
64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms
|
||||||
|
```
|
||||||
|
|
||||||
|
DNS is per-network — instances on `ovn-net1` can resolve each other but
|
||||||
|
NOT instances on `ovn-net2`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 10: Final Network State
|
||||||
|
|
||||||
|
After all setup:
|
||||||
|
|
||||||
|
```
|
||||||
|
+----------+----------+---------+---------------+-------------+---------+
|
||||||
|
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | STATE |
|
||||||
|
+----------+----------+---------+---------------+-------------+---------+
|
||||||
|
| UPLINK | physical | YES | | | CREATED |
|
||||||
|
| incusbr0 | bridge | YES | 10.0.0.1/24 | | CREATED |
|
||||||
|
| ovn-net1 | ovn | YES | 10.10.10.1/24 | | CREATED |
|
||||||
|
| ovn-net2 | ovn | YES | 10.10.20.1/24 | | CREATED |
|
||||||
|
+----------+----------+---------+---------------+-------------+---------+
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network summary
|
||||||
|
|
||||||
|
| Network | Type | Subnet | External IP | Purpose |
|
||||||
|
|----------|----------|----------------|----------------|------------------------|
|
||||||
|
| incusbr0 | bridge | 10.0.0.1/24 | NAT via ens18 | Default, node-local |
|
||||||
|
| UPLINK | physical | - | LAN bridge | OVN external gateway |
|
||||||
|
| ovn-net1 | ovn | 10.10.10.1/24 | 192.168.1.230 | Cross-node workloads |
|
||||||
|
| ovn-net2 | ovn | 10.10.20.1/24 | 192.168.1.231 | Isolated workloads |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment Notes
|
||||||
|
|
||||||
|
### OC-provisioned ISO gotchas
|
||||||
|
|
||||||
|
**Stale ISO on Proxmox storage**: When using `incusos-proxmox --iso`, the
|
||||||
|
script checks if an ISO with the same filename already exists on Proxmox
|
||||||
|
storage. If found, it skips the upload. This means a stale ISO from a
|
||||||
|
previous OC server (with a different IP baked in) will be reused, causing
|
||||||
|
all nodes to fail with "no route to host" errors when trying to reach
|
||||||
|
the old OC server for updates.
|
||||||
|
|
||||||
|
**Fix**: Always delete the old ISO from Proxmox before deploying with a
|
||||||
|
new OC server:
|
||||||
|
```bash
|
||||||
|
# Check for stale ISOs
|
||||||
|
ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso"
|
||||||
|
# Delete if present
|
||||||
|
ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso"
|
||||||
|
```
|
||||||
|
|
||||||
|
The `--cleanup-all --deep` command deletes ISOs matching `IncusOS_*.iso`
|
||||||
|
(underscore pattern) but NOT `IncusOS-oc.iso` (custom hyphenated name).
|
||||||
|
|
||||||
|
**Symptom on console** (verified via screenshots):
|
||||||
|
```
|
||||||
|
ERROR Failed to check for Secure Boot key updates
|
||||||
|
err=http request timed out after five seconds:
|
||||||
|
Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1":
|
||||||
|
dial tcp OLD_IP:8443: connect: no route to host
|
||||||
|
provider=operations-center
|
||||||
|
```
|
||||||
|
|
||||||
|
### OC update prerequisites for clustering
|
||||||
|
|
||||||
|
OC v0.3.0 requires all nodes to show `needs update: false` before
|
||||||
|
`provisioning cluster add` will work. Even when nodes are running the
|
||||||
|
latest IncusOS version, OC may report "update pending" indefinitely.
|
||||||
|
|
||||||
|
**Workaround**: Form the cluster manually using `incus cluster enable`
|
||||||
|
and `incus cluster join` instead of `operations-center provisioning
|
||||||
|
cluster add`.
|
||||||
|
|
||||||
|
### Cluster resource creation with apply_defaults: false
|
||||||
|
|
||||||
|
When nodes are deployed with `apply_defaults: false` (recommended for
|
||||||
|
OC-managed nodes and cluster joining nodes), no storage pool, network,
|
||||||
|
or profile devices are created. After cluster formation, create them
|
||||||
|
manually using the two-step pattern:
|
||||||
|
|
||||||
|
1. Define per-member config with `--target <member>`
|
||||||
|
2. Finalize cluster-wide with the creation command (no `--target`)
|
||||||
|
|
||||||
|
This is required because cluster-specific settings like `source` and
|
||||||
|
`zfs.pool_name` must be defined per-member.
|
||||||
|
|
||||||
|
### IncusOS OVN service activation
|
||||||
|
|
||||||
|
The OVN client services (`ovsdb-server`, `ovs-vswitchd`, `ovn-controller`)
|
||||||
|
on IncusOS are **disabled by default**. They must be explicitly enabled
|
||||||
|
via the IncusOS REST API (`/os/1.0/services/ovn`) before Incus can
|
||||||
|
configure OVN networking.
|
||||||
|
|
||||||
|
Setting `network.ovn.northbound_connection` before enabling OVN services
|
||||||
|
on all nodes will fail with:
|
||||||
|
```
|
||||||
|
Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable OVN on each node with the correct southbound database address and
|
||||||
|
tunnel IP **before** setting the Incus northbound connection.
|
||||||
|
|
||||||
|
### OVN control plane in a container
|
||||||
|
|
||||||
|
When running the OVN control plane as a container on `incusbr0` (bridge
|
||||||
|
network), the container is only directly reachable from the node it runs on.
|
||||||
|
Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the
|
||||||
|
host's LAN IP so all cluster nodes can reach it.
|
||||||
Loading…
Reference in New Issue