maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/networking-guide.md

800 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Incus Networking Guide — Bridge, OVN, and LAN Integration
A hands-on tutorial covering Incus networking from basic bridge networking
to OVN overlay networks, tested on a 4-node IncusOS cluster managed by
Operations Center on Proxmox VE.
## Prerequisites
- 4-node IncusOS cluster (this guide uses `net-node-01` through `net-node-04`)
- Operations Center server (for OC-managed deployment)
- Proxmox VE host with `incusos-proxmox` deployment tool
- `incus` client configured with remotes for all cluster nodes
### Lab configuration
```yaml
# incusos/examples/lab-networking.yaml
defaults:
cores: 4
memory: 8192
disk: 50
start_vmid: 400
vms:
- name: net-node-01
app: incus
apply_defaults: false
- name: net-node-02
app: incus
apply_defaults: false
- name: net-node-03
app: incus
apply_defaults: false
- name: net-node-04
app: incus
apply_defaults: false
```
**Resource budget:** OC server (4 GiB) + 4 nodes × 8 GiB = 36 GiB total.
### Cluster formation
Nodes were deployed with `apply_defaults: false`, so no default storage pool,
network, or profile devices exist after installation. After forming the cluster
manually (OC's `cluster add` requires "needs update: false" which was blocked
by a v0.3.0 tracking issue), we created these resources:
```bash
# Storage pool (per-member source, then cluster-wide finalize)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
incus storage create net-node-01:local zfs \
source=local/incus zfs.pool_name=local/incus --target "$node"
done
incus storage create net-node-01:local zfs
# Bridge network (per-member pending, then finalize with config)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
incus network create net-node-01:incusbr0 --type bridge --target "$node"
done
incus network create net-node-01:incusbr0 --type bridge \
ipv4.address=10.0.0.1/24 ipv4.nat=true ipv6.address=none
# Default profile
incus profile device add net-node-01:default root disk path=/ pool=local
incus profile device add net-node-01:default eth0 nic network=incusbr0 name=eth0
```
---
## Part 1: Bridge Networking (Default)
### What is bridge networking?
When Incus creates a bridge network (type `bridge`), it creates a Linux bridge
on each cluster node. Each bridge is **independent** — there is no connectivity
between bridges on different nodes. This is the simplest networking mode and
works without any additional infrastructure.
### Bridge topology
```
Node 1 (net-node-01) Node 2 (net-node-02)
┌─────────────────────┐ ┌─────────────────────┐
│ incusbr0 │ │ incusbr0 │
│ 10.0.0.1/24 │ │ 10.0.0.1/24 │
│ ┌───┐ ┌───┐ │ │ ┌───┐ │
│ │c1 │ │c2 │ │ │ │c3 │ │
│ │.202│ │.40│ │ │ │.52│ │
│ └───┘ └───┘ │ │ └───┘ │
│ │ │ │ │ │ │
│ ──┴──────┴── │ │ ──┴── │
│ (bridge) │ │ (bridge) │
│ │ │ │ │ │
│ NAT ──> ens18 │ │ NAT ──> ens18 │
│ 192.168.1.209 │ │ 192.168.1.150 │
└─────────────────────┘ └─────────────────────┘
│ │
────┴──────────────────────────────────┴────
LAN (192.168.1.0/24)
```
**Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24).
The bridges are NOT connected to each other.
### Test results
#### Same-node communication: PASS
Containers on the same node share the same bridge (same L2 domain):
```bash
# test-c1 (10.0.0.202) -> test-c2 (10.0.0.40), both on net-node-01
$ incus exec net-node-01:test-c1 -- ping -c 3 10.0.0.40
64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=0.029 ms
# 0% packet loss, ~0.03ms latency
```
#### NAT to internet: PASS
The bridge has `ipv4.nat=true`, so containers can reach the internet:
```bash
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.4 ms
# 0% packet loss, ~10ms to Cloudflare DNS
```
#### Cross-node communication: FAIL
Containers on different nodes CANNOT reach each other:
```bash
# test-c1 (node-01, 10.0.0.202) -> test-c3 (node-02, 10.0.0.52)
$ incus exec net-node-01:test-c1 -- ping -c 3 -W 2 10.0.0.52
From 10.0.0.202 icmp_seq=1 Destination Host Unreachable
# 100% packet loss
```
### When to use bridge networking
- Single-node setups (development, testing)
- Workloads that don't need cross-node communication
- Simple NAT-to-internet access
- As an uplink for OVN networks (see Part 2)
### Bridge network configuration options
```bash
# View current config
incus network show net-node-01:incusbr0
# Key settings
ipv4.address # Bridge IP and subnet (e.g., 10.0.0.1/24)
ipv4.nat # Enable NAT for internet access (true/false)
ipv4.dhcp # Enable DHCP server (true/false, default true)
ipv4.dhcp.ranges # DHCP range (e.g., 10.0.0.100-10.0.0.200)
dns.domain # DNS domain for instances (e.g., incus)
dns.mode # DNS mode: managed, dynamic, or none
ipv6.address # IPv6 address (set to "none" to disable)
```
---
## Part 2: OVN Overlay Networking
### Why OVN?
Bridge networking is node-local. To enable cross-node communication without
routing hacks, you need an overlay network. OVN (Open Virtual Network)
provides:
- **Cross-node L2 connectivity** via Geneve tunnels
- **Network isolation** (multiple independent virtual networks)
- **Distributed routing** (no single point of failure)
- **Network ACLs** (firewall rules at the network level)
- **Load balancers** (built-in L4 load balancing)
- **Network forwards** (port forwarding from external IPs)
- **DNS** (automatic hostname resolution between instances)
### OVN architecture on IncusOS
IncusOS includes OVN **client** components only:
- `ovn-controller` (data plane)
- `ovs-vswitchd` (Open vSwitch)
The OVN **control plane** must run separately:
- `ovn-northd` (translates logical to physical flows)
- `ovsdb-server` (northbound + southbound databases)
Options for deploying the control plane:
1. As a container on the Incus cluster itself (used in this guide)
2. On an external host
### OVN topology
```
┌──────────────────────────────────┐
│ OVN Control Plane │
│ (container: ovn-central) │
│ ovn-northd + ovsdb-server │
│ NB: tcp:192.168.1.209:6641 │
│ SB: tcp:192.168.1.209:6642 │
└──────────┬───────────────────────┘
│ (proxy devices)
┌─────────────────────────┼─────────────────────────┐
│ │ │
Node 1 (net-node-01) Node 2 (net-node-02) Node 3 (net-node-03)
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ ovn-controller │ │ ovn-controller │ │ ovn-controller │
│ ovs-vswitchd │ │ ovs-vswitchd │ │ ovs-vswitchd │
│ │ │ │ │ │
│ ┌───┐ ┌───┐ │ │ ┌───┐ │ │ ┌───┐ │
│ │c1 │ │c2 │ │ │ │c3 │ │ │ │c4 │ │
│ │.2 │ │.3 │ │ │ │.4 │ │ │ │.5 │ │
│ └─┬─┘ └─┬─┘ │ │ └─┬─┘ │ │ └─┬─┘ │
│ └──┬───┘ │ │ │ │ │ │ │
│ OVN logical │ │ OVN logical │ │ OVN logical │
│ switch │ │ switch │ │ switch │
│ 10.10.10.0/24 │ │ 10.10.10.0/24 │ │ 10.10.10.0/24 │
│ │ │ │ │ │ │ │ │
│ Geneve tunnel ──┼──┼── Geneve tunnel ──┼──┼── Geneve tunnel │
│ 192.168.1.209 │ │ 192.168.1.150 │ │ 192.168.1.13 │
└───────────────────┘ └───────────────────┘ └───────────────────┘
│ │ │
────┴──────────────────────┴──────────────────────┴────
LAN (192.168.1.0/24)
```
**Key difference from bridge:** All instances share a single logical switch
connected via Geneve tunnels. Cross-node traffic is encapsulated and sent
over the LAN between nodes.
### Step 1: Deploy OVN control plane
Launch a container to run the OVN central services:
```bash
# Launch OVN central container
incus launch images:debian/12 net-node-01:ovn-central --target net-node-01
# Install OVN central
incus exec net-node-01:ovn-central -- apt-get update -qq
incus exec net-node-01:ovn-central -- apt-get install -y ovn-central
# Configure listening on all interfaces
incus exec net-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641
incus exec net-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642
# Verify
incus exec net-node-01:ovn-central -- ovn-nbctl get-connection # ptcp:6641
incus exec net-node-01:ovn-central -- ovn-sbctl get-connection # ptcp:6642
```
The container runs on `incusbr0` (node-local bridge at 10.0.0.x). Other nodes
can't reach this IP directly. Use proxy devices to expose the ports on the
host IP:
```bash
# Forward NB and SB ports from host to container
incus config device add net-node-01:ovn-central ovn-nb proxy \
listen=tcp:0.0.0.0:6641 connect=tcp:127.0.0.1:6641
incus config device add net-node-01:ovn-central ovn-sb proxy \
listen=tcp:0.0.0.0:6642 connect=tcp:127.0.0.1:6642
```
Now all nodes can reach the OVN databases via `tcp:192.168.1.209:6641` and
`tcp:192.168.1.209:6642`.
### Step 2: Enable OVN on IncusOS nodes
**Critical:** The OVN client services on IncusOS are disabled by default.
You MUST enable them on every node before configuring Incus.
Without this step, `incus config set network.ovn.northbound_connection` fails:
```
Error: failed to notify peer: Failed to connect to OVS:
failed to connect to unix:/run/openvswitch/db.sock: no such file or directory
```
Enable OVN via the IncusOS REST API on each node:
```bash
# Enable OVN on each node (set database to SOUTHBOUND port 6642)
incus query net-node-01:/os/1.0/services/ovn --request PUT --data '{
"config": {
"database": "tcp:192.168.1.209:6642",
"enabled": true,
"tunnel_address": "192.168.1.209",
"tunnel_protocol": "geneve"
},
"state": {}
}'
incus query net-node-02:/os/1.0/services/ovn --request PUT --data '{
"config": {
"database": "tcp:192.168.1.209:6642",
"enabled": true,
"tunnel_address": "192.168.1.150",
"tunnel_protocol": "geneve"
},
"state": {}
}'
# Repeat for net-node-03 (192.168.1.13) and net-node-04 (192.168.1.169)
# with their respective tunnel_address values
```
Key fields:
- `database`: OVN **southbound** DB connection (port 6642, NOT 6641)
- `tunnel_address`: this node's LAN IP for Geneve tunnel encapsulation
- `tunnel_protocol`: `geneve` (default and recommended)
- `enabled`: `true` to start `ovsdb-server`, `ovs-vswitchd`, and `ovn-controller`
Wait ~10 seconds for services to start, then verify:
```bash
incus query net-node-01:/os/1.0/services/ovn
# Should show enabled: true
```
### Step 3: Configure Incus OVN connection
Tell Incus where the OVN **northbound** database is:
```bash
incus config set net-node-01: network.ovn.northbound_connection \
tcp:192.168.1.209:6641
```
This is a cluster-wide setting that propagates to all members. It WILL FAIL
if any member doesn't have OVS running (Step 2 not complete).
Verify all chassis are registered:
```bash
$ incus exec net-node-01:ovn-central -- ovn-sbctl show
Chassis "c1a92a50-..."
hostname: net-node-01
Encap geneve
ip: "192.168.1.209"
Chassis "f4fb8f29-..."
hostname: net-node-02
Encap geneve
ip: "192.168.1.150"
Chassis "88f8cd95-..."
hostname: net-node-03
Encap geneve
ip: "192.168.1.13"
Chassis "6be485be-..."
hostname: net-node-04
Encap geneve
ip: "192.168.1.169"
```
### Step 4: Assign ovn-chassis roles
Designate which nodes participate in OVN gateway HA:
```bash
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
incus cluster role add net-node-01:"$node" ovn-chassis
done
```
### Step 5: Create physical uplink network
OVN networks need an uplink for external/LAN connectivity:
```bash
# Per-member definition (two-step cluster pattern)
for node in net-node-01 net-node-02 net-node-03 net-node-04; do
incus network create net-node-01:UPLINK --type=physical \
parent=ens18 --target "$node"
done
# Finalize with external IP range and gateway
incus network create net-node-01:UPLINK --type=physical \
ipv4.ovn.ranges=192.168.1.230-192.168.1.240 \
ipv4.gateway=192.168.1.1/24 \
dns.nameservers=192.168.1.1
```
- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox)
- `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses
- `ipv4.gateway`: LAN gateway in CIDR format
### Step 6: Create OVN network
```bash
incus network create net-node-01:ovn-net1 --type=ovn \
network=UPLINK \
ipv4.address=10.10.10.1/24 \
ipv4.nat=true \
ipv6.address=none
```
Incus automatically creates:
- **Logical router** with external IP from the uplink range (e.g., 192.168.1.230)
- **Internal logical switch** (10.10.10.0/24)
- **External logical switch** connected to the physical uplink
- **SNAT rule**: 10.10.10.0/24 → 192.168.1.230
---
## Part 3: OVN Test Results
### Cross-node communication: PASS
The fundamental OVN benefit — instances on different nodes can communicate:
```bash
# test-c1 (node-01, 10.10.10.2) -> test-c2 (node-03, 10.10.10.3)
$ incus exec net-node-01:test-c1 -- ping -c 3 10.10.10.3
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.836 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.092 ms
# 0% packet loss, ~0.1-0.8ms via Geneve tunnel
```
Compare with bridge networking (Part 1) where cross-node ping FAILED.
### Full mesh connectivity: PASS
All 12 directional paths between 4 containers on 4 different nodes:
```
test-c1 -> test-c2: PASS (0.672ms) test-c2 -> test-c1: PASS (0.289ms)
test-c1 -> test-c3: PASS (0.525ms) test-c3 -> test-c1: PASS (0.335ms)
test-c1 -> test-c4: PASS (0.709ms) test-c4 -> test-c1: PASS (0.412ms)
test-c2 -> test-c3: PASS (0.539ms) test-c3 -> test-c2: PASS (0.344ms)
test-c2 -> test-c4: PASS (0.730ms) test-c4 -> test-c2: PASS (0.366ms)
test-c3 -> test-c4: PASS (0.704ms) test-c4 -> test-c3: PASS (0.356ms)
```
### Internet access via NAT: PASS
```bash
$ incus exec net-node-01:test-c1 -- ping -c 3 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=10.7 ms
# NAT through OVN router -> physical uplink -> LAN -> internet
```
### LAN access via physical uplink: PASS
OVN instances can reach any host on the LAN:
```bash
# LAN gateway
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.1
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.07 ms
# Proxmox host
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.29
64 bytes from 192.168.1.29: icmp_seq=1 ttl=63 time=0.524 ms
# Cluster nodes
$ incus exec net-node-01:test-lan -- ping -c 1 192.168.1.13
64 bytes from 192.168.1.13: icmp_seq=1 ttl=63 time=0.521 ms
```
---
## Part 4: Network Isolation (Micro-Segmentation)
### Multiple OVN networks
Create a second OVN network with a different subnet:
```bash
incus network create net-node-01:ovn-net2 --type=ovn \
network=UPLINK \
ipv4.address=10.10.20.1/24 \
ipv4.nat=true \
ipv6.address=none
```
Each OVN network gets its own external IP from the uplink range
(ovn-net1: 192.168.1.230, ovn-net2: 192.168.1.231).
### Isolation test results
```
Same network, cross-node:
test-c5 (ovn-net2, node-01) -> test-c6 (ovn-net2, node-03): PASS (0.713ms)
Different network, same node:
test-c1 (ovn-net1, node-01) -> test-c5 (ovn-net2, node-01): BLOCKED (100% loss)
Different network, different node:
test-c5 (ovn-net2, node-01) -> test-c3 (ovn-net1, node-03): BLOCKED (100% loss)
Both networks have independent internet access:
test-c5 (ovn-net2) -> 1.1.1.1: PASS (11.6ms)
```
**Each OVN network is fully isolated.** Instances on different networks cannot
communicate, even when running on the same physical node.
---
## Part 5: Network ACLs
### Basic ACL: block ICMP
```bash
# Create ACL
incus network acl create net-node-01:block-ping
incus network acl rule add net-node-01:block-ping ingress \
action=drop protocol=icmp4
# Apply to an instance's NIC
incus config device set net-node-01:test-c1 eth0 security.acls=block-ping
```
**Result:** All inbound ICMP to test-c1 is dropped, including echo replies.
This means ping FROM test-c1 also fails (reply packets are dropped by the
ingress rule).
### Targeted ACL: block specific source
```bash
# Create ACL with default allow + specific drop
incus network acl create net-node-01:allow-except-c2
incus network acl rule add net-node-01:allow-except-c2 ingress action=allow
incus network acl rule add net-node-01:allow-except-c2 ingress \
action=drop source=10.10.10.3/32
# Apply to test-c1
incus config device set net-node-01:test-c1 eth0 security.acls=allow-except-c2
```
**Results:**
```
test-c3 (allowed) -> test-c1: PASS (0.514ms)
test-c4 (allowed) -> test-c1: PASS (0.786ms)
test-c2 (blocked) -> test-c1: BLOCKED (100% loss)
```
### Remove ACL
```bash
incus config device set net-node-01:test-c1 eth0 security.acls=""
incus network acl delete net-node-01:allow-except-c2
```
---
## Part 6: Network Peering
Connect two previously isolated OVN networks:
```bash
# Create mutual peering (both directions required)
incus network peer create net-node-01:ovn-net1 peer-to-net2 ovn-net2
incus network peer create net-node-01:ovn-net2 peer-to-net1 ovn-net1
```
### Before peering
```bash
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): BLOCKED (100% loss)
```
### After peering
```bash
# test-c1 (ovn-net1) -> test-c5 (ovn-net2): PASS (0.410ms)
# test-c5 (ovn-net2) -> test-c3 (ovn-net1): PASS (0.673ms)
```
Note TTL=62 (vs TTL=64 for same-network) — traffic traverses two OVN routers.
### Remove peering
```bash
incus network peer delete net-node-01:ovn-net1 peer-to-net2
incus network peer delete net-node-01:ovn-net2 peer-to-net1
```
---
## Part 7: Load Balancers
OVN provides built-in L4 load balancing with LAN-routable virtual IPs.
### Create load balancer
```bash
# Create LB with a listen address from the uplink range
incus network load-balancer create net-node-01:ovn-net1 192.168.1.232
# Add backends
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
backend-c1 10.10.10.2 80
incus network load-balancer backend add net-node-01:ovn-net1 192.168.1.232 \
backend-c3 10.10.10.4 80
# Map frontend port to backends
incus network load-balancer port add net-node-01:ovn-net1 192.168.1.232 \
tcp 80 backend-c1,backend-c3
```
### Test results
```bash
# 6 requests to the load balancer VIP
Request 1: Hello from test-c1 (node-01)
Request 2: Hello from test-c1 (node-01)
Request 3: Hello from test-c1 (node-01)
Request 4: Hello from test-c3 (node-03)
Request 5: Hello from test-c3 (node-03)
Request 6: Hello from test-c1 (node-01)
```
Traffic is distributed across both backends. OVN uses connection-based
hashing (not strict round-robin).
### Clean up
```bash
incus network load-balancer delete net-node-01:ovn-net1 192.168.1.232
```
---
## Part 8: Network Forwards (Port Forwarding)
Expose internal services on LAN-routable IPs without load balancing.
### Create forward
```bash
# Allocate a LAN IP for forwarding
incus network forward create net-node-01:ovn-net1 192.168.1.233
# Forward external port 8080 to internal instance port 80
incus network forward port add net-node-01:ovn-net1 192.168.1.233 \
tcp 8080 10.10.10.2 80
```
### Test results
```bash
# From LAN (dev machine)
$ curl http://192.168.1.233:8080/
Hello from test-c1 (node-01)
# From another OVN network
$ incus exec net-node-01:test-c5 -- curl http://192.168.1.233:8080/
Hello from test-c1 (node-01)
```
### Clean up
```bash
incus network forward delete net-node-01:ovn-net1 192.168.1.233
```
---
## Part 9: DNS Resolution
OVN networks provide automatic DNS for instances within the same network.
### Configuration
Instances get `search incus` in `/etc/resolv.conf` via DHCP. DNS queries
for instance hostnames are resolved by the OVN DHCP server.
### Test results
```bash
# Same network: hostname resolution works
$ incus exec net-node-01:test-c2 -- python3 -c "
import socket
print(socket.gethostbyname('test-c1')) # 10.10.10.2
print(socket.gethostbyname('test-c3')) # 10.10.10.4
print(socket.gethostbyname('test-c4')) # 10.10.10.5
"
# Cross-network: resolution fails (expected)
$ incus exec net-node-01:test-c5 -- python3 -c "
import socket
socket.gethostbyname('test-c1') # Raises: Name or service not known
"
# Ping by hostname works within the same OVN network
$ incus exec net-node-01:test-c2 -- ping -c 1 test-c1
PING test-c1 (10.10.10.2) ...
64 bytes from test-c1.incus (10.10.10.2): icmp_seq=1 ttl=64 time=0.467 ms
```
DNS is per-network — instances on `ovn-net1` can resolve each other but
NOT instances on `ovn-net2`.
---
## Part 10: Final Network State
After all setup:
```
+----------+----------+---------+---------------+-------------+---------+
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | STATE |
+----------+----------+---------+---------------+-------------+---------+
| UPLINK | physical | YES | | | CREATED |
| incusbr0 | bridge | YES | 10.0.0.1/24 | | CREATED |
| ovn-net1 | ovn | YES | 10.10.10.1/24 | | CREATED |
| ovn-net2 | ovn | YES | 10.10.20.1/24 | | CREATED |
+----------+----------+---------+---------------+-------------+---------+
```
### Network summary
| Network | Type | Subnet | External IP | Purpose |
|----------|----------|----------------|----------------|------------------------|
| incusbr0 | bridge | 10.0.0.1/24 | NAT via ens18 | Default, node-local |
| UPLINK | physical | - | LAN bridge | OVN external gateway |
| ovn-net1 | ovn | 10.10.10.1/24 | 192.168.1.230 | Cross-node workloads |
| ovn-net2 | ovn | 10.10.20.1/24 | 192.168.1.231 | Isolated workloads |
---
## Deployment Notes
### OC-provisioned ISO gotchas
**Stale ISO on Proxmox storage**: When using `incusos-proxmox --iso`, the
script checks if an ISO with the same filename already exists on Proxmox
storage. If found, it skips the upload. This means a stale ISO from a
previous OC server (with a different IP baked in) will be reused, causing
all nodes to fail with "no route to host" errors when trying to reach
the old OC server for updates.
**Fix**: Always delete the old ISO from Proxmox before deploying with a
new OC server:
```bash
# Check for stale ISOs
ssh root@proxmox "ls -la /var/lib/vz/template/iso/IncusOS-oc.iso"
# Delete if present
ssh root@proxmox "rm -f /var/lib/vz/template/iso/IncusOS-oc.iso"
```
The `--cleanup-all --deep` command deletes ISOs matching `IncusOS_*.iso`
(underscore pattern) but NOT `IncusOS-oc.iso` (custom hyphenated name).
**Symptom on console** (verified via screenshots):
```
ERROR Failed to check for Secure Boot key updates
err=http request timed out after five seconds:
Get "https://OLD_IP:8443/1.0/provisioning/updates?recursion=1":
dial tcp OLD_IP:8443: connect: no route to host
provider=operations-center
```
### OC update prerequisites for clustering
OC v0.3.0 requires all nodes to show `needs update: false` before
`provisioning cluster add` will work. Even when nodes are running the
latest IncusOS version, OC may report "update pending" indefinitely.
**Workaround**: Form the cluster manually using `incus cluster enable`
and `incus cluster join` instead of `operations-center provisioning
cluster add`.
### Cluster resource creation with apply_defaults: false
When nodes are deployed with `apply_defaults: false` (recommended for
OC-managed nodes and cluster joining nodes), no storage pool, network,
or profile devices are created. After cluster formation, create them
manually using the two-step pattern:
1. Define per-member config with `--target <member>`
2. Finalize cluster-wide with the creation command (no `--target`)
This is required because cluster-specific settings like `source` and
`zfs.pool_name` must be defined per-member.
### IncusOS OVN service activation
The OVN client services (`ovsdb-server`, `ovs-vswitchd`, `ovn-controller`)
on IncusOS are **disabled by default**. They must be explicitly enabled
via the IncusOS REST API (`/os/1.0/services/ovn`) before Incus can
configure OVN networking.
Setting `network.ovn.northbound_connection` before enabling OVN services
on all nodes will fail with:
```
Failed to connect to OVS: failed to connect to unix:/run/openvswitch/db.sock
```
Enable OVN on each node with the correct southbound database address and
tunnel IP **before** setting the Incus northbound connection.
### OVN control plane in a container
When running the OVN control plane as a container on `incusbr0` (bridge
network), the container is only directly reachable from the node it runs on.
Use Incus proxy devices to expose the NB (6641) and SB (6642) ports on the
host's LAN IP so all cluster nodes can reach it.
Reference in New Issue View Git Blame Copy Permalink