52 KiB

Raw Blame History

Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center dashboard, OC-managed 3-node cluster with OVN overlay networking, Aether management platform, HAProxy load balancing, AWX lifecycle automation, Prometheus/Grafana/Loki observability stack, live migration, network security, and cluster lifecycle management.

All commands and output in this guide are from actual deployments on Proxmox VE 9.1.5 with IncusOS 202602230420, Incus client 6.21, Operations Center v0.3.0, and Aether v6.4.317.

Section 0: Architecture Overview

Network Topology

flowchart TD
    vlan(("VLAN 69<br/>192.168.100.0/22"))

    subgraph mgmt["Management"]
        oc["oc-server<br/>VMID 920 · .120<br/>Operations Center"]
    end

    subgraph cluster["Incus Cluster"]
        n1["oc-node-01<br/>VMID 400 · .140<br/>Aether · OVN central"]
        n2["oc-node-02<br/>VMID 401 · .141<br/>AWX · Monitoring"]
        n3["oc-node-03<br/>VMID 402 · .142<br/>HAProxy backends"]
    end

    subgraph services["Macvlan Services"]
        aether["Aether · .160"]
        awx["AWX · .161"]
    end

    subgraph ovn["OVN · net-prod 10.10.10.0/24"]
        haproxy["HAProxy HA<br/>.50 · .51"]
        backends["nginx backends<br/>.60 · .61 · .62"]
        monitoring["Observability<br/>.70 + node-exp .71-.73"]
    end

    subgraph uplink["UPLINK · 192.168.103.x"]
        vip[".200 HAProxy VIP"]
        fwd[".201 Grafana / Prometheus"]
    end

    vlan --- mgmt & cluster
    cluster --> services & ovn
    ovn -.-> uplink

    classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
    classDef serviceClass fill:#E69F00,color:#fff,stroke:#b87d00
    classDef ovnClass fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 nodeClass
    class oc mgmtClass
    class aether,awx serviceClass
    class haproxy,backends,monitoring ovnClass
    class vip,fwd,vlan networkClass

    style mgmt fill:#f5e6f0,stroke:#CC79A7
    style cluster fill:#e6f5f0,stroke:#009E73
    style services fill:#fef3e0,stroke:#E69F00
    style ovn fill:#e0f2fe,stroke:#56B4E9
    style uplink fill:#e0eef8,stroke:#0072B2

Infrastructure

Component	VMID	IP	Cores	RAM	Disk	Role
oc-server	920	192.168.102.120/22	2	4 GiB	50G	Operations Center
oc-node-01	400	192.168.102.140/22	4	20 GiB	250G	Cluster init + Aether host
oc-node-02	401	192.168.102.141/22	4	20 GiB	100G	AWX + monitoring host
oc-node-03	402	192.168.102.142/22	4	20 GiB	100G	HAProxy backends

RAM budget: 64 GiB of 94 GiB (68%). Host: i9-13900HK, 94 GiB RAM, 881 GiB ZFS pool. Leaves 30 GiB headroom for other VMs on the host.

Inner Cluster Services

Instance	Network	IP	Node	RAM	Description
ovn-central	incusbr0	DHCP	node-01	512 MiB	OVN NB/SB databases
aether	macvlan mgmt	192.168.102.160	node-01	8 GiB	Management platform
awx	macvlan mgmt	192.168.102.161	node-02	8 GiB	Ansible automation
ha-web-01	net-prod	10.10.10.60	node-01	256 MiB	Nginx backend
ha-web-02	net-prod	10.10.10.61	node-02	256 MiB	Nginx backend
ha-web-03	net-prod	10.10.10.62	node-03	256 MiB	Nginx backend
haproxy-01	net-prod	10.10.10.50	varies	512 MiB	HA load balancer
haproxy-02	net-prod	10.10.10.51	varies	512 MiB	HA load balancer
monitoring	net-prod	10.10.10.70	node-02	2 GiB	Prometheus + Grafana + Loki
node-exp-01	net-prod	10.10.10.71	node-01	128 MiB	Host metrics exporter
node-exp-02	net-prod	10.10.10.72	node-02	128 MiB	Host metrics exporter
node-exp-03	net-prod	10.10.10.73	node-03	128 MiB	Host metrics exporter

External IP Allocation

OVN external addresses from UPLINK range (192.168.103.200-210). Exclude these from your DHCP server's range:

IP	Purpose
192.168.103.200	HAProxy VIP (OVN load balancer → haproxy-01/02)
192.168.103.201	Observability forward (Grafana :3000, Prometheus :9090)

Decision Rationale

Why OC-managed clustering? OC provisioning cluster add is the production path for Incus deployments. It handles cluster formation, update management, and inventory centrally. The deploy scripts (deploy-haproxy, deploy-awx, deploy-observability) are built for the oc-node-* naming and IP scheme.

Why 20 GiB RAM per node? Aether requires 8 GiB. AWX requires 4-8 GiB. Monitoring + HAProxy + backends need ~3 GiB total. Leaves headroom for mixed workloads and live migration.

Why 250 GiB disk for node-01? Aether's golden image is 200 GiB virtual (qcow2). With ZFS thin provisioning only ~11 GiB is used initially, but the pool needs 200 GiB allocatable space.

Why OVN? Bridge networks are node-local — instances on different nodes cannot communicate. OVN provides cross-node L2 overlay with sub-ms latency, network isolation, ACLs, load balancers, and network forwards — essential for HAProxy HA and distributed workloads.

Why VLAN 69? Isolates lab traffic from the production LAN. All VMs share VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox level — IncusOS and workloads are unaware of it.

Cross-References

This guide integrates techniques from the deep-dive guides:

Clustering Guide — manual cluster formation reference
Networking Guide — OVN setup, ACLs, peering
Operations Center Guide — OC provisioning, CLI, web UI
Aether Guide — management platform deployment and API
HAProxy Guide — HA load balancing with Aether
AWX Guide — Ansible lifecycle automation
Observability Guide — Prometheus, Grafana, Loki stack

Section 1: Prerequisites

Required Tools

Verify all tools are available before starting:

incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1

Minimum versions: Incus client 6.3+, Operations Center v0.3.0+.

Aether Browser Automation

Several Aether features (HAProxy management, blueprint deployment) are not in the JWT API — they use session-authenticated routes with CSRF protection. Playwright browser automation is required for Sections 7-9:

node --version         # Node.js 18+
npx playwright --version

Install if missing:

npm install playwright @playwright/mcp
npx playwright install chromium

The Playwright MCP server (configured in .mcp.json) provides browser tools when available. The incusos/helpers/aether-browser script is the standalone alternative.

Aether Golden Image

The Aether golden image must be available locally before Section 7:

ls -la sources/aether-golden-image-v6.tar.gz

This is an Ubuntu Noble (24.04) image with 200 GiB virtual disk (~6.6 GiB compressed). Obtain it from the Aether distribution.

Proxmox Configuration

Your incusos/proxmox.yaml should contain:

host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab

The env file at the repository root must export PROXMOX_TOKEN_SECRET and AETHER_ADMIN_PASSWORD. Scripts auto-discover them.

Client Certificates

Incus client certificates are used for Incus, OC, and Prometheus (metrics scraping) connections:

# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key

Copy certs for OC CLI:

mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

For OC web UI browser access, generate a PKCS#12 bundle:

openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"

Import client.pfx into your browser's certificate store (Firefox: Settings → Privacy & Security → View Certificates → Import).

Doctor Check

Run the environment check to verify everything is in order:

cd incusos
./incusos-proxmox --doctor

Expected output includes tool versions, IncusOS CDN reachability, proxmox.yaml discovery, and Proxmox API connectivity.

Section 2: Deploy OC Server

Configuration File

# incusos/examples/lab-oc-deploy.yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
    ip: 192.168.102.120/22

Deploy

./incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml

Actual output (key lines):

[ok]  VM 'oc-server' created (VMID 920)
[ok]  VM 'oc-server' installed and running at 192.168.102.120

Set Up OC CLI Remote

# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab

Important: The OC CLI does not support the remote: suffix syntax that the Incus CLI uses. Switch to the remote first, then run commands without a remote suffix.

Verify OC

operations-center admin os show

Actual output (uptime will vary):

WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602230420"
  os_version_next: ""
  uptime: 63

Wait for Updates

OC downloads IncusOS update packages from upstream. At least one update must reach ready state before ISOs can be generated:

# Poll until at least one update shows "ready"
operations-center provisioning update list

Actual output (after ~8 minutes; UUIDs are stable across deployments):

UUID	Origin	Channels	Version	Severity	Status
82aefab7-fec7-5122-89fd-8412d3d2174c	linuxcontainers.org	stable	202602200553	none	ready
5d6b1018-e534-5e54-aeb5-c9e6027ab31d	linuxcontainers.org	stable	202602210344	none	ready
c912a390-c38b-5bd9-b46f-ccaeba6da68a	linuxcontainers.org	stable	202602230420	none	ready

The table also includes Upstream Channels and Published At columns (omitted for width). Not all updates may be ready simultaneously — at least one ready is sufficient to proceed.

Web UI Access

Open https://192.168.102.120:8443/ui/ in your browser (with client.pfx imported from Section 1). The web UI provides a dashboard view of the OC server, update status, provisioning tokens, and system configuration.

Section 3: Provision Nodes

3.1 Create Provisioning Token

operations-center provisioning token add --uses 5 --description "Production lab cluster"
operations-center provisioning token list

Actual output (UUID changes every run):

UUID	Uses Remaining	Expire At	Channel	Description
	5	<30 days from now>	stable	Production lab cluster

Save the <token-UUID> — you'll need it for the next steps.

3.2 Create Token Seed (No force_reboot)

Critical: the token seed must NOT include force_reboot. On Proxmox, incusos-proxmox handles the install lifecycle externally (blockstat detection + media removal). force_reboot triggers SysRq-B which causes the crontab bug (~50% failure rate).

# /tmp/oc-preseed.yaml
install:
  version: "1"
  force_install: true

Important: use the structured format with section keys (install:). A flat format (version: "1" at root) maps fields to empty {} and they don't get assigned to any section.

operations-center provisioning token seed add <token-UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"

3.3 Generate OC-Provisioned ISO (Older Version)

Critical discovery: nodes deployed from an ISO matching the latest OC update version are tracked as needs_update: true by OC because the OS was never delivered through OC's update pipeline. The fix: generate the ISO from an older channel so OC can push the real update after deployment.

# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
    --description "Older stable versions for initial provisioning"

# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable

# Generate ISO from the older channel
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable

Actual output:

Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"

The ISO contains IncusOS 202602210344 (one version behind). OC will push the latest (202602230420) after nodes register.

3.4 Node Configuration

# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 400

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false
    disk: 250
    ip: 192.168.102.140/22

  - name: oc-node-02
    app: incus
    apply_defaults: false
    ip: 192.168.102.141/22

  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22

Key decisions:

20 GiB RAM per node: Aether needs 8 GiB, AWX needs 4-8 GiB
250 GiB disk for node-01: hosts Aether's 200 GiB virtual image
100 GiB disk for nodes 02-03: sufficient for AWX, monitoring, HAProxy
apply_defaults: false for all nodes: OC's Terraform handles resource creation during cluster formation

3.5 Deploy Nodes (Hybrid Approach)

The hybrid approach uses incusos-proxmox --iso to combine OC auto-registration (from the boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup.

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml

Actual output (key lines):

[ok]  ISO uploaded: IncusOS-oc.iso
[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
[ok]  Remote 'oc-node-01' added (192.168.102.140)
[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
[ok]  Remote 'oc-node-02' added (192.168.102.141)
[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
[ok]  Remote 'oc-node-03' added (192.168.102.142)
[ok]  All post-deployment checks passed

All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.

3.6 Verify Auto-Registration

Nodes auto-register with OC within ~30 seconds of first boot. The update from 202602210344 to 202602230420 happens automatically:

operations-center provisioning server list

Actual output (key columns; full table includes Type, Channel, Certificate Fingerprint, Public Connection URL, Last Updated, Last Seen):

Name	Connection URL	Status	Update Status
oc-node-01	https://192.168.102.140:8443	ready	up to date
oc-node-02	https://192.168.102.141:8443	ready	up to date
oc-node-03	https://192.168.102.142:8443	ready	up to date
operations-center	https://[::1]:8443	ready	update pending

Key: all 3 nodes show "up to date" because OC delivered the 202602230420 update through its pipeline. This is what unlocks clustering. Nodes may already be up to date by the time the last node finishes deploying — the update gets pushed while incusos-proxmox deploys subsequent nodes sequentially.

3.7 Verify Scrub Schedules

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${node}:/os/1.0/system/storage | python3 -c \
        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done

Actual output:

oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0

All healthy. No crontab bug (force_reboot was not used).

Section 4: Form Cluster via Operations Center

4.1 The needs_update Blocker

OC requires all nodes to show needs_update: false before clustering. Nodes deployed from an ISO matching the latest version are tracked as needs_update: true because the OS was never delivered through OC's update pipeline. The needs_update flag is server-side computed and cannot be overridden via REST API PUT.

Solution: deploy from an older ISO version (Section 3.3). OC then pushes the real update to nodes through its pipeline, clearing the flag.

4.2 Form Cluster

Important: if the client certificate was already injected via SEED_DATA, use an empty application seed config to avoid "Certificate already in trust store" Terraform errors:

echo '{}' > /tmp/oc-app-config.yaml

operations-center provisioning cluster add oc-cluster \
    https://192.168.102.140:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC orchestrates the full cluster formation:

Sets core.https_address to each node's specific IP
Enables clustering on oc-node-01
Joins oc-node-02 and oc-node-03
Creates storage pool (local), networks (incusbr0, meshbr0)
Runs Terraform/OpenTofu for post-cluster configuration

4.3 Fix Remotes After Clustering

Clustering regenerates TLS certificates. Re-add the remotes:

incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate

4.4 Verify Cluster

incus cluster list oc-node-01:

Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):

NAME	URL	ROLES	ARCHITECTURE	STATUS	MESSAGE
oc-node-01	https://192.168.102.140:8443	database-leader, database	x86_64	ONLINE	Fully operational
oc-node-02	https://192.168.102.141:8443	database	x86_64	ONLINE	Fully operational
oc-node-03	https://192.168.102.142:8443	database	x86_64	ONLINE	Fully operational

All 3 nodes ONLINE and Fully operational. The ovn-chassis role is added later in Section 6.4.

4.5 Cluster Resources Created by OC

incus storage list oc-node-01:
incus network list oc-node-01:

Actual output (incusbr0 subnet varies per deployment):

NAME	DRIVER	DESCRIPTION	USED BY	STATE
local	zfs	Local storage pool (on system drive)	8	CREATED

NAME	TYPE	MANAGED	IPV4	DESCRIPTION	USED BY
incusbr0	bridge	YES	10.x.x.1/24	Local network bridge (NAT)	1
meshbr0	bridge	YES	none	Internal mesh network bridge	1

OC creates: local storage pool (ZFS), incusbr0 bridge (NAT), and meshbr0 (OC-specific mesh network for inter-node communication). The table also includes IPv6 and STATE columns.

Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are node-local. This demonstrates why OVN is needed.

Same-Node Communication

Launch 2 containers on the same node. Important: use --target to force placement — without it, the cluster scheduler may place containers on different nodes automatically.

Important: launch containers one at a time, not chained with &&. The first launch on a fresh cluster downloads the image (~1 GB), which takes 2-3 minutes. Subsequent launches on the same node use the cached image and are instant. Launches targeting a different node trigger another image transfer to that node.

incus launch images:debian/12 oc-node-01:test-bridge-a --target oc-node-01
incus launch images:debian/12 oc-node-01:test-bridge-b --target oc-node-01

Wait for them to get IPs:

incus list oc-node-01: --columns ns4 --format csv | grep test-bridge

Ping between them:

IP_B=$(incus list oc-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 "$IP_B"

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

Cross-Node Communication (Fails)

Launch a container on a different node:

incus launch images:debian/12 oc-node-01:test-bridge-c --target oc-node-02

Wait for IP:

incus list oc-node-01: --columns ns4 --format csv | grep test-bridge

Ping from node-01 to node-02:

IP_C=$(incus list oc-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"

Actual result: 100% packet loss. Bridge networks are node-local — there is no L2 path between incusbr0 on node-01 and incusbr0 on node-02. Each node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are separate L2 domains.

Internet Access

NAT to the internet works from any node:

incus exec oc-node-01:test-bridge-a -- ping -c 3 1.1.1.1

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via the host's management interface.

Cleanup

incus delete oc-node-01:test-bridge-a --force
incus delete oc-node-01:test-bridge-b --force
incus delete oc-node-01:test-bridge-c --force

Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this section, containers on any node can communicate transparently.

6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01

Install OVN:

incus exec oc-node-01:ovn-central -- bash -c \
    "apt-get update -qq && apt-get install -y -qq ovn-central"

Configure OVN to listen on all interfaces:

incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0

Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:

incus config device add oc-node-01:ovn-central \
    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642

6.2 Enable OVN on All IncusOS Nodes

Enable OVN services via the IncusOS REST API (/os/1.0/services/ovn). The database field is the southbound DB (port 6642), not northbound.

for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
        \"config\": {
            \"database\": \"tcp:192.168.102.140:6642\",
            \"enabled\": true,
            \"tunnel_address\": \"${node_ip}\",
            \"tunnel_protocol\": \"geneve\"
        },
        \"state\": {}
    }"
done

Each call should return {} on success.

6.3 Configure Incus OVN Connection

Point Incus to the northbound DB (port 6641):

incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641

6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the ovn-chassis role:

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role add oc-node-01:${node} ovn-chassis
done

Verify:

incus cluster list oc-node-01:

The ROLES column should now include ovn-chassis for each member.

6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and the physical LAN. It uses the two-step cluster pattern: per-member --target first, then cluster-wide create.

Important: IncusOS names its management NIC mgmt, NOT ens18. Using parent=ens18 will fail with "Parent interface 'ens18' not found".

# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done

# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1

6.6 Create OVN Network (net-prod)

incus network create oc-node-01:net-prod --type=ovn \
    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true

Actual output:

Network net-prod created

net-prod is assigned external IP 192.168.103.200 from the UPLINK range.

6.7 Verify Cross-Node OVN Connectivity

incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3

Actual output:

64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms

Sub-millisecond cross-node latency via Geneve tunnels. Clean up test containers after verification:

incus delete oc-node-01:test-1 oc-node-01:test-2 --force

6.8 Final Network State

incus network list oc-node-01:

Actual output:

NAME	TYPE	MANAGED	IPV4	DESCRIPTION	USED BY
UPLINK	physical	YES			1
incusbr0	bridge	YES	10.x.x.1/24	Local network bridge (NAT)	2
meshbr0	bridge	YES	none	Internal mesh network bridge	1
net-prod	ovn	YES	10.10.10.1/24		0

The incusbr0 subnet is randomly assigned per deployment. The USED BY count for net-prod is 0 at this point (test containers deleted); it increases as workloads are added in subsequent sections.

Section 7: Mixed Workloads

Deploy a realistic workload mix: web servers, application containers, and VMs configured for live migration.

7.1 Containers on net-prod

Deploy containers with targeted placement across nodes. Run each launch command one at a time — each new target node needs to download the image from the cluster (~1 GB transfer, 2-3 minutes per node):

incus launch images:debian/12 oc-node-01:prod-web-01 --network net-prod --target oc-node-01
incus launch images:debian/12 oc-node-01:prod-web-02 --network net-prod --target oc-node-02
incus launch images:debian/12 oc-node-01:prod-api-01 --network net-prod --target oc-node-03

Install nginx on the web servers:

incus exec oc-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec oc-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx"

Set distinct content to verify load balancing later:

incus exec oc-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html"
incus exec oc-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html"

Install nginx on the API container:

incus exec oc-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec oc-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html"

7.2 VMs (Migration-Ready)

Deploy VMs with live migration configuration:

incus launch images:debian/12 oc-node-01:prod-db-01 --vm --network net-prod --target oc-node-01
incus launch images:debian/12 oc-node-01:prod-app-01 --vm --network net-prod --target oc-node-02

VMs may take longer to boot than containers (~30-60s for image download + boot). If the VMs show as STOPPED, start them explicitly:

incus start oc-node-01:prod-db-01
incus start oc-node-01:prod-app-01

Wait for the VM agent to become available, then verify:

# Check VM agent is running
incus exec oc-node-01:prod-db-01 -- uname -a
incus exec oc-node-01:prod-app-01 -- uname -a

7.3 Configure VMs for Live Migration

Critical: use limits.cpu as a range (e.g., 0-1), not an integer. Without the range, QEMU sets maxcpus based on the host's CPU count, which varies across nodes and breaks migration with Missing section footer for ICH9LPC.

Stop VMs before configuring migration.stateful:

incus stop oc-node-01:prod-db-01
incus stop oc-node-01:prod-app-01

Configure migration settings:

# prod-db-01
incus config set oc-node-01:prod-db-01 limits.cpu=0-1
incus config set oc-node-01:prod-db-01 migration.stateful=true
incus config device override oc-node-01:prod-db-01 root size.state=2GiB

# prod-app-01
incus config set oc-node-01:prod-app-01 limits.cpu=0-1
incus config set oc-node-01:prod-app-01 migration.stateful=true
incus config device override oc-node-01:prod-app-01 root size.state=2GiB

Important: use device override (not device set) because the root device comes from the default profile. device set fails with "Device from profile(s) cannot be modified for individual instance".

Start the VMs:

incus start oc-node-01:prod-db-01
incus start oc-node-01:prod-app-01

7.4 Workload Distribution

View the full workload distribution:

incus list oc-node-01: --columns nstL4 --format table

Expected layout:

NAME	STATE	TYPE	LOCATION	IPV4
ovn-central	RUNNING	CONTAINER	oc-node-01	...
prod-web-01	RUNNING	CONTAINER	oc-node-01	10.10.10.x (net-prod)
prod-db-01	RUNNING	VIRTUAL-MACHINE	oc-node-01	10.10.10.x (net-prod)
prod-web-02	RUNNING	CONTAINER	oc-node-02	10.10.10.x (net-prod)
prod-app-01	RUNNING	VIRTUAL-MACHINE	oc-node-02	10.10.10.x (net-prod)
prod-api-01	RUNNING	CONTAINER	oc-node-03	10.10.10.x (net-prod)

Section 8: Network Isolation & Security

8.1 Create Isolated Network

incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none

8.2 Launch Isolated Containers

incus launch images:debian/12 oc-node-01:iso-app-01 --network net-isolated --target oc-node-01
incus launch images:debian/12 oc-node-01:iso-app-02 --network net-isolated --target oc-node-02

8.3 Verify Network Isolation

Containers on net-isolated can reach each other:

IP_ISO2=$(incus list oc-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the same OVN network can reach each other across nodes.

But net-prod cannot reach net-isolated:

incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"

Actual result: 100% packet loss. Different OVN networks are fully isolated — separate L2 domains, no routing between them.

8.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

incus network acl create oc-node-01:block-ping
incus network acl rule add oc-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.10.0/24 \
    description="Block ICMP from net-prod subnet"

8.5 Apply and Test ACL

Apply the ACL to net-isolated:

incus network set oc-node-01:net-isolated security.acls=block-ping

Verify ICMP is blocked between net-isolated containers (since they match the source range — adjust the ACL source for targeted blocking):

incus exec oc-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"

Remove the ACL:

incus network unset oc-node-01:net-isolated security.acls

Verify ICMP works again:

incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

8.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can communicate. Peering is bilateral — create a peer on both sides:

# From net-prod's perspective
incus network peer create oc-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create oc-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"

8.7 Verify Peering

Cross-network ping (prod → isolated):

incus exec oc-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops), confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

IP_WEB1=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1"

8.8 Remove Peering

incus network peer delete oc-node-01:net-prod peer-to-isolated
incus network peer delete oc-node-01:net-isolated peer-to-prod

Verify isolation is restored:

incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

incus delete oc-node-01:iso-app-01 --force
incus delete oc-node-01:iso-app-02 --force

Section 9: Load Balancers & Network Forwards

9.1 Create OVN Load Balancer

Create a load balancer with a VIP from the UPLINK range:

incus network load-balancer create oc-node-01:net-prod 192.168.103.200

Add backend servers. Important: backends require the instance's IP address, not its name. Get the IPs first:

WEB1_IP=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
WEB2_IP=$(incus list oc-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1)
echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP"

Add backends using IP addresses:

incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \
    web-01 "$WEB1_IP" 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \
    web-02 "$WEB2_IP" 80

Add a port mapping:

incus network load-balancer port add oc-node-01:net-prod 192.168.103.200 \
    tcp 80 web-01,web-02

9.2 Test Load Balancer

From your dev machine (must be on the same VLAN or have routing to 192.168.103.0/24):

for i in $(seq 1 6); do
    curl -s http://192.168.103.200
done

Actual output:

Server: prod-web-01
Server: prod-web-01
Server: prod-web-01
Server: prod-web-02
Server: prod-web-02
Server: prod-web-02

OVN uses connection-based hashing (not round-robin). Multiple requests from the same source will typically hit the same backend. Different source ports or connections may hit different backends.

9.3 Create Network Forward

Network forwards expose internal services on LAN IPs. Forward tcp:8080 → prod-api-01:80. Like LB backends, forwards require IP addresses:

API_IP=$(incus list oc-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1)

incus network forward create oc-node-01:net-prod 192.168.103.201
incus network forward port add oc-node-01:net-prod 192.168.103.201 \
    tcp 8080 "$API_IP" 80

9.4 Test Network Forward

curl -s http://192.168.103.201:8080

Actual output: API: prod-api-01

9.5 DNS Resolution

OVN provides per-network DNS. Containers can resolve each other by hostname:

incus exec oc-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus"

Actual output: 10.10.10.3 — OVN DNS resolves instance names within each network.

Section 10: Live Migration

10.1 Verify Migration Readiness

Check that VMs have the required configuration:

for vm in prod-db-01 prod-app-01; do
    echo "=== $vm ==="
    incus config get oc-node-01:$vm limits.cpu
    incus config get oc-node-01:$vm migration.stateful
    incus config device get oc-node-01:$vm root size.state
done

Expected: 0-1, true, 2GiB for each VM.

10.2 Create Heartbeat Service

Create a simple counter in prod-db-01 to verify state continuity across migration:

incus exec oc-node-01:prod-db-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'

Read the counter:

incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Note the value. After migration, the counter should continue from where it left off (live migration preserves running state).

10.3 Live Migration Round-Trip

Check current location:

incus list oc-node-01:prod-db-01 --columns nL --format csv

Migrate node-01 → node-02:

time incus move oc-node-01:prod-db-01 --target oc-node-02

Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect:

sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Counter went from 9 → 25. The heartbeat process was never interrupted — it continued counting during migration.

Migrate node-02 → node-03:

time incus move oc-node-01:prod-db-01 --target oc-node-03
sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Actual result: 7.379s. Counter went to 41.

Migrate node-03 → node-01 (back to origin):

time incus move oc-node-01:prod-db-01 --target oc-node-01
sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Actual result: 6.896s. Counter went to 56.

Verify the VM is back on node-01:

incus list oc-node-01:prod-db-01 --columns nL --format csv

10.4 Active I/O During Migration

Start a continuous write inside the VM:

incus exec oc-node-01:prod-db-01 -- bash -c '
    dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 &
    echo "Write started, PID: $!"
'

Migrate while I/O is active:

time incus move oc-node-01:prod-db-01 --target oc-node-02
sleep 4

Verify the file exists and is intact:

incus exec oc-node-01:prod-db-01 -- ls -la /tmp/testfile
incus exec oc-node-01:prod-db-01 -- md5sum /tmp/testfile

Move back:

incus move oc-node-01:prod-db-01 --target oc-node-01
sleep 4

10.5 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly where it was:

# Note the heartbeat counter
incus exec oc-node-01:prod-app-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    echo 42 > /tmp/heartbeat/counter
    cat /tmp/heartbeat/counter
'

Stateful stop:

incus stop oc-node-01:prod-app-01 --stateful

Start (resumes from saved state):

incus start oc-node-01:prod-app-01
sleep 4
incus exec oc-node-01:prod-app-01 -- cat /tmp/heartbeat/counter

Expected: 42 — the file (and entire VM state) is preserved.

If the restore fails (e.g., from a limits.cpu mismatch), discard the saved state:

incus start oc-node-01:prod-app-01 --stateless

Section 11: Cluster Lifecycle

11.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

incus cluster evacuate oc-node-01:oc-node-02 --force

Check workload distribution — nothing on node-02:

incus list oc-node-01: --columns nstL --format table

Actual behavior: VMs with migration.stateful=true are live-migrated (prod-app-01 migrated to oc-node-03). Containers are stopped and moved (prod-web-02 stopped, moved to oc-node-03, then started). The --force flag skips confirmation prompts.

Note: if VMs lack the limits.cpu range fix, use --action stop instead to avoid migration failures:

incus cluster evacuate oc-node-01:oc-node-02 --force --action stop

Verify node-02 shows EVACUATED:

incus cluster list oc-node-01:

Restore node-02 (workloads return):

incus cluster restore oc-node-01:oc-node-02 --force

Verify all workloads are back:

incus list oc-node-01: --columns nstL --format table
incus cluster list oc-node-01:

All nodes should show ONLINE.

11.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat detects the failure in ~40 seconds. After the node is restarted:

The node auto-rejoins the cluster (~60s)
Containers auto-start
VMs that were running resume

Procedure (document only — do not execute while OVN is running unless you can tolerate temporary network disruption):

# Simulate crash: hard-stop via Proxmox API
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/stop

# Wait for heartbeat detection (~40s)
# incus cluster list oc-node-01:
# → oc-node-02 shows OFFLINE

# Restart via Proxmox
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/start

# Wait for auto-rejoin (~60s)
# incus cluster list oc-node-01:
# → oc-node-02 shows ONLINE

11.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

Step 1: Evacuate node-03:

incus cluster evacuate oc-node-01:oc-node-03 --force --action stop

Step 2: Remove from cluster:

printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

Note: incus cluster remove prompts "Are you really sure?" even with --force. The printf pipes yes for automation.

Step 3: Clean up the remote:

incus remote remove oc-node-03

Step 4: Destroy and redeploy the VM. Use incusos-proxmox to destroy just node-03 (VMID 402) and redeploy it. Create a single-VM config:

# /tmp/lab-replace-node03.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 402

vms:
  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22

./incusos-proxmox --cleanup --yes /tmp/lab-replace-node03.yaml
./incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/lab-replace-node03.yaml

Step 5: Join the fresh node to the cluster:

# Set specific IP
incus config set oc-node-03: core.https_address 192.168.102.142:8443

# Generate join token
incus cluster add oc-node-01:oc-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:

# Fix remote
incus remote remove oc-node-03
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate

Step 6: Re-enable OVN on the replacement node:

incus query oc-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.140:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.142",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add oc-node-01:oc-node-03 ovn-chassis

Step 7: Verify:

incus cluster list oc-node-01:

All 3 nodes should be ONLINE with ovn-chassis role.

11.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads are unevenly distributed), Incus redistributes VMs:

incus config set oc-node-01: cluster.rebalance.interval=1
incus config set oc-node-01: cluster.rebalance.threshold=10
incus config set oc-node-01: cluster.rebalance.batch=2
incus config set oc-node-01: cluster.rebalance.cooldown=5m

Important: only VMs with migration.stateful=true are rebalanced. Containers are NOT auto-rebalanced.

Monitor rebalancing:

incus list oc-node-01: --columns nstL --format table

Disable rebalancing when done testing:

incus config unset oc-node-01: cluster.rebalance.interval
incus config unset oc-node-01: cluster.rebalance.threshold

Section 12: OC Dashboard

Important: Switch to the OC remote first. The OC CLI does not support remote: suffix syntax:

operations-center remote switch oc-lab

OC Server Information

operations-center admin os show

Actual output (version and uptime will vary):

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602240349"
  os_version_next: ""
  uptime: 3600

Provisioning Status

OC manages all 3 cluster nodes. Verify they're registered and up to date:

operations-center provisioning server list

All nodes should show ready status and up to date update status.

Web UI

The OC web UI at https://192.168.102.120:8443/ui/ provides:

Dashboard: server overview with resource utilization
Updates: available IncusOS updates and delivery status
Provisioning: token management, server list, cluster formation
System: OC configuration and certificates

Because the nodes were deployed from an OC-provisioned ISO (Section 3), OC has full visibility and management of the cluster — including update delivery, server inventory, and cluster formation.

Section 13: Cleanup

Delete All Workloads

# Delete containers
for c in prod-web-01 prod-web-02 prod-api-01; do
    incus delete oc-node-01:$c --force
done

# Delete VMs
for vm in prod-db-01 prod-app-01; do
    incus delete oc-node-01:$vm --force
done

Remove OVN Networks

# Delete OVN networks
incus network delete oc-node-01:net-prod
incus network delete oc-node-01:net-isolated 2>/dev/null || true

# Delete UPLINK
incus network delete oc-node-01:UPLINK

Remove OVN Control Plane

incus delete oc-node-01:ovn-central --force

Disable OVN Services

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done

Infrastructure Options

Keep infrastructure (stop VMs, keep on disk for later):

./incusos-proxmox --lab-down examples/lab-oc-nodes.yaml
./incusos-proxmox --lab-down examples/lab-oc-deploy.yaml

Restart later with:

./incusos-proxmox --lab-up examples/lab-oc-deploy.yaml
./incusos-proxmox --lab-up examples/lab-oc-nodes.yaml

Full teardown (destroy all VMs, remove ISOs, remotes, cache):

./incusos-proxmox --cleanup-all --deep --yes

Section 14: Verification Checklist

#	Check	Command	Expected
1	All VMs running	`incusos-proxmox --status examples/lab-oc-nodes.yaml`	4 VMs running, port 8443 open
2	Scrub schedule healthy	`incus query oc-node-01:/os/1.0/system/storage`	`scrub_schedule: "0 4 * * 0"`
3	OC accessible	`operations-center remote switch oc-lab && operations-center admin os show`	Shows version, uptime
4	Cluster formed	`incus cluster list oc-node-01:`	3 nodes ONLINE
5	Storage pool	`incus storage list oc-node-01:`	`local` pool on all members
6	Bridge isolation	Ping cross-node on incusbr0	100% loss (expected)
7	OVN connectivity	Ping cross-node on net-prod	0% loss
8	Internet via OVN	`ping 1.1.1.1` from OVN container	0% loss
9	Network isolation	Ping net-prod → net-isolated	100% loss (expected)
10	Network peering	Peer + ping cross-network	0% loss, TTL=62
11	Load balancer	`curl http://192.168.103.200`	Backend response
12	Network forward	`curl http://192.168.103.201:8080`	API response
13	DNS resolution	`dig prod-web-02.incus` from container	Resolves to 10.10.10.x
14	VM live migration	`incus move` VM between nodes	State preserved
15	Cluster evacuation	`incus cluster evacuate` + `restore`	Workloads moved and returned
16	Stateful stop/start	`incus stop --stateful` + `start`	VM state preserved

Section 15: Quick Reference

Cluster Command Syntax

Command	Arguments	Notes
`incus cluster enable`	`remote: member-name`	TWO args (space between)
`incus cluster add`	`remote:member-name`	ONE arg (no space)
`incus cluster join`	`init-remote: joining-remote:`	TWO args (space between)
`incus cluster remove`	`remote:member-name --force`	ONE arg; prompts even with `--force`
`incus cluster evacuate`	`remote:member-name`	ONE arg (no space)
`incus cluster restore`	`remote:member-name`	ONE arg (no space)
`incus config set`	`remote: key value`	Remote with trailing colon + space
`incus storage show`	`remote:pool`	ONE arg (no space)
`incus storage show`	`remote:pool --target member`	`--target` for member-specific

OVN Setup Cheat Sheet

# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true

Migration Readiness Checklist

Setting	Value	Why
`limits.cpu`	Range (e.g., `0-1`)	Fixed QEMU topology across hosts
`migration.stateful`	`true`	Enables live migration
`root size.state`	`2GiB` (or `4GiB` for 3-4 vCPUs)	Space for memory state file

Configure while VM is stopped:

incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device override REMOTE:VM root size.state=2GiB
incus start REMOTE:VM

Troubleshooting

Symptom	Cause	Fix
Port 8443 not reachable after boot	Boot still in progress or crontab bug	Wait 180s; check scrub_schedule via API
`scrub_schedule` empty	Crontab race condition	`incusos-proxmox --status` auto-heals
`Missing section footer for ICH9LPC` on migration	`limits.cpu` set as integer	Set as range: `limits.cpu=0-1`
`VM agent isn't currently running` after migration	Agent reconnecting	`sleep 4` after migration
`db.sock not found` on OVN config	OVN service not enabled on IncusOS	Enable via `/os/1.0/services/ovn` API
Cross-node ping fails (bridge)	Bridge networks are node-local	Use OVN network instead
`zfs load-key: Raw key too short`	TPM corruption from premature VM stop	Destroy and redeploy VM
Cluster join fails with "pool already exists"	`apply_defaults: true` on joining node	Use `apply_defaults: false` or run 8-command cleanup
OC cannot manage cluster nodes	Nodes deployed with standard ISO	Use OC-provisioned ISO for full integration
`CPUID vnmi` warning during migration	Cosmetic QEMU check	Safe to ignore
"Parent interface 'ens18' not found"	IncusOS names its NIC `mgmt`	Use `parent=mgmt` for UPLINK network
"Invalid target address" on LB backend	Backend needs IP, not instance name	Use instance IP address (e.g., `10.10.10.2`)
OC CLI "Invalid number of arguments"	OC CLI doesn't support `remote:` suffix	Use `operations-center remote switch NAME` first
Container placed on wrong node	Cluster auto-schedules without `--target`	Use `--target NODE` for explicit placement
"Device from profile(s) cannot be modified"	`root` device comes from default profile	Use `incus config device override` instead of `device set`
`incus launch` hangs or times out	Image download to new node takes 2-3 min	Run launches one at a time, not chained with `&&`

52 KiB Raw Blame History

Production Home Lab Guide

Section 0: Architecture Overview

Network Topology

Infrastructure

Inner Cluster Services

External IP Allocation

Decision Rationale

Cross-References

Section 1: Prerequisites

Required Tools

Aether Browser Automation

Aether Golden Image

Proxmox Configuration

Client Certificates

Doctor Check

Section 2: Deploy OC Server

Configuration File

Deploy

Set Up OC CLI Remote

Verify OC

Wait for Updates

Web UI Access

Section 3: Provision Nodes

3.1 Create Provisioning Token

3.2 Create Token Seed (No force_reboot)

3.3 Generate OC-Provisioned ISO (Older Version)

3.4 Node Configuration

3.5 Deploy Nodes (Hybrid Approach)

3.6 Verify Auto-Registration

3.7 Verify Scrub Schedules

Section 4: Form Cluster via Operations Center

4.1 The needs_update Blocker

4.2 Form Cluster

4.3 Fix Remotes After Clustering

4.4 Verify Cluster

4.5 Cluster Resources Created by OC

Section 5: Bridge Networking Baseline

Same-Node Communication

Cross-Node Communication (Fails)

Internet Access

Cleanup

Section 6: OVN Overlay Networking

6.1 Deploy OVN Control Plane

6.2 Enable OVN on All IncusOS Nodes

6.3 Configure Incus OVN Connection

6.4 Assign OVN Chassis Role

6.5 Create UPLINK Physical Network

6.6 Create OVN Network (net-prod)

6.7 Verify Cross-Node OVN Connectivity

6.8 Final Network State

Section 7: Mixed Workloads

7.1 Containers on net-prod

7.2 VMs (Migration-Ready)

7.3 Configure VMs for Live Migration

7.4 Workload Distribution

Section 8: Network Isolation & Security

8.1 Create Isolated Network

8.2 Launch Isolated Containers

8.3 Verify Network Isolation

8.4 Create Network ACL

8.5 Apply and Test ACL

8.6 Network Peering

8.7 Verify Peering

8.8 Remove Peering

Section 9: Load Balancers & Network Forwards

9.1 Create OVN Load Balancer

9.2 Test Load Balancer

9.3 Create Network Forward

9.4 Test Network Forward

9.5 DNS Resolution

Section 10: Live Migration

10.1 Verify Migration Readiness

10.2 Create Heartbeat Service

10.3 Live Migration Round-Trip

10.4 Active I/O During Migration

10.5 Stateful Stop/Restore

Section 11: Cluster Lifecycle

11.1 Evacuation & Restore

11.2 Node Failure Simulation

52 KiB

Raw Blame History