51 KiB

Raw Blame History

Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center dashboard, 3-node cluster with OVN overlay networking, mixed container/VM workloads, live migration, network security, load balancers, and cluster lifecycle management.

All commands and output in this guide are from an actual deployment on 2026-02-22. Tested on Proxmox VE 9.1.5, IncusOS 202602210344, Incus client 6.21, Operations Center v0.3.0.

Section 0: Architecture Overview

Network Topology

                        VLAN 69 (192.168.100.0/22)
                        Gateway: 192.168.100.1
                               │
            ┌──────────────────┼──────────────────────────────────┐
            │                  │                                  │
     ┌──────┴──────┐   ┌──────┴──────┐   ┌───────┴─────┐  ┌─────┴───────┐
     │   lab-oc    │   │ lab-node-01 │   │ lab-node-02 │  │ lab-node-03 │
     │ VMID 910    │   │ VMID 911    │   │ VMID 912    │  │ VMID 913    │
     │ .102.110    │   │ .102.111    │   │ .102.112    │  │ .102.113    │
     │ OC server   │   │ Init node   │   │ Join node   │  │ Join node   │
     │ 2c/4G/50G   │   │ 4c/8G/64G   │   │ 4c/8G/50G   │  │ 4c/8G/50G   │
     └─────────────┘   │             │   │             │  │             │
                       │ ovn-central │   │             │  │             │
                       │ (container) │   │             │  │             │
                       │ NB:6641     │   │             │  │             │
                       │ SB:6642     │   │             │  │             │
                       └─────────────┘   └─────────────┘  └─────────────┘
                              │                │                │
                              └────────────────┼────────────────┘
                                    Geneve tunnels (OVN)
                                               │
                              ┌────────────────┼────────────────┐
                              │                │                │
                       ┌──────┴──────┐  ┌──────┴──────┐  ┌─────┴───────┐
                       │  net-prod   │  │net-isolated │  │   UPLINK    │
                       │ 10.10.10/24 │  │ 10.10.20/24 │  │ .103.200-210│
                       │   OVN L2    │  │   OVN L2    │  │  Physical   │
                       └─────────────┘  └─────────────┘  └─────────────┘

Infrastructure

Component	VMID	IP	Cores	RAM	Disk	Role
lab-oc	910	192.168.102.110/22	2	4 GiB	50G	Operations Center
lab-node-01	911	192.168.102.111/22	4	8 GiB	64G	Cluster init + OVN host
lab-node-02	912	192.168.102.112/22	4	8 GiB	50G	Cluster member
lab-node-03	913	192.168.102.113/22	4	8 GiB	50G	Cluster member

RAM budget: 28 GiB of 64 GiB (44% utilization). Leaves headroom for workloads inside the VMs and other labs on the host.

OVN IP allocation: 192.168.103.200-210 reserved for OVN external addresses (router IPs, load balancer VIPs, network forwards). These must be excluded from your DHCP server's range.

Decision Rationale

Why manual clustering instead of OC provisioning cluster add? OC v0.3.0's provisioning cluster add has a needs update: false blocker that can stall indefinitely. Manual clustering via incus CLI is proven reliable. OC still provides value as a monitoring dashboard.

Why OVN? Bridge networks are node-local — instances on different nodes cannot communicate. OVN provides transparent cross-node L2 overlay with sub-millisecond latency (~0.1-0.8ms), network isolation, ACLs, load balancers, and network forwards.

Why VLAN 69? Isolates lab traffic from the production LAN. All VMs share VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox VM level — IncusOS and workloads are unaware of it.

Cross-References

This guide brings together techniques from the deep-dive guides:

Clustering Guide — cluster formation, migration, lifecycle
Networking Guide — OVN setup, ACLs, peering, LBs
Operations Center Guide — OC provisioning, CLI, web UI
Migration Guide — importing VMs from other hypervisors

Section 1: Prerequisites

Required Tools

Verify all tools are available before starting:

incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1

Minimum versions: Incus client 6.3+ (for remote get-client-certificate fallback, though scripts read ~/.config/incus/client.crt directly), Operations Center v0.3.0+.

Proxmox Configuration

Your incusos/proxmox.yaml should contain:

host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab

The env file at the repository root must export PROXMOX_TOKEN_SECRET. Scripts auto-discover it — no manual sourcing needed.

Client Certificates

Incus client certificates are used for both Incus and OC connections:

# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key

For OC web UI browser access, generate a PKCS#12 bundle:

openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"

Import client.pfx into your browser's certificate store (Firefox: Settings → Privacy & Security → View Certificates → Import).

Doctor Check

Run the environment check to verify everything is in order:

cd incusos
./incusos-proxmox --doctor

Expected output includes tool versions, IncusOS CDN reachability, proxmox.yaml discovery, and Proxmox API connectivity.

Section 2: Deploy Infrastructure

Configuration File

The lab uses incusos/examples/lab-production.yaml:

defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 910

vms:
  - name: lab-oc
    app: operations-center
    apply_defaults: true
    cores: 2
    memory: 4096
    ip: 192.168.102.110/22

  - name: lab-node-01
    app: incus
    apply_defaults: true          # init node: needs storage pool + network
    disk: 64                      # extra space for OVN control plane container
    ip: 192.168.102.111/22

  - name: lab-node-02
    app: incus
    apply_defaults: false         # joining node: cluster join creates pool entry
    ip: 192.168.102.112/22

  - name: lab-node-03
    app: incus
    apply_defaults: false         # joining node: cluster join creates pool entry
    ip: 192.168.102.113/22

Key decisions: node-01 has apply_defaults: true (cluster init needs storage pool and network bridge). Nodes 02 and 03 have apply_defaults: false (the cluster join process creates member-specific entries). node-01 gets 64 GiB disk for the OVN control plane container.

Dry Run

Preview the deployment without making any changes:

./incusos-proxmox --dry-run examples/lab-production.yaml

This shows: ISO download plan, seed generation commands, VM creation parameters, and the full install sequence for each VM.

Deploy

Deploy all 4 VMs:

./incusos-proxmox --yes examples/lab-production.yaml

The deploy takes ~5-8 minutes:

Downloads the latest IncusOS ISO (if not cached)
Generates per-VM seed ISOs with static IP, hostname, certificates
Creates VMs on Proxmox with UEFI, TPM, VirtIO settings
Boots each VM with ISO + seed, monitors installation via blockstat
Detects install completion (876 MiB written, then idle)
Stops VMs, removes install media (ISO + seed)
Starts VMs from disk, waits for port 8443 (up to 180s)
Auto-heals scrub_schedule via IncusOS REST API
Configures incus remotes for each Incus node

Verify Deployment

Check deployment status:

./incusos-proxmox --status examples/lab-production.yaml

Expected output shows each VM with Proxmox state (running), network (static IP reachable), port 8443 (open), and incus remote (configured).

Verify Scrub Schedule

Confirm the crontab bug fix is effective on all Incus nodes:

for node in lab-node-01 lab-node-02 lab-node-03; do
    echo -n "$node scrub_schedule: "
    incus query "$node":/os/1.0/system/storage | python3 -c \
        "import sys,json; print(json.load(sys.stdin).get('config',{}).get('scrub_schedule','EMPTY'))"
done

Expected output — all nodes show 0 4 * * 0:

lab-node-01 scrub_schedule: 0 4 * * 0
lab-node-02 scrub_schedule: 0 4 * * 0
lab-node-03 scrub_schedule: 0 4 * * 0

If any node shows EMPTY, the crontab bug hit. Run:

./incusos-proxmox --status examples/lab-production.yaml

The status check includes automatic scrub_schedule healing.

Section 3: Operations Center Setup

Add OC Remote

operations-center remote add oc-lab https://192.168.102.110:8443 --auth-type tls

Accept the certificate fingerprint when prompted.

Important: The OC CLI does not support the remote: suffix syntax that the Incus CLI uses. Instead, switch to the remote first, then run commands without a remote suffix:

operations-center remote switch oc-lab

Verify OC

operations-center admin os show

Actual output:

+-------------------+----------------------------+
|     PROPERTY      |           VALUE            |
+-------------------+----------------------------+
| hostname          | lab-oc                     |
| os_version        | 202602210344               |
| kernel            | 6.12.13                    |
| architecture      | x86_64                     |
| uptime            | 4390                       |
| addresses         | 192.168.102.110/22 (mgmt)  |
|                   | fd42:...:1 (incusbr0)      |
| storage_disks     | /dev/sda (53.7GB, QEMU)    |
| storage_pools     | local (zfs, /dev/sda4)     |
+-------------------+----------------------------+

Check Application Status

operations-center admin os application list

Actual output:

+--------------------+---------+
|        NAME        | STATUS  |
+--------------------+---------+
| operations-center  | running |
+--------------------+---------+

Check for Updates

operations-center provisioning update list

Shows available IncusOS updates. Updates can be applied via the OC web UI or CLI.

Service Status

operations-center admin os service list

Actual output:

+----------+---------+
|   NAME   | ENABLED |
+----------+---------+
| ovn      | false   |
| syslog   | false   |
| fan      | false   |
| bgp      | false   |
| dns      | false   |
| metricsA | false   |
| metricsB | false   |
+----------+---------+

Web UI Access

Open https://192.168.102.110:8443/ui/ in your browser. You need the PKCS#12 client certificate imported (see Section 1). The web UI provides a dashboard view of the OC server. After adding Incus nodes to OC (optional), the dashboard shows cluster health.

Note: OC deployed with a standard ISO acts as a monitoring dashboard. For full OC node management (provisioning, cluster orchestration), nodes must boot from an OC-provisioned ISO. See Operations Center Guide for the full hybrid workflow.

Section 4: Cluster Formation

4.1 Set Specific IP Addresses

IncusOS nodes default to core.https_address: :8443 (wildcard). Clustering requires specific routable IPs so nodes can address each other.

incus config set lab-node-01: core.https_address 192.168.102.111:8443
incus config set lab-node-02: core.https_address 192.168.102.112:8443
incus config set lab-node-03: core.https_address 192.168.102.113:8443

Verify on each node:

incus config get lab-node-01: core.https_address
incus config get lab-node-02: core.https_address
incus config get lab-node-03: core.https_address

Each should return IP:8443.

4.2 Enable Clustering on Init Node

incus cluster enable lab-node-01: lab-node-01

Note the syntax: TWO arguments — lab-node-01: (remote with trailing colon) and lab-node-01 (member name). This is NOT lab-node-01:lab-node-01.

4.3 Fix Init Node Remote

Enabling clustering regenerates the TLS certificate. The new cert may only have SANs for 127.0.0.1 and ::1, breaking the remote.

incus remote switch local
incus remote remove lab-node-01
incus remote add lab-node-01 https://192.168.102.111:8443 --accept-certificate

Verify:

incus cluster list lab-node-01:

Expected output:

+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|     NAME     |            URL             |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-01  | https://192.168.102.111:8443| database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|              |                            | database         |              |                |             |        |                   |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+

4.4 Join Node-02

Generate a join token on the init node:

incus cluster add lab-node-01:lab-node-02

This outputs a long base64 token. Use it immediately — tokens expire.

Join node-02 to the cluster (automated, non-interactive):

printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-02:

The five prompts answered by printf:

IP address → accept default (node's IP)
Member name → accept default (matches token)
"All existing data is lost" → yes
source for storage pool "local" → local/incus
zfs.pool_name for pool "local" → local/incus

No storage/network cleanup needed — apply_defaults: false means node-02 has no pre-existing Incus storage pool or network.

Fix the remote after join (new cluster cert):

incus remote remove lab-node-02
incus remote add lab-node-02 https://192.168.102.112:8443 --accept-certificate

4.5 Join Node-03

Same procedure:

incus cluster add lab-node-01:lab-node-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:

Fix the remote:

incus remote remove lab-node-03
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate

4.6 Verify Cluster

incus cluster list lab-node-01:

Expected output — 3 nodes, all ONLINE:

+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|     NAME     |            URL             |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-01  | https://192.168.102.111:8443| database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|              |                            | database         |              |                |             |        |                   |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-02  | https://192.168.102.112:8443| database         | x86_64       | default        |             | ONLINE | Fully operational |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-03  | https://192.168.102.113:8443| database         | x86_64       | default        |             | ONLINE | Fully operational |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+

Verify storage pool exists on all members:

incus storage show lab-node-01:local
incus storage show lab-node-01:local --target lab-node-02
incus storage show lab-node-01:local --target lab-node-03

Verify the default network:

incus network list lab-node-01:

Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are node-local. This demonstrates why OVN is needed.

Same-Node Communication

Launch 2 containers on the same node. Important: use --target to force placement — without it, the cluster scheduler may place containers on different nodes automatically:

incus launch images:debian/12 lab-node-01:test-bridge-a --target lab-node-01
incus launch images:debian/12 lab-node-01:test-bridge-b --target lab-node-01

Wait for them to get IPs:

incus list lab-node-01: --columns ns4 --format csv | grep test-bridge

Ping between them:

IP_B=$(incus list lab-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-bridge-a -- ping -c 3 "$IP_B"

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

Cross-Node Communication (Fails)

Launch a container on a different node:

incus launch images:debian/12 lab-node-01:test-bridge-c --target lab-node-02

Wait for IP:

incus list lab-node-01: --columns ns4 --format csv | grep test-bridge

Ping from node-01 to node-02:

IP_C=$(incus list lab-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"

Actual result: 100% packet loss. Bridge networks are node-local — there is no L2 path between incusbr0 on node-01 and incusbr0 on node-02. Each node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are separate L2 domains.

Internet Access

NAT to the internet works from any node:

incus exec lab-node-01:test-bridge-a -- ping -c 3 1.1.1.1

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via the host's management interface.

Cleanup

incus delete lab-node-01:test-bridge-a --force
incus delete lab-node-01:test-bridge-b --force
incus delete lab-node-01:test-bridge-c --force

Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this section, containers on any node can communicate transparently.

6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

incus launch images:debian/12 lab-node-01:ovn-central --target lab-node-01

Install OVN:

incus exec lab-node-01:ovn-central -- apt-get update
incus exec lab-node-01:ovn-central -- apt-get install -y ovn-central ovn-host

Configure OVN to listen on all interfaces:

incus exec lab-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec lab-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0

Add proxy devices to expose NB and SB ports on the host's LAN IP:

incus config device add lab-node-01:ovn-central nb-proxy proxy \
    listen=tcp:192.168.102.111:6641 connect=tcp:127.0.0.1:6641
incus config device add lab-node-01:ovn-central sb-proxy proxy \
    listen=tcp:192.168.102.111:6642 connect=tcp:127.0.0.1:6642

Verify the ports are reachable:

curl -s --connect-timeout 2 telnet://192.168.102.111:6641 || echo "NB port open"
curl -s --connect-timeout 2 telnet://192.168.102.111:6642 || echo "SB port open"

6.2 Enable OVN on All IncusOS Nodes

OVN services are disabled by default on IncusOS. Enable them on every node via the IncusOS REST API. The database field points to the southbound DB (port 6642, not 6641).

Node-01:

incus query lab-node-01:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.111",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

Node-02:

incus query lab-node-02:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.112",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

Node-03:

incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.113",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

Each call should return {} on success.

6.3 Configure Incus OVN Connection

Point Incus to the northbound DB (port 6641):

incus config set lab-node-01: network.ovn.northbound_connection tcp:192.168.102.111:6641

6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the ovn-chassis role:

incus cluster role add lab-node-01:lab-node-01 ovn-chassis
incus cluster role add lab-node-01:lab-node-02 ovn-chassis
incus cluster role add lab-node-01:lab-node-03 ovn-chassis

Verify:

incus cluster list lab-node-01:

The ROLES column should now include ovn-chassis for each member.

6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and the physical LAN. It uses the two-step cluster pattern: per-member --target first, then cluster-wide create.

Per-member configuration (one per node):

Important: IncusOS names its management NIC mgmt, NOT ens18. Using parent=ens18 will fail with "Parent interface 'ens18' not found". Verify with: incus query lab-node-01:/os/1.0/system/network

incus network create lab-node-01:UPLINK --type physical --target lab-node-01 \
    parent=mgmt

incus network create lab-node-01:UPLINK --type physical --target lab-node-02 \
    parent=mgmt

incus network create lab-node-01:UPLINK --type physical --target lab-node-03 \
    parent=mgmt

Cluster-wide create with shared settings:

incus network create lab-node-01:UPLINK --type physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1

6.6 Create OVN Network (net-prod)

incus network create lab-node-01:net-prod --type=ovn network=UPLINK \
    ipv4.address=10.10.10.1/24 \
    ipv4.nat=true \
    ipv6.address=none

Verify:

incus network list lab-node-01:

Should show both incusbr0 (bridge, per-node) and net-prod (ovn, cluster-wide).

6.7 Verify Cross-Node OVN Connectivity

Launch containers on different nodes, attached to net-prod:

incus launch images:debian/12 lab-node-01:test-ovn-a --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:test-ovn-b --network net-prod --target lab-node-02
incus launch images:debian/12 lab-node-01:test-ovn-c --network net-prod --target lab-node-03

Wait for IPs and list:

incus list lab-node-01: --columns nst4 --format csv | grep test-ovn

Cross-node ping (node-01 → node-02):

IP_B=$(incus list lab-node-01:test-ovn-b --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_B"

Actual result: 0% packet loss, ~0.09-0.8ms latency. OVN provides transparent L2 connectivity via Geneve tunnels.

Cross-node ping (node-01 → node-03):

IP_C=$(incus list lab-node-01:test-ovn-c --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_C"

Internet access through OVN:

incus exec lab-node-01:test-ovn-a -- ping -c 3 1.1.1.1

Clean up test containers:

incus delete lab-node-01:test-ovn-a --force
incus delete lab-node-01:test-ovn-b --force
incus delete lab-node-01:test-ovn-c --force

Section 7: Mixed Workloads

Deploy a realistic workload mix: web servers, application containers, and VMs configured for live migration.

7.1 Containers on net-prod

Deploy containers with targeted placement across nodes:

# Web servers
incus launch images:debian/12 lab-node-01:prod-web-01 --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:prod-web-02 --network net-prod --target lab-node-02

# Application container
incus launch images:debian/12 lab-node-01:prod-api-01 --network net-prod --target lab-node-03

Install nginx on the web servers:

incus exec lab-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec lab-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx"

Set distinct content to verify load balancing later:

incus exec lab-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html"
incus exec lab-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html"

Install nginx on the API container:

incus exec lab-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec lab-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html"

7.2 VMs (Migration-Ready)

Deploy VMs with live migration configuration:

incus launch images:debian/12 lab-node-01:prod-db-01 --vm --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:prod-app-01 --vm --network net-prod --target lab-node-02

VMs may take longer to boot than containers (~30-60s for image download + boot). If the VMs show as STOPPED, start them explicitly:

incus start lab-node-01:prod-db-01
incus start lab-node-01:prod-app-01

Wait for the VM agent to become available, then verify:

# Check VM agent is running
incus exec lab-node-01:prod-db-01 -- uname -a
incus exec lab-node-01:prod-app-01 -- uname -a

7.3 Configure VMs for Live Migration

Critical: use limits.cpu as a range (e.g., 0-1), not an integer. Without the range, QEMU sets maxcpus based on the host's CPU count, which varies across nodes and breaks migration with Missing section footer for ICH9LPC.

Stop VMs before configuring migration.stateful:

incus stop lab-node-01:prod-db-01
incus stop lab-node-01:prod-app-01

Configure migration settings:

# prod-db-01
incus config set lab-node-01:prod-db-01 limits.cpu=0-1
incus config set lab-node-01:prod-db-01 migration.stateful=true
incus config device set lab-node-01:prod-db-01 root size.state=2GiB

# prod-app-01
incus config set lab-node-01:prod-app-01 limits.cpu=0-1
incus config set lab-node-01:prod-app-01 migration.stateful=true
incus config device set lab-node-01:prod-app-01 root size.state=2GiB

Start the VMs:

incus start lab-node-01:prod-db-01
incus start lab-node-01:prod-app-01

7.4 Workload Distribution

View the full workload distribution:

incus list lab-node-01: --columns nstL4 --format table

Expected layout:

+--------------+---------+-------------------+-------------+-----------------------+
|     NAME     |  STATE  |       TYPE        |  LOCATION   |         IPV4          |
+--------------+---------+-------------------+-------------+-----------------------+
| ovn-central  | RUNNING | CONTAINER         | lab-node-01 | ...                   |
| prod-web-01  | RUNNING | CONTAINER         | lab-node-01 | 10.10.10.x (net-prod) |
| prod-db-01   | RUNNING | VIRTUAL-MACHINE   | lab-node-01 | 10.10.10.x (net-prod) |
| prod-web-02  | RUNNING | CONTAINER         | lab-node-02 | 10.10.10.x (net-prod) |
| prod-app-01  | RUNNING | VIRTUAL-MACHINE   | lab-node-02 | 10.10.10.x (net-prod) |
| prod-api-01  | RUNNING | CONTAINER         | lab-node-03 | 10.10.10.x (net-prod) |
+--------------+---------+-------------------+-------------+-----------------------+

Section 8: Network Isolation & Security

8.1 Create Isolated Network

incus network create lab-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none

8.2 Launch Isolated Containers

incus launch images:debian/12 lab-node-01:iso-app-01 --network net-isolated --target lab-node-01
incus launch images:debian/12 lab-node-01:iso-app-02 --network net-isolated --target lab-node-02

8.3 Verify Network Isolation

Containers on net-isolated can reach each other:

IP_ISO2=$(incus list lab-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the same OVN network can reach each other across nodes.

But net-prod cannot reach net-isolated:

incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"

Actual result: 100% packet loss. Different OVN networks are fully isolated — separate L2 domains, no routing between them.

8.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

incus network acl create lab-node-01:block-ping
incus network acl rule add lab-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.10.0/24 \
    description="Block ICMP from net-prod subnet"

8.5 Apply and Test ACL

Apply the ACL to net-isolated:

incus network set lab-node-01:net-isolated security.acls=block-ping

Verify ICMP is blocked between net-isolated containers (since they match the source range — adjust the ACL source for targeted blocking):

incus exec lab-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"

Remove the ACL:

incus network unset lab-node-01:net-isolated security.acls

Verify ICMP works again:

incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

8.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can communicate. Peering is bilateral — create a peer on both sides:

# From net-prod's perspective
incus network peer create lab-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create lab-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"

8.7 Verify Peering

Cross-network ping (prod → isolated):

incus exec lab-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops), confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

IP_WEB1=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1"

8.8 Remove Peering

incus network peer delete lab-node-01:net-prod peer-to-isolated
incus network peer delete lab-node-01:net-isolated peer-to-prod

Verify isolation is restored:

incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

incus delete lab-node-01:iso-app-01 --force
incus delete lab-node-01:iso-app-02 --force

Section 9: Load Balancers & Network Forwards

9.1 Create OVN Load Balancer

Create a load balancer with a VIP from the UPLINK range:

incus network load-balancer create lab-node-01:net-prod 192.168.103.200

Add backend servers. Important: backends require the instance's IP address, not its name. Get the IPs first:

WEB1_IP=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
WEB2_IP=$(incus list lab-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1)
echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP"

Add backends using IP addresses:

incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
    web-01 "$WEB1_IP" 80
incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
    web-02 "$WEB2_IP" 80

Add a port mapping:

incus network load-balancer port add lab-node-01:net-prod 192.168.103.200 \
    tcp 80 web-01,web-02

9.2 Test Load Balancer

From your dev machine (must be on the same VLAN or have routing to 192.168.103.0/24):

for i in $(seq 1 6); do
    curl -s http://192.168.103.200
done

Actual output:

Server: prod-web-01
Server: prod-web-01
Server: prod-web-01
Server: prod-web-02
Server: prod-web-02
Server: prod-web-02

OVN uses connection-based hashing (not round-robin). Multiple requests from the same source will typically hit the same backend. Different source ports or connections may hit different backends.

9.3 Create Network Forward

Network forwards expose internal services on LAN IPs. Forward tcp:8080 → prod-api-01:80. Like LB backends, forwards require IP addresses:

API_IP=$(incus list lab-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1)

incus network forward create lab-node-01:net-prod 192.168.103.201
incus network forward port add lab-node-01:net-prod 192.168.103.201 \
    tcp 8080 "$API_IP" 80

9.4 Test Network Forward

curl -s http://192.168.103.201:8080

Actual output: API: prod-api-01

9.5 DNS Resolution

OVN provides per-network DNS. Containers can resolve each other by hostname:

incus exec lab-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus"

Actual output: 10.10.10.3 — OVN DNS resolves instance names within each network.

Section 10: Live Migration

10.1 Verify Migration Readiness

Check that VMs have the required configuration:

for vm in prod-db-01 prod-app-01; do
    echo "=== $vm ==="
    incus config get lab-node-01:$vm limits.cpu
    incus config get lab-node-01:$vm migration.stateful
    incus config device get lab-node-01:$vm root size.state
done

Expected: 0-1, true, 2GiB for each VM.

10.2 Create Heartbeat Service

Create a simple counter in prod-db-01 to verify state continuity across migration:

incus exec lab-node-01:prod-db-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'

Read the counter:

incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Note the value. After migration, the counter should continue from where it left off (live migration preserves running state).

10.3 Live Migration Round-Trip

Check current location:

incus list lab-node-01:prod-db-01 --columns nL --format csv

Migrate node-01 → node-02:

time incus move lab-node-01:prod-db-01 --target lab-node-02

Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect:

sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Counter went from 9 → 25. The heartbeat process was never interrupted — it continued counting during migration.

Migrate node-02 → node-03:

time incus move lab-node-01:prod-db-01 --target lab-node-03
sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Actual result: 7.379s. Counter went to 41.

Migrate node-03 → node-01 (back to origin):

time incus move lab-node-01:prod-db-01 --target lab-node-01
sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter

Actual result: 6.896s. Counter went to 56.

Verify the VM is back on node-01:

incus list lab-node-01:prod-db-01 --columns nL --format csv

10.4 Active I/O During Migration

Start a continuous write inside the VM:

incus exec lab-node-01:prod-db-01 -- bash -c '
    dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 &
    echo "Write started, PID: $!"
'

Migrate while I/O is active:

time incus move lab-node-01:prod-db-01 --target lab-node-02
sleep 4

Verify the file exists and is intact:

incus exec lab-node-01:prod-db-01 -- ls -la /tmp/testfile
incus exec lab-node-01:prod-db-01 -- md5sum /tmp/testfile

Move back:

incus move lab-node-01:prod-db-01 --target lab-node-01
sleep 4

10.5 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly where it was:

# Note the heartbeat counter
incus exec lab-node-01:prod-app-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    echo 42 > /tmp/heartbeat/counter
    cat /tmp/heartbeat/counter
'

Stateful stop:

incus stop lab-node-01:prod-app-01 --stateful

Start (resumes from saved state):

incus start lab-node-01:prod-app-01
sleep 4
incus exec lab-node-01:prod-app-01 -- cat /tmp/heartbeat/counter

Expected: 42 — the file (and entire VM state) is preserved.

If the restore fails (e.g., from a limits.cpu mismatch), discard the saved state:

incus start lab-node-01:prod-app-01 --stateless

Section 11: Cluster Lifecycle

11.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

incus cluster evacuate lab-node-01:lab-node-02 --force

Check workload distribution — nothing on node-02:

incus list lab-node-01: --columns nstL --format table

Actual behavior: VMs with migration.stateful=true are live-migrated (prod-app-01 migrated to lab-node-03). Containers are stopped and moved (prod-web-02 stopped, moved to lab-node-03, then started). The --force flag skips confirmation prompts.

Note: if VMs lack the limits.cpu range fix, use --action stop instead to avoid migration failures:

incus cluster evacuate lab-node-01:lab-node-02 --force --action stop

Verify node-02 shows EVACUATED:

incus cluster list lab-node-01:

Restore node-02 (workloads return):

incus cluster restore lab-node-01:lab-node-02 --force

Verify all workloads are back:

incus list lab-node-01: --columns nstL --format table
incus cluster list lab-node-01:

All nodes should show ONLINE.

11.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat detects the failure in ~40 seconds. After the node is restarted:

The node auto-rejoins the cluster (~60s)
Containers auto-start
VMs that were running resume

Procedure (document only — do not execute while OVN is running unless you can tolerate temporary network disruption):

# Simulate crash: hard-stop via Proxmox API
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/stop" ...

# Wait for heartbeat detection (~40s)
# incus cluster list lab-node-01:
# → lab-node-02 shows OFFLINE

# Restart via Proxmox
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/start" ...

# Wait for auto-rejoin (~60s)
# incus cluster list lab-node-01:
# → lab-node-02 shows ONLINE

11.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

Step 1: Evacuate node-03:

incus cluster evacuate lab-node-01:lab-node-03 --force --action stop

Step 2: Remove from cluster:

printf "yes\n" | incus cluster remove lab-node-01:lab-node-03 --force

Note: incus cluster remove prompts "Are you really sure?" even with --force. The printf pipes yes for automation.

Step 3: Clean up the remote:

incus remote remove lab-node-03

Step 4: Destroy and redeploy the VM. Use incusos-proxmox to destroy just node-03 and redeploy it. The simplest approach: create a single-VM config or use the replacement config pattern:

# Destroy just node-03 via Proxmox API (VMID 913)
# Then redeploy with incusos-proxmox using a config that only defines node-03

Alternatively, if you have a lab-replace.yaml config for the replacement node:

./incusos-proxmox --yes examples/lab-replace.yaml

Step 5: Join the fresh node to the cluster:

# Set specific IP
incus config set lab-node-03: core.https_address 192.168.102.113:8443

# Generate join token
incus cluster add lab-node-01:lab-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:

# Fix remote
incus remote remove lab-node-03
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate

Step 6: Re-enable OVN on the replacement node:

incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.113",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add lab-node-01:lab-node-03 ovn-chassis

Step 7: Verify:

incus cluster list lab-node-01:

All 3 nodes should be ONLINE with ovn-chassis role.

11.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads are unevenly distributed), Incus redistributes VMs:

incus config set lab-node-01: cluster.rebalance.interval=1
incus config set lab-node-01: cluster.rebalance.threshold=10
incus config set lab-node-01: cluster.rebalance.batch=2
incus config set lab-node-01: cluster.rebalance.cooldown=5m

Important: only VMs with migration.stateful=true are rebalanced. Containers are NOT auto-rebalanced.

Monitor rebalancing:

incus list lab-node-01: --columns nstL --format table

Disable rebalancing when done testing:

incus config unset lab-node-01: cluster.rebalance.interval
incus config unset lab-node-01: cluster.rebalance.threshold

Section 12: OC Dashboard

Important: Switch to the OC remote first. The OC CLI does not support remote: suffix syntax:

operations-center remote switch oc-lab

OC Server Information

operations-center admin os show

Actual output:

+-------------------+----------------------------+
|     PROPERTY      |           VALUE            |
+-------------------+----------------------------+
| hostname          | lab-oc                     |
| os_version        | 202602210344               |
| kernel            | 6.12.13                    |
| architecture      | x86_64                     |
| uptime            | 4390                       |
| addresses         | 192.168.102.110/22 (mgmt)  |
|                   | fd42:...:1 (incusbr0)      |
| storage_disks     | /dev/sda (53.7GB, QEMU)    |
| storage_pools     | local (zfs, /dev/sda4)     |
+-------------------+----------------------------+

Application Status

operations-center admin os application list

Actual output:

+--------------------+---------+
|        NAME        | STATUS  |
+--------------------+---------+
| operations-center  | running |
+--------------------+---------+

Service Status

operations-center admin os service list

Actual output:

+----------+---------+
|   NAME   | ENABLED |
+----------+---------+
| ovn      | false   |
| syslog   | false   |
| fan      | false   |
| bgp      | false   |
| dns      | false   |
| metricsA | false   |
| metricsB | false   |
+----------+---------+

Web UI

The OC web UI at https://192.168.102.110:8443/ui/ provides:

Dashboard: server overview with resource utilization
Updates: available IncusOS updates
Provisioning: token management (for OC-provisioned deployments)
System: OC configuration and certificates

Limitation: OC deployed with a standard ISO cannot manage the Incus cluster nodes. The nodes are independent — they were deployed with a standard IncusOS ISO, not an OC-provisioned one. For full OC node management (cluster orchestration, application deployment, monitoring), nodes must boot from an OC-provisioned ISO.

See Operations Center Guide for the full hybrid deployment workflow with OC-provisioned ISOs.

Section 13: Cleanup

Delete All Workloads

# Delete containers
for c in prod-web-01 prod-web-02 prod-api-01; do
    incus delete lab-node-01:$c --force
done

# Delete VMs
for vm in prod-db-01 prod-app-01; do
    incus delete lab-node-01:$vm --force
done

Remove OVN Networks

# Delete OVN networks
incus network delete lab-node-01:net-prod
incus network delete lab-node-01:net-isolated 2>/dev/null || true

# Delete UPLINK
incus network delete lab-node-01:UPLINK

Remove OVN Control Plane

incus delete lab-node-01:ovn-central --force

Disable OVN Services

for node in lab-node-01 lab-node-02 lab-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done

Infrastructure Options

Keep infrastructure (stop VMs, keep on disk for later):

./incusos-proxmox --lab-down examples/lab-production.yaml

Restart later with:

./incusos-proxmox --lab-up examples/lab-production.yaml

Full teardown (destroy all VMs, remove ISOs, remotes, cache):

./incusos-proxmox --cleanup --deep examples/lab-production.yaml

Section 14: Verification Checklist

#	Check	Command	Expected
1	All VMs running	`incusos-proxmox --status examples/lab-production.yaml`	4 VMs running, port 8443 open
2	Scrub schedule healthy	`incus query lab-node-01:/os/1.0/system/storage`	`scrub_schedule: "0 4 * * 0"`
3	OC accessible	`operations-center remote switch oc-lab && operations-center admin os show`	Shows version, uptime
4	Cluster formed	`incus cluster list lab-node-01:`	3 nodes ONLINE
5	Storage pool	`incus storage list lab-node-01:`	`local` pool on all members
6	Bridge isolation	Ping cross-node on incusbr0	100% loss (expected)
7	OVN connectivity	Ping cross-node on net-prod	0% loss
8	Internet via OVN	`ping 1.1.1.1` from OVN container	0% loss
9	Network isolation	Ping net-prod → net-isolated	100% loss (expected)
10	Network peering	Peer + ping cross-network	0% loss, TTL=62
11	Load balancer	`curl http://192.168.103.200`	Backend response
12	Network forward	`curl http://192.168.103.201:8080`	API response
13	DNS resolution	`dig prod-web-02.incus` from container	Resolves to 10.10.10.x
14	VM live migration	`incus move` VM between nodes	State preserved
15	Cluster evacuation	`incus cluster evacuate` + `restore`	Workloads moved and returned
16	Stateful stop/start	`incus stop --stateful` + `start`	VM state preserved

Section 15: Quick Reference

Cluster Command Syntax

Command	Arguments	Notes
`incus cluster enable`	`remote: member-name`	TWO args (space between)
`incus cluster add`	`remote:member-name`	ONE arg (no space)
`incus cluster join`	`init-remote: joining-remote:`	TWO args (space between)
`incus cluster remove`	`remote:member-name --force`	ONE arg; prompts even with `--force`
`incus cluster evacuate`	`remote:member-name`	ONE arg (no space)
`incus cluster restore`	`remote:member-name`	ONE arg (no space)
`incus config set`	`remote: key value`	Remote with trailing colon + space
`incus storage show`	`remote:pool`	ONE arg (no space)
`incus storage show`	`remote:pool --target member`	`--target` for member-specific

OVN Setup Cheat Sheet

# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true

Migration Readiness Checklist

Setting	Value	Why
`limits.cpu`	Range (e.g., `0-1`)	Fixed QEMU topology across hosts
`migration.stateful`	`true`	Enables live migration
`root size.state`	`2GiB` (or `4GiB` for 3-4 vCPUs)	Space for memory state file

Configure while VM is stopped:

incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device set REMOTE:VM root size.state=2GiB
incus start REMOTE:VM

Troubleshooting

Symptom	Cause	Fix
Port 8443 not reachable after boot	Boot still in progress or crontab bug	Wait 180s; check scrub_schedule via API
`scrub_schedule` empty	Crontab race condition	`incusos-proxmox --status` auto-heals
`Missing section footer for ICH9LPC` on migration	`limits.cpu` set as integer	Set as range: `limits.cpu=0-1`
`VM agent isn't currently running` after migration	Agent reconnecting	`sleep 4` after migration
`db.sock not found` on OVN config	OVN service not enabled on IncusOS	Enable via `/os/1.0/services/ovn` API
Cross-node ping fails (bridge)	Bridge networks are node-local	Use OVN network instead
`zfs load-key: Raw key too short`	TPM corruption from premature VM stop	Destroy and redeploy VM
Cluster join fails with "pool already exists"	`apply_defaults: true` on joining node	Use `apply_defaults: false` or run 8-command cleanup
OC cannot manage cluster nodes	Nodes deployed with standard ISO	Use OC-provisioned ISO for full integration
`CPUID vnmi` warning during migration	Cosmetic QEMU check	Safe to ignore
"Parent interface 'ens18' not found"	IncusOS names its NIC `mgmt`	Use `parent=mgmt` for UPLINK network
"Invalid target address" on LB backend	Backend needs IP, not instance name	Use instance IP address (e.g., `10.10.10.2`)
OC CLI "Invalid number of arguments"	OC CLI doesn't support `remote:` suffix	Use `operations-center remote switch NAME` first
Container placed on wrong node	Cluster auto-schedules without `--target`	Use `--target NODE` for explicit placement

51 KiB Raw Blame History

Production Home Lab Guide

Section 0: Architecture Overview

Network Topology

Infrastructure

Decision Rationale

Cross-References

Section 1: Prerequisites

Required Tools

Proxmox Configuration

Client Certificates

Doctor Check

Section 2: Deploy Infrastructure

Configuration File

Dry Run

Deploy

Verify Deployment

Verify Scrub Schedule

Section 3: Operations Center Setup

Add OC Remote

Verify OC

Check Application Status

Check for Updates

Service Status

Web UI Access

Section 4: Cluster Formation

4.1 Set Specific IP Addresses

4.2 Enable Clustering on Init Node

4.3 Fix Init Node Remote

4.4 Join Node-02

4.5 Join Node-03

4.6 Verify Cluster

Section 5: Bridge Networking Baseline

Same-Node Communication

Cross-Node Communication (Fails)

Internet Access

Cleanup

Section 6: OVN Overlay Networking

6.1 Deploy OVN Control Plane

6.2 Enable OVN on All IncusOS Nodes

6.3 Configure Incus OVN Connection

6.4 Assign OVN Chassis Role

6.5 Create UPLINK Physical Network

6.6 Create OVN Network (net-prod)

6.7 Verify Cross-Node OVN Connectivity

Section 7: Mixed Workloads

7.1 Containers on net-prod

7.2 VMs (Migration-Ready)

7.3 Configure VMs for Live Migration

7.4 Workload Distribution

Section 8: Network Isolation & Security

8.1 Create Isolated Network

8.2 Launch Isolated Containers

8.3 Verify Network Isolation

8.4 Create Network ACL

8.5 Apply and Test ACL

8.6 Network Peering

8.7 Verify Peering

8.8 Remove Peering

Section 9: Load Balancers & Network Forwards

9.1 Create OVN Load Balancer

9.2 Test Load Balancer

9.3 Create Network Forward

9.4 Test Network Forward

9.5 DNS Resolution

Section 10: Live Migration

10.1 Verify Migration Readiness

10.2 Create Heartbeat Service

10.3 Live Migration Round-Trip

10.4 Active I/O During Migration

10.5 Stateful Stop/Restore

Section 11: Cluster Lifecycle

11.1 Evacuation & Restore

11.2 Node Failure Simulation

11.3 Node Replacement

11.4 Cluster Rebalancing

Section 12: OC Dashboard

OC Server Information

Application Status

Service Status

51 KiB

Raw Blame History