60 KiB

Raw Blame History

Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center dashboard, OC-managed 3-node cluster with OVN overlay networking, Aether management platform, HAProxy load balancing, AWX lifecycle automation, Prometheus/Grafana/Loki observability stack, live migration, network security, and cluster lifecycle management.

All commands and output in this guide are from actual deployments on Proxmox VE 9.1.5 with IncusOS 202602230420, Incus client 6.21, Operations Center v0.3.0, and Aether v6.4.317.

Section 0: Architecture Overview

Network Topology

flowchart TD
    vlan(("VLAN 69<br/>192.168.100.0/22"))

    subgraph mgmt["Management"]
        oc["oc-server<br/>VMID 920 · .120<br/>Operations Center"]
    end

    subgraph cluster["Incus Cluster"]
        n1["oc-node-01<br/>VMID 400 · .140<br/>Aether · OVN central"]
        n2["oc-node-02<br/>VMID 401 · .141<br/>AWX · Monitoring"]
        n3["oc-node-03<br/>VMID 402 · .142<br/>HAProxy backends"]
    end

    subgraph services["Macvlan Services"]
        aether["Aether · .160"]
        awx["AWX · .161"]
    end

    subgraph ovn["OVN · net-prod 10.10.10.0/24"]
        haproxy["HAProxy HA<br/>.50 · .51"]
        backends["nginx backends<br/>.60 · .61 · .62"]
        monitoring["Observability<br/>.70 + node-exp .71-.73"]
    end

    subgraph uplink["UPLINK · 192.168.103.x"]
        vip[".200 HAProxy VIP"]
        fwd[".201 Grafana / Prometheus"]
    end

    vlan --- mgmt & cluster
    cluster --> services & ovn
    ovn -.-> uplink

    classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
    classDef serviceClass fill:#E69F00,color:#fff,stroke:#b87d00
    classDef ovnClass fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 nodeClass
    class oc mgmtClass
    class aether,awx serviceClass
    class haproxy,backends,monitoring ovnClass
    class vip,fwd,vlan networkClass

    style mgmt fill:#f5e6f0,stroke:#CC79A7
    style cluster fill:#e6f5f0,stroke:#009E73
    style services fill:#fef3e0,stroke:#E69F00
    style ovn fill:#e0f2fe,stroke:#56B4E9
    style uplink fill:#e0eef8,stroke:#0072B2

Infrastructure

Component	VMID	IP	Cores	RAM	Disk	Role
oc-server	920	192.168.102.120/22	2	4 GiB	50G	Operations Center
oc-node-01	400	192.168.102.140/22	4	20 GiB	250G	Cluster init + Aether host
oc-node-02	401	192.168.102.141/22	4	20 GiB	100G	AWX + monitoring host
oc-node-03	402	192.168.102.142/22	4	20 GiB	100G	HAProxy backends

RAM budget: 64 GiB of 94 GiB (68%). Host: i9-13900HK, 94 GiB RAM, 881 GiB ZFS pool. Leaves 30 GiB headroom for other VMs on the host.

Inner Cluster Services

Instance	Network	IP	Node	RAM	Description
ovn-central	incusbr0	DHCP	node-01	512 MiB	OVN NB/SB databases
aether	macvlan mgmt	192.168.102.160	node-01	8 GiB	Management platform
awx	macvlan mgmt	192.168.102.161	node-02	8 GiB	Ansible automation
ha-web-01	net-prod	10.10.10.60	node-01	256 MiB	Nginx backend
ha-web-02	net-prod	10.10.10.61	node-02	256 MiB	Nginx backend
ha-web-03	net-prod	10.10.10.62	node-03	256 MiB	Nginx backend
haproxy-01	net-prod	10.10.10.50	varies	512 MiB	HA load balancer
haproxy-02	net-prod	10.10.10.51	varies	512 MiB	HA load balancer
monitoring	net-prod	10.10.10.70	node-02	2 GiB	Prometheus + Grafana + Loki
node-exp-01	net-prod	10.10.10.71	node-01	128 MiB	Host metrics exporter
node-exp-02	net-prod	10.10.10.72	node-02	128 MiB	Host metrics exporter
node-exp-03	net-prod	10.10.10.73	node-03	128 MiB	Host metrics exporter

External IP Allocation

OVN external addresses from UPLINK range (192.168.103.200-210). Exclude these from your DHCP server's range:

IP	Purpose
192.168.103.200	HAProxy VIP (OVN load balancer → haproxy-01/02)
192.168.103.201	Observability forward (Grafana :3000, Prometheus :9090)

Decision Rationale

Why OC-managed clustering? OC provisioning cluster add is the production path for Incus deployments. It handles cluster formation, update management, and inventory centrally. The deploy scripts (deploy-haproxy, deploy-awx, deploy-observability) are built for the oc-node-* naming and IP scheme.

Why 20 GiB RAM per node? Aether requires 8 GiB. AWX requires 4-8 GiB. Monitoring + HAProxy + backends need ~3 GiB total. Leaves headroom for mixed workloads and live migration.

Why 250 GiB disk for node-01? Aether's golden image is 200 GiB virtual (qcow2). With ZFS thin provisioning only ~11 GiB is used initially, but the pool needs 200 GiB allocatable space.

Why OVN? Bridge networks are node-local — instances on different nodes cannot communicate. OVN provides cross-node L2 overlay with sub-ms latency, network isolation, ACLs, load balancers, and network forwards — essential for HAProxy HA and distributed workloads.

Why VLAN 69? Isolates lab traffic from the production LAN. All VMs share VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox level — IncusOS and workloads are unaware of it.

Cross-References

This guide integrates techniques from the deep-dive guides:

Clustering Guide — manual cluster formation reference
Networking Guide — OVN setup, ACLs, peering
Operations Center Guide — OC provisioning, CLI, web UI
Aether Guide — management platform deployment and API
HAProxy Guide — HA load balancing with Aether
AWX Guide — Ansible lifecycle automation
Observability Guide — Prometheus, Grafana, Loki stack

Section 1: Prerequisites

Required Tools

Verify all tools are available before starting:

incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1

Minimum versions: Incus client 6.3+, Operations Center v0.3.0+.

Aether Browser Automation

Several Aether features (HAProxy management, blueprint deployment) are not in the JWT API — they use session-authenticated routes with CSRF protection. Playwright browser automation is required for Sections 7-9:

node --version         # Node.js 18+
npx playwright --version

Install if missing:

npm install playwright @playwright/mcp
npx playwright install chromium

The Playwright MCP server (configured in .mcp.json) provides browser tools when available. The incusos/helpers/aether-browser script is the standalone alternative.

Aether Golden Image

The Aether golden image must be available locally before Section 7:

ls -la sources/aether-golden-image-v6.tar.gz

This is an Ubuntu Noble (24.04) image with 200 GiB virtual disk (~6.6 GiB compressed). Obtain it from the Aether distribution.

Proxmox Configuration

Your incusos/proxmox.yaml should contain:

host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab

The env file at the repository root must export PROXMOX_TOKEN_SECRET and AETHER_ADMIN_PASSWORD. Scripts auto-discover them.

Client Certificates

Incus client certificates are used for Incus, OC, and Prometheus (metrics scraping) connections:

# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key

Copy certs for OC CLI:

mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

For OC web UI browser access, generate a PKCS#12 bundle:

openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"

Import client.pfx into your browser's certificate store (Firefox: Settings → Privacy & Security → View Certificates → Import).

Doctor Check

Run the environment check to verify everything is in order:

cd incusos
./incusos-proxmox --doctor

Expected output includes tool versions, IncusOS CDN reachability, proxmox.yaml discovery, and Proxmox API connectivity.

Section 2: Deploy OC Server

Configuration File

# incusos/examples/lab-oc-deploy.yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
    ip: 192.168.102.120/22

Deploy

./incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml

Actual output (key lines):

[ok]  VM 'oc-server' created (VMID 920)
[ok]  VM 'oc-server' installed and running at 192.168.102.120

Set Up OC CLI Remote

# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab

Important: The OC CLI does not support the remote: suffix syntax that the Incus CLI uses. Switch to the remote first, then run commands without a remote suffix.

Verify OC

operations-center admin os show

Actual output (uptime will vary):

WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602230420"
  os_version_next: ""
  uptime: 63

Wait for Updates

OC downloads IncusOS update packages from upstream. At least one update must reach ready state before ISOs can be generated:

# Poll until at least one update shows "ready"
operations-center provisioning update list

Actual output (after ~8 minutes; UUIDs are stable across deployments):

UUID	Origin	Channels	Version	Severity	Status
82aefab7-fec7-5122-89fd-8412d3d2174c	linuxcontainers.org	stable	202602200553	none	ready
5d6b1018-e534-5e54-aeb5-c9e6027ab31d	linuxcontainers.org	stable	202602210344	none	ready
c912a390-c38b-5bd9-b46f-ccaeba6da68a	linuxcontainers.org	stable	202602230420	none	ready

The table also includes Upstream Channels and Published At columns (omitted for width). Not all updates may be ready simultaneously — at least one ready is sufficient to proceed.

Web UI Access

Open https://192.168.102.120:8443/ui/ in your browser (with client.pfx imported from Section 1). The web UI provides a dashboard view of the OC server, update status, provisioning tokens, and system configuration.

Section 3: Provision Nodes

3.1 Create Provisioning Token

operations-center provisioning token add --uses 5 --description "Production lab cluster"
operations-center provisioning token list

Actual output (UUID changes every run):

UUID	Uses Remaining	Expire At	Channel	Description
	5	<30 days from now>	stable	Production lab cluster

Save the <token-UUID> — you'll need it for the next steps.

3.2 Create Token Seed (No force_reboot)

Critical: the token seed must NOT include force_reboot. On Proxmox, incusos-proxmox handles the install lifecycle externally (blockstat detection + media removal). force_reboot triggers SysRq-B which causes the crontab bug (~50% failure rate).

# /tmp/oc-preseed.yaml
install:
  version: "1"
  force_install: true

Important: use the structured format with section keys (install:). A flat format (version: "1" at root) maps fields to empty {} and they don't get assigned to any section.

operations-center provisioning token seed add <token-UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"

3.3 Generate OC-Provisioned ISO (Older Version)

Critical discovery: nodes deployed from an ISO matching the latest OC update version are tracked as needs_update: true by OC because the OS was never delivered through OC's update pipeline. The fix: generate the ISO from an older channel so OC can push the real update after deployment.

# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
    --description "Older stable versions for initial provisioning"

# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable

# Generate ISO from the older channel
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable

Actual output:

Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"

The ISO contains IncusOS 202602210344 (one version behind). OC will push the latest (202602230420) after nodes register.

3.4 Node Configuration

# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 400

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false
    disk: 250
    ip: 192.168.102.140/22

  - name: oc-node-02
    app: incus
    apply_defaults: false
    ip: 192.168.102.141/22

  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22

Key decisions:

20 GiB RAM per node: Aether needs 8 GiB, AWX needs 4-8 GiB
250 GiB disk for node-01: hosts Aether's 200 GiB virtual image
100 GiB disk for nodes 02-03: sufficient for AWX, monitoring, HAProxy
apply_defaults: false for all nodes: OC's Terraform handles resource creation during cluster formation

3.5 Deploy Nodes (Hybrid Approach)

The hybrid approach uses incusos-proxmox --iso to combine OC auto-registration (from the boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup.

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml

Actual output (key lines):

[ok]  ISO uploaded: IncusOS-oc.iso
[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
[ok]  Remote 'oc-node-01' added (192.168.102.140)
[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
[ok]  Remote 'oc-node-02' added (192.168.102.141)
[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
[ok]  Remote 'oc-node-03' added (192.168.102.142)
[ok]  All post-deployment checks passed

All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.

3.6 Verify Auto-Registration

Nodes auto-register with OC within ~30 seconds of first boot. The update from 202602210344 to 202602230420 happens automatically:

operations-center provisioning server list

Actual output (key columns; full table includes Type, Channel, Certificate Fingerprint, Public Connection URL, Last Updated, Last Seen):

Name	Connection URL	Status	Update Status
oc-node-01	https://192.168.102.140:8443	ready	up to date
oc-node-02	https://192.168.102.141:8443	ready	up to date
oc-node-03	https://192.168.102.142:8443	ready	up to date
operations-center	https://[::1]:8443	ready	update pending

Key: all 3 nodes show "up to date" because OC delivered the 202602230420 update through its pipeline. This is what unlocks clustering. Nodes may already be up to date by the time the last node finishes deploying — the update gets pushed while incusos-proxmox deploys subsequent nodes sequentially.

3.7 Verify Scrub Schedules

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${node}:/os/1.0/system/storage | python3 -c \
        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done

Actual output:

oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0

All healthy. No crontab bug (force_reboot was not used).

Section 4: Form Cluster via Operations Center

4.1 The needs_update Blocker

OC requires all nodes to show needs_update: false before clustering. Nodes deployed from an ISO matching the latest version are tracked as needs_update: true because the OS was never delivered through OC's update pipeline. The needs_update flag is server-side computed and cannot be overridden via REST API PUT.

Solution: deploy from an older ISO version (Section 3.3). OC then pushes the real update to nodes through its pipeline, clearing the flag.

4.2 Form Cluster

Important: if the client certificate was already injected via SEED_DATA, use an empty application seed config to avoid "Certificate already in trust store" Terraform errors:

echo '{}' > /tmp/oc-app-config.yaml

operations-center provisioning cluster add oc-cluster \
    https://192.168.102.140:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC orchestrates the full cluster formation:

Sets core.https_address to each node's specific IP
Enables clustering on oc-node-01
Joins oc-node-02 and oc-node-03
Creates storage pool (local), networks (incusbr0, meshbr0)
Runs Terraform/OpenTofu for post-cluster configuration

4.3 Fix Remotes After Clustering

Clustering regenerates TLS certificates. Re-add the remotes:

incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate

4.4 Verify Cluster

incus cluster list oc-node-01:

Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):

NAME	URL	ROLES	ARCHITECTURE	STATUS	MESSAGE
oc-node-01	https://192.168.102.140:8443	database-leader, database	x86_64	ONLINE	Fully operational
oc-node-02	https://192.168.102.141:8443	database	x86_64	ONLINE	Fully operational
oc-node-03	https://192.168.102.142:8443	database	x86_64	ONLINE	Fully operational

All 3 nodes ONLINE and Fully operational. The ovn-chassis role is added later in Section 6.4.

4.5 Cluster Resources Created by OC

incus storage list oc-node-01:
incus network list oc-node-01:

Actual output (incusbr0 subnet varies per deployment):

NAME	DRIVER	DESCRIPTION	USED BY	STATE
local	zfs	Local storage pool (on system drive)	8	CREATED

NAME	TYPE	MANAGED	IPV4	DESCRIPTION	USED BY
incusbr0	bridge	YES	10.x.x.1/24	Local network bridge (NAT)	1
meshbr0	bridge	YES	none	Internal mesh network bridge	1

OC creates: local storage pool (ZFS), incusbr0 bridge (NAT), and meshbr0 (OC-specific mesh network for inter-node communication). The table also includes IPv6 and STATE columns.

Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are node-local. This demonstrates why OVN is needed.

Same-Node Communication

Launch 2 containers on the same node. Important: use --target to force placement — without it, the cluster scheduler may place containers on different nodes automatically.

Important: launch containers one at a time, not chained with &&. The first launch on a fresh cluster downloads the image (~1 GB), which takes 2-3 minutes. Subsequent launches on the same node use the cached image and are instant. Launches targeting a different node trigger another image transfer to that node.

incus launch images:debian/12 oc-node-01:test-bridge-a --target oc-node-01
incus launch images:debian/12 oc-node-01:test-bridge-b --target oc-node-01

Wait for them to get IPs:

incus list oc-node-01: --columns ns4 --format csv | grep test-bridge

Ping between them:

IP_B=$(incus list oc-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 "$IP_B"

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

Cross-Node Communication (Fails)

Launch a container on a different node:

incus launch images:debian/12 oc-node-01:test-bridge-c --target oc-node-02

Wait for IP:

incus list oc-node-01: --columns ns4 --format csv | grep test-bridge

Ping from node-01 to node-02:

IP_C=$(incus list oc-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"

Actual result: 100% packet loss. Bridge networks are node-local — there is no L2 path between incusbr0 on node-01 and incusbr0 on node-02. Each node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are separate L2 domains.

Internet Access

NAT to the internet works from any node:

incus exec oc-node-01:test-bridge-a -- ping -c 3 1.1.1.1

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via the host's management interface.

Cleanup

incus delete oc-node-01:test-bridge-a --force
incus delete oc-node-01:test-bridge-b --force
incus delete oc-node-01:test-bridge-c --force

Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this section, containers on any node can communicate transparently.

6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01

Install OVN:

incus exec oc-node-01:ovn-central -- bash -c \
    "apt-get update -qq && apt-get install -y -qq ovn-central"

Configure OVN to listen on all interfaces:

incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0

Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:

incus config device add oc-node-01:ovn-central \
    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642

6.2 Enable OVN on All IncusOS Nodes

Enable OVN services via the IncusOS REST API (/os/1.0/services/ovn). The database field is the southbound DB (port 6642), not northbound.

for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
        \"config\": {
            \"database\": \"tcp:192.168.102.140:6642\",
            \"enabled\": true,
            \"tunnel_address\": \"${node_ip}\",
            \"tunnel_protocol\": \"geneve\"
        },
        \"state\": {}
    }"
done

Each call should return {} on success.

6.3 Configure Incus OVN Connection

Point Incus to the northbound DB (port 6641):

incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641

6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the ovn-chassis role:

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role add oc-node-01:${node} ovn-chassis
done

Verify:

incus cluster list oc-node-01:

The ROLES column should now include ovn-chassis for each member.

6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and the physical LAN. It uses the two-step cluster pattern: per-member --target first, then cluster-wide create.

Important: IncusOS names its management NIC mgmt, NOT ens18. Using parent=ens18 will fail with "Parent interface 'ens18' not found".

# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done

# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1

6.6 Create OVN Network (net-prod)

incus network create oc-node-01:net-prod --type=ovn \
    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true

Actual output:

Network net-prod created

net-prod is assigned external IP 192.168.103.200 from the UPLINK range.

6.7 Verify Cross-Node OVN Connectivity

incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3

Actual output:

64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms

Sub-millisecond cross-node latency via Geneve tunnels. Clean up test containers after verification:

incus delete oc-node-01:test-1 oc-node-01:test-2 --force

6.8 Final Network State

incus network list oc-node-01:

Actual output:

NAME	TYPE	MANAGED	IPV4	DESCRIPTION	USED BY
UPLINK	physical	YES			1
incusbr0	bridge	YES	10.x.x.1/24	Local network bridge (NAT)	2
meshbr0	bridge	YES	none	Internal mesh network bridge	1
net-prod	ovn	YES	10.10.10.1/24		0

The incusbr0 subnet is randomly assigned per deployment. The USED BY count for net-prod is 0 at this point (test containers deleted); it increases as workloads are added in subsequent sections.

Section 7: Deploy Aether

Aether is the management platform for the Incus cluster. It provides instance deployment via blueprints, HAProxy load balancing, AWX automation integration, firewall rule management, and RBAC — all through a web UI and partial REST API.

7.1 Import Golden Image

The golden image is a Ubuntu Noble (24.04) image with the Aether application pre-installed. Import it to the cluster:

incus image import sources/aether-golden-image-v6.tar.gz \
    --alias aether-golden-image-v6 oc-node-01:

This takes ~2 minutes. The image is ~6.6 GiB compressed with a 200 GiB virtual disk.

7.2 Create and Configure VM

Use incus init (not launch) — the VM needs NIC configuration before first boot:

incus init oc-node-01:aether-golden-image-v6 oc-node-01:aether --vm \
    --target oc-node-01 --config limits.memory=8GiB -d root,size=200GiB

Important: node-01 must have a 250 GiB disk (configured in Section 3). The 200 GiB virtual disk is thin-provisioned (~11 GiB actual data), but the storage pool needs 200 GiB of allocatable space.

Switch the NIC from bridge to macvlan for direct VLAN access:

incus config device remove oc-node-01:aether eth0 2>/dev/null || true
incus config device add oc-node-01:aether eth0 nic nictype=macvlan parent=mgmt

7.3 Start and Run Post-Deploy

incus start oc-node-01:aether

Wait for the VM agent (~30-60s), then run the post-deploy script. This configures static networking, regenerates SSH keys, initializes the database, and starts the Aether service:

# Wait for agent
sleep 30

incus exec oc-node-01:aether -- /home/ffsdn/post_deploy.sh \
    192.168.102.160/22 192.168.100.1 192.168.100.1

Expected output (7 configuration steps):

Step 1: Configuring network...
Step 2: Applying network configuration...
Step 3: Regenerating SSH host keys...
Step 4: Waiting for FFSDN to create PostgreSQL user...
Step 5: Transferring database ownership...
Step 6: Transferring table and sequence ownership...
Step 7: Restarting FFSDN service...
Post-deploy completed successfully.

The script deletes itself after successful execution.

7.4 Set Admin Password

Access the Aether web UI at https://192.168.102.160:8443. The initial setup prompts for an admin password. Set it to the value in your env file (AETHER_ADMIN_PASSWORD).

7.5 Connect Incus Cluster

Generate a trust token on the Incus cluster:

incus config trust add oc-node-01:AETHER

In the Aether web UI:

Navigate to Manage INCUS Clusters (/incus-infra)
Click Add/Edit/Delete INCUS clusters from/to AETHER
Fill in the form:
- Cluster Name: oc-lab-cluster
- URL: https://192.168.102.141:8443
- Trust Token: paste the token from above
Click Add Cluster

Note: use any cluster member's IP as the URL. Aether connects to the cluster through this endpoint and discovers all members automatically.

After adding, select the cluster from the dropdown on the INCUS Infrastructure Management page. You should see all 3 nodes, their status, memory usage, and OVN roles.

7.6 Connect Operations Center

In the Aether web UI:

Navigate to Operations Center (/operationcenter)
Click + Add Operations Center
Fill in the form:
- Name: oc-lab
- URL: https://192.168.102.120:8443
- Certificate (PEM): contents of ~/.config/incus/client.crt
- Private Key (PEM): contents of ~/.config/incus/client.key
Click Test Connection — expect: Connected! API: 1.0 (devel)
Click Save

7.7 Verify

# Get a JWT token and query clusters
curl -sk https://192.168.102.160:8443/api/clusters \
    -H "Authorization: Bearer $(curl -sk -X POST \
    https://192.168.102.160:8443/api/auth/token \
    -H 'Content-Type: application/json' \
    -d '{"username":"admin","password":"'"$(grep AETHER_ADMIN_PASSWORD env \
        | cut -d= -f2 | tr -d \"\')"'"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")" \
    | python3 -m json.tool

Expected: JSON array containing oc-lab-cluster with cluster members, ID, and connection status.

Section 8: HAProxy HA Load Balancing

Deploy an HA load balancer pair with test backends using the deploy-haproxy script. This automates Aether's HAProxy management features: image build, infrastructure deployment, and service creation.

8.1 Prerequisites Check

cd incusos
./deploy-haproxy --doctor

Expected output:

[step]    Checking prerequisites
[ok]      incus CLI: 6.21
[ok]      curl: 8.x.x
[ok]      python3: 3.x.x
[ok]      Aether password: configured
[ok]      Incus remote 'oc-node-01': reachable
[ok]      Aether: reachable at https://192.168.102.160:8443 (HTTP 200)
[info]    No HAProxy infrastructure on cluster 52 (--deploy to create)
[ok]      All prerequisites satisfied

8.2 Preview

Preview the full deployment without executing:

./deploy-haproxy --deploy --dry-run

This shows all actions that would be taken: backend creation, image build, infrastructure deployment, and service configuration.

8.3 Deploy

./deploy-haproxy --deploy

The script runs 5 phases:

Deploy test backends: 3 nginx containers (nginx-lb-01/02/03) on net-prod at 10.10.10.60-62, each serving a unique HTML page
Build HAProxy image: builds a HAProxy container image via Aether's session API (5-10 minutes on first run, cached afterwards)
Deploy infrastructure: creates the HA pair (ffsdn-haproxy-52-01/02) on net-prod at 10.10.10.50/51, with an OVN load balancer VIP at 192.168.103.200
Create service: configures the web-test HTTP service with health checks and round-robin balancing across the 3 backends
Verify: tests VIP connectivity and traffic distribution

Full deployment takes 10-15 minutes (dominated by image build on first run).

8.4 Verify Load Balancing

From your dev machine (must have routing to 192.168.103.0/24):

for i in $(seq 1 6); do curl -s http://192.168.103.200; echo; done

Expected output shows responses from the 3 backends:

<html><body>
<h1>Backend: nginx-lb-01</h1>
<p>IP: 10.10.10.60</p>
</body></html>

Traffic distribution across multiple backends confirms HA load balancing is working. OVN uses connection-based hashing — sequential requests from the same source may hit the same backend.

8.5 Health and Status

./deploy-haproxy --status

Shows HAProxy container health, backend status, VIP connectivity, and Aether service health stats including per-backend UP/DOWN status and current session count.

8.6 ACL Notes

Aether creates per-container ACLs with default-deny rules. The deploy script configures services to work within this model. If external access to the VIP doesn't work, check the HAProxy containers' ACLs:

incus network acl list oc-node-01:

Look for ACLs named 52-ffsdn-haproxy-52-01-aether-acl and similar. The script's service creation should have added the necessary ingress rules for backend traffic.

Section 9: AWX Lifecycle Automation

Deploy AWX (the open-source Ansible Tower) on a Debian 12 VM with K3s, configure lifecycle playbooks, and register with Aether for automated post-deploy and decommission hooks.

9.1 Prerequisites Check

./deploy-awx --doctor

9.2 Deploy

./deploy-awx --deploy

The script runs 6 phases:

Create VM: Debian 12 with 4 vCPU, 8 GiB RAM, 40 GiB disk on oc-node-02 with macvlan NIC
Configure network: static IP 192.168.102.161/22
Install K3s: lightweight Kubernetes (--write-kubeconfig-mode 644)
Deploy AWX Operator: from kustomize manifests in awx-manifests/
Deploy AWX instance: web UI on NodePort 30080, task engine, PostgreSQL
Verify: API ping, admin password retrieval

Full deployment takes 15-20 minutes. AWX is accessible via HTTP (not HTTPS) at port 30080 — Traefik's ingress returns 404 for IP-based access.

9.3 Configure

Set up the AWX project, inventory, credentials, and job templates:

./deploy-awx --configure

This creates:

Project: incus-contrib (manual/local_path — playbooks are pushed directly to the AWX task pod since the private Git repo is not reachable from the Execution Environment container)
Inventory: incus-instances
Credential: incus-instances (machine type — required by AWX templates, though the playbooks use Incus API not SSH)
Templates: post-deploy and decommission

The Incus client cert/key are pushed to the AWX task pod at /var/lib/awx/projects/incus-contrib/. During job execution, the EE container mounts this as /runner/project/ — the playbooks reference certs at that path.

9.4 Register with Aether

./deploy-awx --join-aether

This creates a Personal Access Token (PAT) in AWX and prints the manual steps to complete Aether integration. In the Aether web UI:

Navigate to Settings → Ansible Automation (/awx-endpoints)
Add Endpoint:
- Name: lab-awx
- URL: http://192.168.102.161:30080
- Token: the PAT printed by the script
- Verify SSL: unchecked
Configure cluster AWX binding (from the cluster settings):
- Post-deploy template ID: printed by the script
- Decommission template ID: printed by the script
- Job timeout: 600 seconds

9.5 How the Lifecycle Hooks Work

When Aether creates an instance, it triggers the AWX post-deploy template with ffsdn_* variables (instance name, IP, cluster, deployer, image info). The playbook:

Waits for the instance to reach Running state
Pushes a setup script via Incus file API
Executes it via Incus exec API (sets hostname, installs base packages, configures unattended upgrades, writes deployment metadata)
Verifies success

When Aether deletes an instance, it triggers the decommission template. The playbook gracefully stops application services and records the event in a ledger.

Important: post-deploy failures cause Aether to auto-rollback (delete the instance). Decommission failures do NOT block deletion.

9.6 Verify

# AWX API health
curl -s http://192.168.102.161:30080/api/v2/ping/ | python3 -m json.tool

# Full status
./deploy-awx --status

The status output shows VM health, K3s state, pod status, admin password, AWX version, and Aether connectivity.

Section 10: Observability Stack

Deploy Prometheus, Grafana, Loki, Promtail, and node-exporters for full cluster observability. The stack scrapes Incus metrics (with TLS client certs), HAProxy stats, and host-level metrics.

10.1 Deploy

./deploy-observability --deploy

The script runs 9 phases:

Monitoring container: Debian 12, 2 GiB RAM, 20 GiB disk on oc-node-02 at 10.10.10.70
Prometheus: scrapes Incus nodes (:8443/1.0/metrics with client certs), HAProxy stats (:8404), and node-exporters (:9100)
Loki + Promtail: log aggregation from systemd journal and /var/log
Grafana: dashboards with Prometheus and Loki datasources
Node-exporters: 3 Alpine containers (node-exp-01/02/03) with host filesystem bind-mounts for full host metrics
ACL: monitoring-allow ACL on all observability containers
HAProxy exporter: prometheus exporter route added to HAProxy containers, plus monitoring ACL rules for scrape access on port 8404
Network forward: 192.168.103.201 → Grafana (:3000) and Prometheus (:9090)
Dashboards: pre-built JSON dashboards uploaded to Grafana

Full deployment takes ~5 minutes.

10.2 Access Dashboards

Grafana: http://192.168.103.201:3000 (admin / admin)
Prometheus: http://192.168.103.201:9090

Pre-loaded dashboards:

Dashboard	Description
Incus Cluster Overview	Instances, memory, CPU across all 3 nodes
HAProxy Traffic	Request rate, backend health, error rates
Host Resources	CPU, memory, disk, network per node

10.3 Verify

./deploy-observability --status

Shows container health, service status (prometheus, grafana-server, loki, promtail), Prometheus target scrape results, Grafana API health, Loki ingestion status, and network forward connectivity.

For detailed diagnostics:

./deploy-observability --doctor

This checks container networking (gateway, node-exporter, HAProxy reachability), Prometheus scrape errors, service health with log output on failure, and disk space.

Section 11: Network Isolation & Security

11.1 Create Isolated Network

incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none

11.2 Launch Isolated Containers

incus launch images:debian/12 oc-node-01:iso-app-01 --network net-isolated --target oc-node-01
incus launch images:debian/12 oc-node-01:iso-app-02 --network net-isolated --target oc-node-02

11.3 Verify Network Isolation

Containers on net-isolated can reach each other:

IP_ISO2=$(incus list oc-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the same OVN network can reach each other across nodes.

But net-prod cannot reach net-isolated:

incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2"

Actual result: 100% packet loss. Different OVN networks are fully isolated — separate L2 domains, no routing between them.

11.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

incus network acl create oc-node-01:block-ping
incus network acl rule add oc-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.20.0/24 \
    description="Block ICMP within net-isolated"

11.5 Apply and Test ACL

Apply the ACL to net-isolated:

incus network set oc-node-01:net-isolated security.acls=block-ping

Verify ICMP is blocked:

incus exec oc-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"

Remove the ACL:

incus network unset oc-node-01:net-isolated security.acls

Verify ICMP works again:

incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"

11.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can communicate. Peering is bilateral — create a peer on both sides:

# From net-prod's perspective
incus network peer create oc-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create oc-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"

11.7 Verify Peering

Cross-network ping (prod → isolated):

incus exec oc-node-01:nginx-lb-01 -- ping -c 3 "$IP_ISO2"

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops), confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

NLB1_IP=$(incus list oc-node-01:nginx-lb-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$NLB1_IP"

11.8 Remove Peering

incus network peer delete oc-node-01:net-prod peer-to-isolated
incus network peer delete oc-node-01:net-isolated peer-to-prod

Verify isolation is restored:

incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2"

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

incus delete oc-node-01:iso-app-01 --force
incus delete oc-node-01:iso-app-02 --force

Section 12: Live Migration

12.1 Deploy Test VM

Deploy a temporary VM on net-prod for migration testing. Live migration requires VMs (containers use a different migration path).

incus launch images:debian/12 oc-node-01:migration-test --vm \
    --network net-prod --target oc-node-01

Wait for boot (~30-60s), then configure for stateful migration:

incus stop oc-node-01:migration-test
incus config set oc-node-01:migration-test limits.cpu=0-1
incus config set oc-node-01:migration-test migration.stateful=true
incus config device override oc-node-01:migration-test root size.state=2GiB
incus start oc-node-01:migration-test

Critical: limits.cpu must be a range (e.g., 0-1), not an integer. Without the range, QEMU sets maxcpus based on the host's CPU count, which varies across nodes and breaks migration with Missing section footer for ICH9LPC.

Important: use device override (not device set) because the root device comes from the default profile.

12.2 Create Heartbeat Service

Create a simple counter to verify state continuity across migration:

incus exec oc-node-01:migration-test -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'

Read the counter:

incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

Note the value. After migration, the counter should continue from where it left off (live migration preserves running state).

12.3 Live Migration Round-Trip

Check current location:

incus list oc-node-01:migration-test --columns nL --format csv

Migrate node-01 → node-02:

time incus move oc-node-01:migration-test --target oc-node-02

Actual result: ~7s (~140 MB/s). Wait for the VM agent to reconnect:

sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

The heartbeat counter continues incrementing — the process was never interrupted during migration.

Migrate node-02 → node-03:

time incus move oc-node-01:migration-test --target oc-node-03
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

Migrate node-03 → node-01 (back to origin):

time incus move oc-node-01:migration-test --target oc-node-01
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

Verify the VM is back on node-01:

incus list oc-node-01:migration-test --columns nL --format csv

12.4 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly where it was:

# Note the heartbeat counter
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

Stateful stop:

incus stop oc-node-01:migration-test --stateful

Start (resumes from saved state):

incus start oc-node-01:migration-test
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter

The counter resumes from the saved value — the entire VM state is preserved across stateful stop/start cycles.

If the restore fails (e.g., from a limits.cpu mismatch), discard the saved state:

incus start oc-node-01:migration-test --stateless

12.5 Clean Up Test VM

incus delete oc-node-01:migration-test --force

Section 13: Cluster Lifecycle

13.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

incus cluster evacuate oc-node-01:oc-node-02 --force

Check workload distribution — nothing on node-02:

incus list oc-node-01: --columns nstL --format table

Actual behavior: containers are stopped, moved to other nodes, then restarted. The --force flag skips confirmation prompts.

Verify node-02 shows EVACUATED:

incus cluster list oc-node-01:

Restore node-02 (workloads return):

incus cluster restore oc-node-01:oc-node-02 --force

Verify all workloads are back:

incus list oc-node-01: --columns nstL --format table
incus cluster list oc-node-01:

All nodes should show ONLINE.

13.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat detects the failure in ~40 seconds. After the node is restarted:

The node auto-rejoins the cluster (~60s)
Containers auto-start
VMs that were running resume

Procedure (document only — do not execute while OVN is running unless you can tolerate temporary network disruption):

# Simulate crash: hard-stop via Proxmox API
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/stop

# Wait for heartbeat detection (~40s)
# incus cluster list oc-node-01:
# → oc-node-02 shows OFFLINE

# Restart via Proxmox
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/start

# Wait for auto-rejoin (~60s)
# incus cluster list oc-node-01:
# → oc-node-02 shows ONLINE

13.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

Step 1: Evacuate node-03:

incus cluster evacuate oc-node-01:oc-node-03 --force --action stop

Step 2: Remove from cluster:

printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

Note: incus cluster remove prompts "Are you really sure?" even with --force. The printf pipes yes for automation.

Step 3: Clean up the remote:

incus remote remove oc-node-03

Step 4: Destroy and redeploy the VM. Use incusos-proxmox to destroy just node-03 (VMID 402) and redeploy it. Create a single-VM config:

# /tmp/lab-replace-node03.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 402

vms:
  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22

./incusos-proxmox --cleanup --yes /tmp/lab-replace-node03.yaml
./incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/lab-replace-node03.yaml

Step 5: Join the fresh node to the cluster:

# Set specific IP
incus config set oc-node-03: core.https_address 192.168.102.142:8443

# Generate join token
incus cluster add oc-node-01:oc-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:

# Fix remote
incus remote remove oc-node-03
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate

Step 6: Re-enable OVN on the replacement node:

incus query oc-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.140:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.142",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add oc-node-01:oc-node-03 ovn-chassis

Step 7: Verify:

incus cluster list oc-node-01:

All 3 nodes should be ONLINE with ovn-chassis role.

13.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads are unevenly distributed), Incus redistributes VMs:

incus config set oc-node-01: cluster.rebalance.interval=1
incus config set oc-node-01: cluster.rebalance.threshold=10
incus config set oc-node-01: cluster.rebalance.batch=2
incus config set oc-node-01: cluster.rebalance.cooldown=5m

Important: only VMs with migration.stateful=true are rebalanced. Containers are NOT auto-rebalanced.

Monitor rebalancing:

incus list oc-node-01: --columns nstL --format table

Disable rebalancing when done testing:

incus config unset oc-node-01: cluster.rebalance.interval
incus config unset oc-node-01: cluster.rebalance.threshold

Section 14: OC Dashboard

Important: Switch to the OC remote first. The OC CLI does not support remote: suffix syntax:

operations-center remote switch oc-lab

OC Server Information

operations-center admin os show

Actual output (version and uptime will vary):

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602240349"
  os_version_next: ""
  uptime: 3600

Provisioning Status

OC manages all 3 cluster nodes. Verify they're registered and up to date:

operations-center provisioning server list

All nodes should show ready status and up to date update status.

Web UI

The OC web UI at https://192.168.102.120:8443/ui/ provides:

Dashboard: server overview with resource utilization
Updates: available IncusOS updates and delivery status
Provisioning: token management, server list, cluster formation
System: OC configuration and certificates

Because the nodes were deployed from an OC-provisioned ISO (Section 3), OC has full visibility and management of the cluster — including update delivery, server inventory, and cluster formation.

Section 15: Cleanup

Tear Down Services (Reverse Order)

Tear down deployed services in reverse order. Each script includes an interactive confirmation prompt:

cd incusos

# 1. Observability stack
./deploy-observability --cleanup

# 2. AWX VM
./deploy-awx --cleanup

# 3. HAProxy infrastructure + backends
./deploy-haproxy --cleanup

Delete Aether VM

incus stop oc-node-01:aether --force
incus delete oc-node-01:aether --force
incus image delete oc-node-01:aether-golden-image-v6 2>/dev/null || true

Remove Isolated Network (if created)

incus network delete oc-node-01:net-isolated 2>/dev/null || true
incus network acl delete oc-node-01:block-ping 2>/dev/null || true

Remove OVN Networks

incus network delete oc-node-01:net-prod
incus network delete oc-node-01:UPLINK

Remove OVN Control Plane

incus delete oc-node-01:ovn-central --force

Disable OVN Services

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done

Infrastructure Options

Keep infrastructure (stop VMs, keep on disk for later):

./incusos-proxmox --lab-down examples/lab-oc-nodes.yaml
./incusos-proxmox --lab-down examples/lab-oc-deploy.yaml

Restart later with:

./incusos-proxmox --lab-up examples/lab-oc-deploy.yaml
./incusos-proxmox --lab-up examples/lab-oc-nodes.yaml

Full teardown (destroy all VMs, remove ISOs, remotes, cache):

./incusos-proxmox --cleanup-all --deep --yes

Section 16: Verification Checklist

#	Check	Command	Expected
1	All VMs running	`incusos-proxmox --status examples/lab-oc-nodes.yaml`	4 VMs running, port 8443 open
2	Scrub schedule	`incus query oc-node-01:/os/1.0/system/storage`	`scrub_schedule: "0 4 * * 0"`
3	OC accessible	`operations-center admin os show`	Shows version, uptime
4	Cluster formed	`incus cluster list oc-node-01:`	3 nodes ONLINE
5	OVN connectivity	Ping cross-node on net-prod	0% loss
6	Aether web UI	`https://192.168.102.160:8443`	Login page loads
7	Aether API	`GET /api/clusters` with JWT	Returns cluster data
8	HAProxy VIP	`curl http://192.168.103.200`	Backend response
9	HAProxy HA	`./deploy-haproxy --status`	Both containers running
10	AWX API	`curl http://192.168.102.161:30080/api/v2/ping/`	JSON with version
11	AWX templates	`./deploy-awx --status`	Pods running, templates configured
12	Grafana	`http://192.168.103.201:3000`	Dashboard loads
13	Prometheus targets	`./deploy-observability --status`	All targets UP
14	Network isolation	Ping net-prod → net-isolated	100% loss (expected)
15	Network peering	Peer + ping cross-network	0% loss, TTL=62
16	VM live migration	`incus move` VM between nodes	State preserved
17	Cluster evacuation	`incus cluster evacuate` + `restore`	Workloads moved and returned
18	Stateful stop/start	`incus stop --stateful` + `start`	VM state preserved
19	DNS resolution	`dig hostname.incus` from container	Resolves to 10.10.10.x

Section 17: Quick Reference

Service Endpoints

Service	URL	Auth
Aether	`https://192.168.102.160:8443`	admin / (env file)
AWX	`http://192.168.102.161:30080`	admin / (from K8s secret)
Grafana	`http://192.168.103.201:3000`	admin / admin
Prometheus	`http://192.168.103.201:9090`	none
HAProxy VIP	`http://192.168.103.200`	none
OC Web UI	`https://192.168.102.120:8443/ui/`	client certificate

Deploy Script Commands

Script	Action	Description
`deploy-haproxy --doctor`	Prerequisites	Check tools, Aether, cluster
`deploy-haproxy --deploy`	Full deploy	Backends + image + infra + service
`deploy-haproxy --status`	Health check	Container + VIP + backend status
`deploy-haproxy --cleanup`	Tear down	Remove all HAProxy resources
`deploy-awx --deploy`	Full deploy	VM + K3s + AWX operator + instance
`deploy-awx --configure`	Configure	Project + inventory + templates
`deploy-awx --join-aether`	Register	Create PAT, print Aether steps
`deploy-awx --status`	Health check	VM + K3s + pods + API
`deploy-awx --heal`	Fix issues	Restart pods, sync project
`deploy-awx --cleanup`	Tear down	Destroy AWX VM
`deploy-observability --deploy`	Full deploy	All 9 phases
`deploy-observability --status`	Health check	Services + targets + forward
`deploy-observability --doctor`	Diagnose	Network + scrape + disk
`deploy-observability --cleanup`	Tear down	Remove all observability

Cluster Command Syntax

Command	Arguments	Notes
`incus cluster enable`	`remote: member-name`	TWO args (space between)
`incus cluster add`	`remote:member-name`	ONE arg (no space)
`incus cluster join`	`init-remote: joining-remote:`	TWO args (space between)
`incus cluster remove`	`remote:member-name --force`	ONE arg; prompts even with `--force`
`incus cluster evacuate`	`remote:member-name`	ONE arg (no space)
`incus cluster restore`	`remote:member-name`	ONE arg (no space)
`incus config set`	`remote: key value`	Remote with trailing colon + space
`incus storage show`	`remote:pool`	ONE arg (no space)
`incus storage show`	`remote:pool --target member`	`--target` for member-specific

OVN Setup Cheat Sheet

# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true

Migration Readiness Checklist

Setting	Value	Why
`limits.cpu`	Range (e.g., `0-1`)	Fixed QEMU topology across hosts
`migration.stateful`	`true`	Enables live migration
`root size.state`	`2GiB` (or `4GiB` for 3-4 vCPUs)	Space for memory state file

Configure while VM is stopped:

incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device override REMOTE:VM root size.state=2GiB
incus start REMOTE:VM

Troubleshooting

Symptom	Cause	Fix
Port 8443 not reachable after boot	Boot still in progress or crontab bug	Wait 180s; check scrub_schedule via API
`scrub_schedule` empty	Crontab race condition	`incusos-proxmox --status` auto-heals
`Missing section footer for ICH9LPC` on migration	`limits.cpu` set as integer	Set as range: `limits.cpu=0-1`
`VM agent isn't currently running` after migration	Agent reconnecting	`sleep 4` after migration
`db.sock not found` on OVN config	OVN service not enabled on IncusOS	Enable via `/os/1.0/services/ovn` API
Cross-node ping fails (bridge)	Bridge networks are node-local	Use OVN network instead
`zfs load-key: Raw key too short`	TPM corruption from premature VM stop	Destroy and redeploy VM
Cluster join fails with "pool already exists"	`apply_defaults: true` on joining node	Use `apply_defaults: false` or run 8-command cleanup
OC cannot manage cluster nodes	Nodes deployed with standard ISO	Use OC-provisioned ISO for full integration
`CPUID vnmi` warning during migration	Cosmetic QEMU check	Safe to ignore
"Parent interface 'ens18' not found"	IncusOS names its NIC `mgmt`	Use `parent=mgmt` for UPLINK network
OC CLI "Invalid number of arguments"	OC CLI doesn't support `remote:` suffix	Use `operations-center remote switch NAME` first
Container placed on wrong node	Cluster auto-schedules without `--target`	Use `--target NODE` for explicit placement
"Device from profile(s) cannot be modified"	`root` device comes from default profile	Use `incus config device override` instead of `device set`
`incus launch` hangs or times out	Image download to new node takes 2-3 min	Run launches one at a time, not chained with `&&`
Aether CSRF error on API call	Session auth endpoints need CSRF token	Use Playwright or `aether-browser` helper
AWX pods OOMKilled	Web pod needs 2 GiB minimum	Check `awx-manifests/base/awx.yaml` resource limits
HAProxy 405 on health check	HTTP mode sends OPTIONS requests	Use TCP mode for nginx backends
Prometheus scrape failing on Incus	Client certs missing or wrong perms	Check `/etc/prometheus/certs/` in monitoring container
Grafana not reachable via forward	Network forward not created	`./deploy-observability --doctor` to diagnose

60 KiB Raw Blame History

Production Home Lab Guide

Section 0: Architecture Overview

Network Topology

Infrastructure

Inner Cluster Services

External IP Allocation

Decision Rationale

Cross-References

Section 1: Prerequisites

Required Tools

Aether Browser Automation

Aether Golden Image

Proxmox Configuration

Client Certificates

Doctor Check

Section 2: Deploy OC Server

Configuration File

Deploy

Set Up OC CLI Remote

Verify OC

Wait for Updates

Web UI Access

Section 3: Provision Nodes

3.1 Create Provisioning Token

3.2 Create Token Seed (No force_reboot)

3.3 Generate OC-Provisioned ISO (Older Version)

3.4 Node Configuration

3.5 Deploy Nodes (Hybrid Approach)

3.6 Verify Auto-Registration

3.7 Verify Scrub Schedules

Section 4: Form Cluster via Operations Center

4.1 The needs_update Blocker

4.2 Form Cluster

4.3 Fix Remotes After Clustering

4.4 Verify Cluster

4.5 Cluster Resources Created by OC

Section 5: Bridge Networking Baseline

Same-Node Communication

Cross-Node Communication (Fails)

Internet Access

Cleanup

Section 6: OVN Overlay Networking

6.1 Deploy OVN Control Plane

6.2 Enable OVN on All IncusOS Nodes

6.3 Configure Incus OVN Connection

6.4 Assign OVN Chassis Role

6.5 Create UPLINK Physical Network

6.6 Create OVN Network (net-prod)

6.7 Verify Cross-Node OVN Connectivity

6.8 Final Network State

Section 7: Deploy Aether

7.1 Import Golden Image

7.2 Create and Configure VM

7.3 Start and Run Post-Deploy

7.4 Set Admin Password

7.5 Connect Incus Cluster

7.6 Connect Operations Center

7.7 Verify

Section 8: HAProxy HA Load Balancing

8.1 Prerequisites Check

8.2 Preview

8.3 Deploy

8.4 Verify Load Balancing

8.5 Health and Status

8.6 ACL Notes

Section 9: AWX Lifecycle Automation

9.1 Prerequisites Check

9.2 Deploy

9.3 Configure

9.4 Register with Aether

9.5 How the Lifecycle Hooks Work

9.6 Verify

Section 10: Observability Stack

10.1 Deploy

10.2 Access Dashboards

10.3 Verify

Section 11: Network Isolation & Security

11.1 Create Isolated Network

11.2 Launch Isolated Containers

60 KiB

Raw Blame History