maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/operations-center-guide.md

37 KiB
Raw Blame History

Operations Center Guide for IncusOS

End-to-end guide for deploying an OC-managed Incus cluster with OVN overlay networking and an HA nginx workload. All commands and output captured from a real deployment on 2026-02-23 using Operations Center v0.3.0 and IncusOS build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).

Section 0: Architecture Overview

flowchart TD
    subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"]
        subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"]
            n1["oc-node-01<br/>VMID 400 · .140"]
            n2["oc-node-02<br/>VMID 401 · .141"]
            n3["oc-node-03<br/>VMID 402 · .142"]
        end
        oc["Operations Center<br/>VMID 920 · .120"]
    end

    vmbr0(("VLAN 69<br/>192.168.100.0/22"))
    ext["OVN external IPs<br/>192.168.103.200-210"]

    proxmox --- vmbr0
    cluster -.->|"external gateway"| ext

    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 node
    class oc mgmt
    class vmbr0,ext network

    style proxmox fill:#f5f5f5,stroke:#999
    style cluster fill:#e6f5f0,stroke:#009E73

Infrastructure

Component VMID IP Cores RAM Disk Role
oc-server 920 192.168.102.120/22 2 4 GiB 50G Operations Center
oc-node-01 400 192.168.102.140/22 4 8 GiB 64G Cluster init + OVN central
oc-node-02 401 192.168.102.141/22 4 8 GiB 50G Cluster member
oc-node-03 402 192.168.102.142/22 4 8 GiB 50G Cluster member

RAM budget: 28 GiB of 64 GiB (44%). VLAN 69, gateway 192.168.100.1.

Decision Rationale

  • OC-managed cluster: OC is the future default clustering path for Incus. Supports provisioning, update management, inventory, and centralized control.
  • OVN overlay: cross-node L2 connectivity for HA workloads. Sub-ms latency.
  • HA nginx: demonstrates load balancing, failover, and network forwards.
  • Static IPs: required for VLAN 69 (ARP-based detection doesn't work across VLANs).
  • apply_defaults: false: recommended for OC-managed nodes. OC's Terraform handles resource creation during cluster formation.

Cross-References

Section 1: Prerequisites

Required Tools

# Operations Center CLI
operations-center --version
# 0.3.0

# Incus client
incus version
# Client version: 6.21

# incusos-proxmox (from this repo)
./incusos/incusos-proxmox --doctor

Client Certificates

OC uses the same TLS client certificate format as Incus:

# Copy Incus certs for OC CLI
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# Generate PKCS#12 for browser access to OC web UI
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import

Section 2: Deploy OC Server

Configuration File

# incusos/examples/lab-oc-deploy.yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
    ip: 192.168.102.120/22

Deploy

./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml

Actual output (key lines):

[ok]  VM 'oc-server' created (VMID 920)
[ok]  VM 'oc-server' installed and running at 192.168.102.120

Set Up OC CLI Remote

# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab

Verify OC

operations-center admin os show

Actual output (uptime will vary):

WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602230420"
  os_version_next: ""
  uptime: 63

Wait for Updates

OC downloads IncusOS update packages from upstream. At least one update must reach ready state before ISOs can be generated:

# Poll until at least one update shows "ready"
operations-center provisioning update list

Actual output (after ~8 minutes; UUIDs are stable across deployments):

+--------------------------------------+---------------------+----------+--------------+----------+--------+
|                 UUID                 |       Origin        | Channels |   Version    | Severity | Status |
+--------------------------------------+---------------------+----------+--------------+----------+--------+
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable   | 202602200553 | none     | ready  |
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable   | 202602210344 | none     | ready  |
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable   | 202602230420 | none     | ready  |
+--------------------------------------+---------------------+----------+--------------+----------+--------+

The table also includes Upstream Channels and Published At columns (omitted for width). All 3 updates may not be ready at the same time -- at least one ready is sufficient to proceed. The latest version (202602230420) typically reaches ready first.

Section 3: Provision Nodes

3.1 Create Provisioning Token

operations-center provisioning token add --uses 5 --description "OC-managed cluster v2"
operations-center provisioning token list

Actual output (UUID changes every run):

+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
|                 UUID                 | Uses Remaining |           Expire At           | Channel |      Description      |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
| <token-UUID>                         | 5              | <30 days from now>            | stable  | OC-managed cluster v2 |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+

Save the <token-UUID> -- you'll need it for the next steps.

3.2 Create Token Seed (No force_reboot)

Critical: the token seed must NOT include force_reboot. On Proxmox, incusos-proxmox handles the install lifecycle externally (blockstat detection + media removal). force_reboot triggers SysRq-B which causes the ~50% crontab bug (see CLAUDE.md for full investigation).

# /tmp/oc-preseed.yaml
install:
  version: "1"
  force_install: true

Important: use the structured format with section keys (install:). A flat format (version: "1" at root) maps fields to empty {} and they don't get assigned to any section.

operations-center provisioning token seed add <token-UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"

3.3 Generate OC-Provisioned ISO (Older Version)

Critical discovery: nodes deployed from an ISO matching the latest OC update version are tracked as needs_update: true by OC because the OS was never delivered through OC's update pipeline. The fix: generate the ISO from an older channel so OC can push the real update after deployment.

# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
    --description "Older stable versions for initial provisioning"

# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable

# Generate ISO from the older channel
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable

Actual output:

Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"

The ISO contains IncusOS 202602210344 (one version behind). OC will push the latest (202602230420) after nodes register.

3.4 Node Configuration

# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false
    disk: 64
    ip: 192.168.102.140/22

  - name: oc-node-02
    app: incus
    apply_defaults: false
    ip: 192.168.102.141/22

  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22

3.5 Deploy Nodes (Hybrid Approach)

The hybrid approach uses incusos-proxmox --iso to combine OC auto-registration (from the boot ISO token) with incusos-proxmox VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup.

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml

Actual output (key lines):

[ok]  ISO uploaded: IncusOS-oc.iso
[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
[ok]  Remote 'oc-node-01' added (192.168.102.140)
[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
[ok]  Remote 'oc-node-02' added (192.168.102.141)
[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
[ok]  Remote 'oc-node-03' added (192.168.102.142)
[ok]  All post-deployment checks passed

All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.

3.6 Verify Auto-Registration

Nodes auto-register with OC within ~30 seconds of first boot. The update from 202602210344 to 202602230420 happens automatically:

operations-center provisioning server list

Actual output (key columns; full table includes Type, Channel, Certificate Fingerprint, Public Connection URL, Last Updated, Last Seen):

+---------+-------------------+------------------------------+--------+----------------+
| Cluster |       Name        |        Connection URL        | Status | Update Status  |
+---------+-------------------+------------------------------+--------+----------------+
|         | oc-node-01        | https://192.168.102.140:8443 | ready  | up to date     |
|         | oc-node-02        | https://192.168.102.141:8443 | ready  | up to date     |
|         | oc-node-03        | https://192.168.102.142:8443 | ready  | up to date     |
|         | operations-center | https://[::1]:8443           | ready  | update pending |
+---------+-------------------+------------------------------+--------+----------------+

Key: all 3 nodes show "up to date" because OC delivered the 202602230420 update through its pipeline. This is what unlocks clustering. Nodes may already be up to date by the time the last node finishes deploying -- the update gets pushed while incusos-proxmox deploys subsequent nodes sequentially.

3.7 Verify Scrub Schedules

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${node}:/os/1.0/system/storage | python3 -c \
        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done

Actual output:

oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0

All healthy. No crontab bug (force_reboot was not used).

Section 4: Form Cluster via Operations Center

4.1 The needs_update Blocker

OC requires all nodes to show needs_update: false before clustering. Nodes deployed from an ISO matching the latest version are tracked as needs_update: true because the OS was never delivered through OC's update pipeline. The needs_update flag is server-side computed and cannot be overridden via REST API PUT.

Solution: deploy from an older ISO version (Section 3.3). OC then pushes the real update to nodes through its pipeline, clearing the flag.

4.2 Form Cluster

Important: if the client certificate was already injected via SEED_DATA, use an empty application seed config to avoid "Certificate already in trust store" Terraform errors:

echo '{}' > /tmp/oc-app-config.yaml

operations-center provisioning cluster add oc-cluster \
    https://192.168.102.140:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC orchestrates the full cluster formation:

  1. Sets core.https_address to each node's specific IP
  2. Enables clustering on oc-node-01
  3. Joins oc-node-02 and oc-node-03
  4. Creates storage pool (local), networks (incusbr0, meshbr0)
  5. Runs Terraform/OpenTofu for post-cluster configuration

4.3 Fix Remotes After Clustering

Clustering regenerates TLS certificates. Re-add the remotes:

incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate

4.4 Verify Cluster

incus cluster list oc-node-01:

Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):

+------------+------------------------------+-----------------+--------------+--------+-------------------+
|    NAME    |             URL              |      ROLES      | ARCHITECTURE | STATUS |      MESSAGE      |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-01 | https://192.168.102.140:8443 | database-leader | x86_64       | ONLINE | Fully operational |
|            |                              | database        |              |        |                   |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-02 | https://192.168.102.141:8443 | database        | x86_64       | ONLINE | Fully operational |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-03 | https://192.168.102.142:8443 | database        | x86_64       | ONLINE | Fully operational |
+------------+------------------------------+-----------------+--------------+--------+-------------------+

All 3 nodes ONLINE and Fully operational. The ovn-chassis role is added later in Section 5.4.

4.5 Cluster Resources Created by OC

incus storage list oc-node-01:
incus network list oc-node-01:

Actual output (incusbr0 subnet varies per deployment):

+-------+--------+--------------------------------------+---------+---------+
| NAME  | DRIVER |             DESCRIPTION              | USED BY |  STATE  |
+-------+--------+--------------------------------------+---------+---------+
| local | zfs    | Local storage pool (on system drive) | 8       | CREATED |
+-------+--------+--------------------------------------+---------+---------+

+----------+--------+---------+-----------------+------------------------------+---------+
|   NAME   |  TYPE  | MANAGED |      IPV4       |         DESCRIPTION          | USED BY |
+----------+--------+---------+-----------------+------------------------------+---------+
| incusbr0 | bridge | YES     | 10.x.x.1/24    | Local network bridge (NAT)   | 1       |
| meshbr0  | bridge | YES     | none            | Internal mesh network bridge | 1       |
+----------+--------+---------+-----------------+------------------------------+---------+

OC creates: local storage pool (ZFS), incusbr0 bridge (NAT), and meshbr0 (OC-specific mesh network for inter-node communication). The table also includes IPv6 and STATE columns.

Section 5: OVN Overlay Networking

5.1 Deploy OVN Control Plane

Launch an ovn-central container on oc-node-01:

incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
incus exec oc-node-01:ovn-central -- bash -c \
    "apt-get update -qq && apt-get install -y -qq ovn-central"

Configure OVN to listen on all interfaces:

incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0

Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:

incus config device add oc-node-01:ovn-central \
    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642

5.2 Enable OVN on All IncusOS Nodes

Enable OVN services via the IncusOS REST API (/os/1.0/services/ovn). The database field is the southbound DB (port 6642), not northbound.

for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
        \"config\": {
            \"database\": \"tcp:192.168.102.140:6642\",
            \"enabled\": true,
            \"tunnel_address\": \"${node_ip}\",
            \"tunnel_protocol\": \"geneve\"
        },
        \"state\": {}
    }"
done

5.3 Configure Incus OVN Connection

incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641

5.4 Assign OVN Chassis Role

for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role add oc-node-01:${node} ovn-chassis
done

5.5 Create UPLINK Physical Network

Uses parent=mgmt (IncusOS management NIC). Two-step cluster pattern:

# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done

# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1

5.6 Create OVN Network (net-prod)

incus network create oc-node-01:net-prod --type=ovn \
    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true

Actual output:

Network net-prod created

net-prod is assigned external IP 192.168.103.200 from the UPLINK range.

5.7 Verify Cross-Node OVN Connectivity

incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3

Actual output:

64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms

Sub-millisecond cross-node latency via Geneve tunnels. Clean up test containers after verification:

incus delete oc-node-01:test-1 oc-node-01:test-2 --force

5.8 Final Network State

incus network list oc-node-01:

Actual output:

+----------+----------+---------+----------------+------------------------------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4      |         DESCRIPTION          | USED BY |
+----------+----------+---------+----------------+------------------------------+---------+
| UPLINK   | physical | YES     |                |                              | 1       |
| incusbr0 | bridge   | YES     | 10.x.x.1/24   | Local network bridge (NAT)   | 2       |
| meshbr0  | bridge   | YES     | none           | Internal mesh network bridge | 1       |
| net-prod | ovn      | YES     | 10.10.10.1/24  |                              | 0       |
+----------+----------+---------+----------------+------------------------------+---------+

The incusbr0 subnet is randomly assigned per deployment. The USED BY count for net-prod is 0 at this point (test containers deleted); it increases as workloads are added in Section 6.

Section 6: HA Nginx Workload

6.1 Launch Nginx Containers

Launch 3 nginx containers across all nodes on net-prod:

incus launch images:debian/12 oc-node-01:ha-web-01 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:ha-web-02 --target oc-node-02 -n net-prod
incus launch images:debian/12 oc-node-01:ha-web-03 --target oc-node-03 -n net-prod

6.2 Install Nginx and Set Distinct Content

for i in 01 02 03; do
    incus exec oc-node-01:ha-web-${i} -- bash -c \
        "apt-get update -qq && apt-get install -y -qq nginx"
    incus exec oc-node-01:ha-web-${i} -- bash -c \
        "echo '<h1>ha-web-${i} on oc-node-${i}</h1>' > /var/www/html/index.html"
    incus exec oc-node-01:ha-web-${i} -- systemctl start nginx
done

6.3 Create OVN Load Balancer

incus network load-balancer create oc-node-01:net-prod 192.168.103.201

# Add backends (must use IP addresses, not instance names)
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
    ha-web-01 10.10.10.2 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
    ha-web-02 10.10.10.3 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
    ha-web-03 10.10.10.4 80

# Add port forwarding rule
incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \
    tcp 80 ha-web-01,ha-web-02,ha-web-03

6.4 Test Load Balancer

for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done

Actual output (distribution varies per run due to connection-based hashing):

<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1>

Traffic distributes across backends (connection-based hashing, not round-robin). Not all backends may appear in every 5-request sample.

6.5 HA Failover Test

# Stop ha-web-01
incus stop oc-node-01:ha-web-01
sleep 2

# Test with one backend down
for i in 1 2 3 4 5; do curl -s --connect-timeout 5 http://192.168.103.201; done

Actual result: remaining backends (ha-web-02, ha-web-03) serve traffic. Some requests to the dead backend return empty (OVN LB uses connection hashing, not health checks — no active health monitoring).

# Restart ha-web-01
incus start oc-node-01:ha-web-01
sleep 3

# All 3 backends serve again
for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done

Actual output after restart (all 3 backends serving again):

<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>

All 3 backends recovered.

6.6 Network Forward (Direct Backend Access)

incus network forward create oc-node-01:net-prod 192.168.103.202
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8080 10.10.10.2 80
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8081 10.10.10.3 80
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8082 10.10.10.4 80

Test direct access:

curl -s http://192.168.103.202:8080  # ha-web-01
curl -s http://192.168.103.202:8081  # ha-web-02
curl -s http://192.168.103.202:8082  # ha-web-03

Actual output:

<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-03 on oc-node-03</h1>

6.7 Final Instance State

incus list oc-node-01:

Actual output (key columns; full table includes IPv6, SNAPSHOTS):

+-------------+---------+-----------------------+-----------+------------+
|    NAME     |  STATE  |         IPV4          |   TYPE    |  LOCATION  |
+-------------+---------+-----------------------+-----------+------------+
| ha-web-01   | RUNNING | 10.10.10.2 (eth0)     | CONTAINER | oc-node-01 |
| ha-web-02   | RUNNING | 10.10.10.3 (eth0)     | CONTAINER | oc-node-02 |
| ha-web-03   | RUNNING | 10.10.10.4 (eth0)     | CONTAINER | oc-node-03 |
| ovn-central | RUNNING | 10.x.x.x (eth0)      | CONTAINER | oc-node-01 |
+-------------+---------+-----------------------+-----------+------------+

The ovn-central container is on incusbr0 (random subnet per deployment). The ha-web containers are on net-prod (10.10.10.0/24, fixed).

Section 7: OC Management

7.1 Inventory Sync

OC inventory is not real-time — requires explicit cluster resync:

operations-center provisioning cluster resync oc-cluster

7.2 Inventory Browsing

operations-center inventory instance list --cluster oc-cluster

Actual output (UUIDs are deterministic based on name/project/cluster):

+--------------------------------------+-------------+---------+------------+------------+
|                 UUID                 |    Name     | Project |  Cluster   |   Server   |
+--------------------------------------+-------------+---------+------------+------------+
| cf27717e-5f87-5a22-9e82-cc6616fd665d | ha-web-01   | default | oc-cluster | oc-node-01 |
| 657447d3-7773-5005-af07-b62d9f789ffc | ha-web-02   | default | oc-cluster | oc-node-02 |
| 7b79cab9-37f9-50d8-9d4f-20daafd3d932 | ha-web-03   | default | oc-cluster | oc-node-03 |
| 17133a45-6318-5aa5-b6a8-fc5a5871f679 | ovn-central | default | oc-cluster | oc-node-01 |
+--------------------------------------+-------------+---------+------------+------------+

The full table also includes a Last Updated column.

operations-center inventory network list --cluster oc-cluster

Actual output:

+--------------------------------------+----------+----------+---------+------------+
|                 UUID                 |   Name   |   Type   | Project |  Cluster   |
+--------------------------------------+----------+----------+---------+------------+
| 628df562-c980-5c27-8268-2ba69211eb7f | UPLINK   | physical | default | oc-cluster |
| e35ecd83-25c7-54ae-8fd9-66bc241a002f | incusbr0 | bridge   | default | oc-cluster |
| 9cc58dd0-9821-5649-807a-88e0cbce8f5c | meshbr0  | bridge   | default | oc-cluster |
| 9c3ee7cb-6b7e-5ea9-bf91-e8d44f0f04da | net-prod | ovn      | default | oc-cluster |
+--------------------------------------+----------+----------+---------+------------+

The full table also includes a Last Updated column.

operations-center inventory network-load-balancer list
operations-center inventory network-forward list
operations-center inventory storage-pool list

All resources visible in OC inventory after resync.

7.3 Server Management

operations-center provisioning server list

Actual output (key columns):

+------------+-------------------+------------------------------+--------+----------------+
|  Cluster   |       Name        |        Connection URL        | Status | Update Status  |
+------------+-------------------+------------------------------+--------+----------------+
| oc-cluster | oc-node-01        | https://192.168.102.140:8443 | ready  | up to date     |
| oc-cluster | oc-node-02        | https://192.168.102.141:8443 | ready  | up to date     |
| oc-cluster | oc-node-03        | https://192.168.102.142:8443 | ready  | up to date     |
|            | operations-center | https://[::1]:8443           | ready  | update pending |
+------------+-------------------+------------------------------+--------+----------------+

Nodes now show their cluster membership. The OC server itself always shows update pending (it cannot update itself via its own pipeline).

7.4 Update Management

operations-center provisioning update list
operations-center provisioning update show <UUID>
operations-center provisioning update refresh     # Check for new updates
operations-center provisioning update assign-channels <UUID> --channel <name>

Channel management: updates auto-link to channels based on upstream filter_expression. Use assign-channels to explicitly control which updates appear in which channels.

7.5 Web UI

Navigate to https://192.168.102.120:8443/ui/ in Firefox (with client.pfx imported). The web UI shows: servers, clusters, instances, networks, storage pools, load balancers, network forwards, and update status.

Section 8: Cluster Lifecycle

8.1 Evacuation & Restore

incus cluster evacuate oc-node-01:oc-node-01 --force
# OC still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
# Inventory shows workloads moved to other nodes

incus cluster restore oc-node-01:oc-node-01 --force

8.2 Node Failure Simulation

Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s). Node auto-rejoins after Proxmox restart (~60s boot):

# OC does NOT detect node failures
# provisioning server show still reports "Status: ready" for offline nodes

8.3 Reboot Considerations

WARNING: operations-center provisioning server system reboot can break OC-managed nodes on Proxmox. The OC agent pushes config via the IncusOS REST API that gets persisted to state.txt. On reboot, invalid values crash the daemon. See CLAUDE.md for full analysis.

Safe maintenance procedure:

  1. incus cluster evacuate <remote>:<member> --force
  2. Proxmox stop the VM
  3. Proxmox start the VM
  4. Wait for IncusOS to boot (~60-90s)
  5. incus cluster restore <remote>:<member> --force

Section 9: Cleanup

Note: this section documents the cleanup procedure but is NOT executed. The lab is left fully running with all workloads, OVN networking, and load balancer active for exploration via OC web UI and CLI.

# Delete workloads
incus delete oc-node-01:ha-web-01 oc-node-01:ha-web-02 oc-node-01:ha-web-03 --force

# Delete load balancer and forward
incus network load-balancer delete oc-node-01:net-prod 192.168.103.201
incus network forward delete oc-node-01:net-prod 192.168.103.202

# Delete OVN network
incus network delete oc-node-01:net-prod

# Delete OVN control plane
incus delete oc-node-01:ovn-central --force

# Delete UPLINK
incus network delete oc-node-01:UPLINK

# Disable OVN services on all nodes
for remote in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data '{
        "config": {"enabled": false}, "state": {}
    }'
done

# Remove OVN chassis role
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role remove oc-node-01:${node} ovn-chassis
done

# Destroy infrastructure (--yes to skip confirmation prompts)
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml

# Remove OC remote
operations-center remote remove oc-lab

Section 10: Verification Checklist

# Check Status
1 OC server deployed and reachable PASS
2 OC remote configured PASS
3 Updates downloaded and ready PASS
4 3 nodes provisioned from OC ISO PASS
5 All nodes auto-registered with OC PASS
6 All nodes "up to date" (not "update pending") PASS
7 No crontab bug (force_reboot not used) PASS
8 Cluster formed via OC provisioning cluster add PASS
9 All 3 nodes ONLINE + Fully operational PASS
10 OVN cross-node connectivity (sub-ms) PASS
11 3 nginx instances on 3 nodes PASS
12 LB distributes across all backends PASS
13 HA: LB serves with one backend down PASS
14 Network forward: direct backend access PASS
15 OC inventory shows all resources PASS
16 OC web UI accessible PASS

Section 11: Quick Reference

OC Provisioning Workflow

1. Deploy OC server (incusos-proxmox)
2. Add OC remote (operations-center remote add)
3. Wait for updates (provisioning update list -- at least one "ready")
4. Create token (provisioning token add)
5. Create token seed (provisioning token seed add) -- NO force_reboot
6. Create old-stable channel (provisioning channel add old-stable)
7. Assign older update to old-stable (provisioning update assign-channels)
8. Generate ISO from OLDER channel (token seed get-image --channel old-stable)
9. Deploy nodes (incusos-proxmox --iso)
10. Wait for auto-registration + update delivery
11. Form cluster (provisioning cluster add)
12. Set up OVN networking (OVN container + services + UPLINK + net-prod)

Key OC CLI Commands

Command Description
remote add <name> <url> --auth-type tls Add OC remote
remote switch <name> Set default remote
admin os show IncusOS hostname, version, uptime
provisioning token add --uses N Create provisioning token
provisioning token seed add <UUID> <name> <file> Attach seed to token
provisioning token seed get-image <UUID> <seed> <file> --channel <ch> Generate ISO
provisioning channel add <name> --description <desc> Create provisioning channel
provisioning update list List available updates
provisioning update refresh Check for new updates
provisioning update assign-channels <UUID> --channel <name> Assign update to channel
provisioning server list List registered servers
provisioning server resync <name> Resync server state
provisioning cluster add <name> <url> --server-names ... -a <seed> Form cluster
provisioning cluster resync <name> Force inventory resync
provisioning cluster list List clusters
inventory instance list --cluster <name> List instances
inventory network list --cluster <name> List networks
inventory network-load-balancer list List load balancers
inventory network-forward list List network forwards
inventory storage-pool list List storage pools

Incus Cluster Commands (via remote)

Command Description
incus cluster list <remote>: List cluster members
incus cluster role add <remote>:<member> ovn-chassis Add OVN role
incus cluster evacuate <remote>:<member> --force Evacuate member
incus cluster restore <remote>:<member> --force Restore member
incus network create <remote>:<net> --type=physical --target=<member> parent=mgmt Per-target network
incus network create <remote>:<net> --type=ovn network=UPLINK OVN network
incus network load-balancer create <remote>:<net> <ip> Create LB
incus network forward create <remote>:<net> <ip> Create forward

Section 12: Troubleshooting

Symptom Cause Fix
needs_update: true blocks clustering Nodes deployed from ISO matching latest version Deploy from older ISO (--channel old-stable), let OC push the update
Terraform "Certificate already in trust store" Cert in both SEED_DATA and app-seed-config Use empty {} for app-seed-config when cert is in SEED_DATA
Terraform "Network/Pool not in pending state" apply_defaults: true pre-created resources Use apply_defaults: false (recommended)
TLS cert error after clustering Cluster cert regenerated incus remote remove/add with --accept-certificate
get-image fails No updates in ready state Wait ~5 min for OC to download; provisioning update list
Node doesn't self-register Bad ISO or token exhausted Verify ISO from get-image; check token list
Token seed fields all {} Flat YAML format Use structured: install: section key
OC web UI 403 Client cert not imported Import client.pfx into browser
OC shows "ready" for offline node OC doesn't detect failures Check via incus cluster list instead
OC reboot breaks node OC agent pushes invalid config Use Proxmox stop/start, not OC reboot
UPLINK type mismatch error Missing --type=physical on cluster-wide create Include --type=physical on both per-target and cluster-wide
Stale ISO on Proxmox incusos-proxmox skipped upload Fixed in script: --iso now replaces existing same-named ISOs
assign-channels "Not found" Channel doesn't exist yet Run provisioning channel add old-stable first
Bash ! in API token History expansion corrupts auth header Use Python urllib.request or set +H

Section 13: Known Limitations (v0.3.0)

  • No brownfield adoption: nodes must boot from OC-provisioned ISO
  • No real-time inventory sync: requires explicit cluster resync
  • No cluster member state tracking: OC always shows ready for all nodes
  • Stale server entries: out-of-band changes create stale entries
  • needs_update tracking: based on OC's delivery pipeline, not version comparison
  • OC reboot breaks Proxmox VMs: OC agent boot failures on restart
  • No cluster resize: cannot add/remove members from existing OC clusters
  • Token seeds require structured YAML: section keys, not flat format
  • OVN LB has no health checks: connection hashing distributes to dead backends

References