# Operations Center Guide for IncusOS
End-to-end guide for deploying an OC-managed Incus cluster with OVN overlay
networking and an HA nginx workload. All commands and output captured from a
real deployment on 2026-02-23 using Operations Center v0.3.0 and IncusOS
build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).
---
## Section 0: Architecture Overview
```mermaid
flowchart TD
subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"]
subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"]
n1["oc-node-01
VMID 400 · .140"]
n2["oc-node-02
VMID 401 · .141"]
n3["oc-node-03
VMID 402 · .142"]
end
oc["Operations Center
VMID 920 · .120"]
end
vmbr0(("VLAN 69
192.168.100.0/22"))
ext["OVN external IPs
192.168.103.200-210"]
proxmox --- vmbr0
cluster -.->|"external gateway"| ext
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
class n1,n2,n3 node
class oc mgmt
class vmbr0,ext network
style proxmox fill:#f5f5f5,stroke:#999
style cluster fill:#e6f5f0,stroke:#009E73
```
### Infrastructure
| Component | VMID | IP | Cores | RAM | Disk | Role |
|-----------|------|-----|-------|-----|------|------|
| oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center |
| oc-node-01 | 400 | 192.168.102.140/22 | 4 | 8 GiB | 64G | Cluster init + OVN central |
| oc-node-02 | 401 | 192.168.102.141/22 | 4 | 8 GiB | 50G | Cluster member |
| oc-node-03 | 402 | 192.168.102.142/22 | 4 | 8 GiB | 50G | Cluster member |
**RAM budget**: 28 GiB of 64 GiB (44%). VLAN 69, gateway 192.168.100.1.
### Decision Rationale
- **OC-managed cluster**: OC is the future default clustering path for Incus.
Supports provisioning, update management, inventory, and centralized control.
- **OVN overlay**: cross-node L2 connectivity for HA workloads. Sub-ms latency.
- **HA nginx**: demonstrates load balancing, failover, and network forwards.
- **Static IPs**: required for VLAN 69 (ARP-based detection doesn't work
across VLANs).
- **apply_defaults: false**: recommended for OC-managed nodes. OC's Terraform
handles resource creation during cluster formation.
### Cross-References
- [Clustering guide](clustering-guide.md) -- manual cluster formation reference
- [Networking guide](networking-guide.md) -- OVN overlay tutorial
- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA
---
## Section 1: Prerequisites
### Required Tools
```bash
# Operations Center CLI
operations-center --version
# 0.3.0
# Incus client
incus version
# Client version: 6.21
# incusos-proxmox (from this repo)
./incusos/incusos-proxmox --doctor
```
### Client Certificates
OC uses the same TLS client certificate format as Incus:
```bash
# Copy Incus certs for OC CLI
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
# Generate PKCS#12 for browser access to OC web UI
openssl pkcs12 -export \
-inkey ~/.config/incus/client.key \
-in ~/.config/incus/client.crt \
-out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
```
---
## Section 2: Deploy OC Server
### Configuration File
```yaml
# incusos/examples/lab-oc-deploy.yaml
defaults:
cores: 2
memory: 4096
disk: 50
start_vmid: 920
proxmox:
gateway: 192.168.100.1
dns: 192.168.100.1
vms:
- name: oc-server
app: operations-center
apply_defaults: true
ip: 192.168.102.120/22
```
### Deploy
```bash
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
```
Actual output (key lines):
```
[ok] VM 'oc-server' created (VMID 920)
[ok] VM 'oc-server' installed and running at 192.168.102.120
```
### Set Up OC CLI Remote
```bash
# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab
```
### Verify OC
```bash
operations-center admin os show
```
Actual output (uptime will vary):
```
WARNING: The IncusOS API and configuration is subject to change
environment:
hostname: oc-server
os_name: IncusOS
os_version: "202602230420"
os_version_next: ""
uptime: 63
```
### Wait for Updates
OC downloads IncusOS update packages from upstream. At least one update must
reach `ready` state before ISOs can be generated:
```bash
# Poll until at least one update shows "ready"
operations-center provisioning update list
```
Actual output (after ~8 minutes; UUIDs are stable across deployments):
```
+--------------------------------------+---------------------+----------+--------------+----------+--------+
| UUID | Origin | Channels | Version | Severity | Status |
+--------------------------------------+---------------------+----------+--------------+----------+--------+
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready |
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready |
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready |
+--------------------------------------+---------------------+----------+--------------+----------+--------+
```
The table also includes `Upstream Channels` and `Published At` columns (omitted
for width). All 3 updates may not be ready at the same time -- at least one
`ready` is sufficient to proceed. The latest version (202602230420) typically
reaches `ready` first.
---
## Section 3: Provision Nodes
### 3.1 Create Provisioning Token
```bash
operations-center provisioning token add --uses 5 --description "OC-managed cluster v2"
operations-center provisioning token list
```
Actual output (UUID changes every run):
```
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
| UUID | Uses Remaining | Expire At | Channel | Description |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
| | 5 | <30 days from now> | stable | OC-managed cluster v2 |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
```
Save the `` -- you'll need it for the next steps.
### 3.2 Create Token Seed (No force_reboot)
**Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
`incusos-proxmox` handles the install lifecycle externally (blockstat
detection + media removal). `force_reboot` triggers SysRq-B which causes
the ~50% crontab bug (see CLAUDE.md for full investigation).
```yaml
# /tmp/oc-preseed.yaml
install:
version: "1"
force_install: true
```
**Important**: use the structured format with section keys (`install:`). A flat
format (`version: "1"` at root) maps fields to empty `{}` and they don't get
assigned to any section.
```bash
operations-center provisioning token seed add proxmox-preseed \
/tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
```
### 3.3 Generate OC-Provisioned ISO (Older Version)
**Critical discovery**: nodes deployed from an ISO matching the latest OC
update version are tracked as `needs_update: true` by OC because the OS
was never delivered through OC's update pipeline. The fix: generate the ISO
from an older channel so OC can push the real update after deployment.
```bash
# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
--description "Older stable versions for initial provisioning"
# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels --channel old-stable
# Generate ISO from the older channel
operations-center provisioning token seed get-image proxmox-preseed \
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
```
Actual output:
```
Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"
```
The ISO contains IncusOS 202602210344 (one version behind). OC will push
the latest (202602230420) after nodes register.
### 3.4 Node Configuration
```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
cores: 4
memory: 8192
disk: 50
start_vmid: 400
proxmox:
gateway: 192.168.100.1
dns: 192.168.100.1
vms:
- name: oc-node-01
app: incus
apply_defaults: false
disk: 64
ip: 192.168.102.140/22
- name: oc-node-02
app: incus
apply_defaults: false
ip: 192.168.102.141/22
- name: oc-node-03
app: incus
apply_defaults: false
ip: 192.168.102.142/22
```
### 3.5 Deploy Nodes (Hybrid Approach)
The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration
(from the boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA
(hostname, static IP), install monitoring, and media cleanup.
```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml
```
Actual output (key lines):
```
[ok] ISO uploaded: IncusOS-oc.iso
[ok] VM 'oc-node-01' installed and running at 192.168.102.140
[ok] Remote 'oc-node-01' added (192.168.102.140)
[ok] VM 'oc-node-02' installed and running at 192.168.102.141
[ok] Remote 'oc-node-02' added (192.168.102.141)
[ok] VM 'oc-node-03' installed and running at 192.168.102.142
[ok] Remote 'oc-node-03' added (192.168.102.142)
[ok] All post-deployment checks passed
```
All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.
### 3.6 Verify Auto-Registration
Nodes auto-register with OC within ~30 seconds of first boot. The update
from 202602210344 to 202602230420 happens automatically:
```bash
operations-center provisioning server list
```
Actual output (key columns; full table includes Type, Channel, Certificate
Fingerprint, Public Connection URL, Last Updated, Last Seen):
```
+---------+-------------------+------------------------------+--------+----------------+
| Cluster | Name | Connection URL | Status | Update Status |
+---------+-------------------+------------------------------+--------+----------------+
| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pending |
+---------+-------------------+------------------------------+--------+----------------+
```
**Key**: all 3 nodes show **"up to date"** because OC delivered the
202602230420 update through its pipeline. This is what unlocks clustering.
Nodes may already be up to date by the time the last node finishes
deploying -- the update gets pushed while `incusos-proxmox` deploys
subsequent nodes sequentially.
### 3.7 Verify Scrub Schedules
```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
incus query ${node}:/os/1.0/system/storage | python3 -c \
"import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done
```
Actual output:
```
oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0
```
All healthy. No crontab bug (force_reboot was not used).
---
## Section 4: Form Cluster via Operations Center
### 4.1 The needs_update Blocker
OC requires all nodes to show `needs_update: false` before clustering.
Nodes deployed from an ISO matching the latest version are tracked as
`needs_update: true` because the OS was never delivered through OC's update
pipeline. The `needs_update` flag is server-side computed and cannot be
overridden via REST API PUT.
**Solution**: deploy from an older ISO version (Section 3.3). OC then pushes
the real update to nodes through its pipeline, clearing the flag.
### 4.2 Form Cluster
**Important**: if the client certificate was already injected via SEED_DATA,
use an empty application seed config to avoid "Certificate already in trust
store" Terraform errors:
```bash
echo '{}' > /tmp/oc-app-config.yaml
operations-center provisioning cluster add oc-cluster \
https://192.168.102.140:8443 \
--server-names oc-node-01,oc-node-02,oc-node-03 \
--server-type incus \
--application-seed-config /tmp/oc-app-config.yaml
```
OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on oc-node-01
3. Joins oc-node-02 and oc-node-03
4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`)
5. Runs Terraform/OpenTofu for post-cluster configuration
### 4.3 Fix Remotes After Clustering
Clustering regenerates TLS certificates. Re-add the remotes:
```bash
incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
```
### 4.4 Verify Cluster
```bash
incus cluster list oc-node-01:
```
Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):
```
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-01 | https://192.168.102.140:8443 | database-leader | x86_64 | ONLINE | Fully operational |
| | | database | | | |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational |
+------------+------------------------------+-----------------+--------------+--------+-------------------+
```
All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added
later in Section 5.4.
### 4.5 Cluster Resources Created by OC
```bash
incus storage list oc-node-01:
incus network list oc-node-01:
```
Actual output (incusbr0 subnet varies per deployment):
```
+-------+--------+--------------------------------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+-------+--------+--------------------------------------+---------+---------+
| local | zfs | Local storage pool (on system drive) | 8 | CREATED |
+-------+--------+--------------------------------------+---------+---------+
+----------+--------+---------+-----------------+------------------------------+---------+
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
+----------+--------+---------+-----------------+------------------------------+---------+
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
+----------+--------+---------+-----------------+------------------------------+---------+
```
OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
`meshbr0` (OC-specific mesh network for inter-node communication). The
table also includes IPv6 and STATE columns.
---
## Section 5: OVN Overlay Networking
### 5.1 Deploy OVN Control Plane
Launch an `ovn-central` container on oc-node-01:
```bash
incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
incus exec oc-node-01:ovn-central -- bash -c \
"apt-get update -qq && apt-get install -y -qq ovn-central"
```
Configure OVN to listen on all interfaces:
```bash
incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
```
Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:
```bash
incus config device add oc-node-01:ovn-central \
nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642
```
### 5.2 Enable OVN on All IncusOS Nodes
Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`).
The `database` field is the **southbound** DB (port 6642), not northbound.
```bash
for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
\"config\": {
\"database\": \"tcp:192.168.102.140:6642\",
\"enabled\": true,
\"tunnel_address\": \"${node_ip}\",
\"tunnel_protocol\": \"geneve\"
},
\"state\": {}
}"
done
```
### 5.3 Configure Incus OVN Connection
```bash
incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641
```
### 5.4 Assign OVN Chassis Role
```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
incus cluster role add oc-node-01:${node} ovn-chassis
done
```
### 5.5 Create UPLINK Physical Network
Uses `parent=mgmt` (IncusOS management NIC). Two-step cluster pattern:
```bash
# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done
# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
ipv4.gateway=192.168.100.1/22 \
dns.nameservers=192.168.100.1
```
### 5.6 Create OVN Network (net-prod)
```bash
incus network create oc-node-01:net-prod --type=ovn \
network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true
```
Actual output:
```
Network net-prod created
```
net-prod is assigned external IP `192.168.103.200` from the UPLINK range.
### 5.7 Verify Cross-Node OVN Connectivity
```bash
incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
```
Actual output:
```
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms
```
Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
containers after verification:
```bash
incus delete oc-node-01:test-1 oc-node-01:test-2 --force
```
### 5.8 Final Network State
```bash
incus network list oc-node-01:
```
Actual output:
```
+----------+----------+---------+----------------+------------------------------+---------+
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
+----------+----------+---------+----------------+------------------------------+---------+
| UPLINK | physical | YES | | | 1 |
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
| net-prod | ovn | YES | 10.10.10.1/24 | | 0 |
+----------+----------+---------+----------------+------------------------------+---------+
```
The incusbr0 subnet is randomly assigned per deployment. The USED BY count
for net-prod is 0 at this point (test containers deleted); it increases as
workloads are added in Section 6.
---
## Section 6: HA Nginx Workload
### 6.1 Launch Nginx Containers
Launch 3 nginx containers across all nodes on net-prod:
```bash
incus launch images:debian/12 oc-node-01:ha-web-01 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:ha-web-02 --target oc-node-02 -n net-prod
incus launch images:debian/12 oc-node-01:ha-web-03 --target oc-node-03 -n net-prod
```
### 6.2 Install Nginx and Set Distinct Content
```bash
for i in 01 02 03; do
incus exec oc-node-01:ha-web-${i} -- bash -c \
"apt-get update -qq && apt-get install -y -qq nginx"
incus exec oc-node-01:ha-web-${i} -- bash -c \
"echo 'ha-web-${i} on oc-node-${i}
' > /var/www/html/index.html"
incus exec oc-node-01:ha-web-${i} -- systemctl start nginx
done
```
### 6.3 Create OVN Load Balancer
```bash
incus network load-balancer create oc-node-01:net-prod 192.168.103.201
# Add backends (must use IP addresses, not instance names)
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
ha-web-01 10.10.10.2 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
ha-web-02 10.10.10.3 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
ha-web-03 10.10.10.4 80
# Add port forwarding rule
incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \
tcp 80 ha-web-01,ha-web-02,ha-web-03
```
### 6.4 Test Load Balancer
```bash
for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done
```
Actual output (distribution varies per run due to connection-based hashing):
```
ha-web-01 on oc-node-01
ha-web-02 on oc-node-02
ha-web-01 on oc-node-01
ha-web-01 on oc-node-01
ha-web-02 on oc-node-02
```
Traffic distributes across backends (connection-based hashing, not
round-robin). Not all backends may appear in every 5-request sample.
### 6.5 HA Failover Test
```bash
# Stop ha-web-01
incus stop oc-node-01:ha-web-01
sleep 2
# Test with one backend down
for i in 1 2 3 4 5; do curl -s --connect-timeout 5 http://192.168.103.201; done
```
Actual result: remaining backends (ha-web-02, ha-web-03) serve traffic. Some
requests to the dead backend return empty (OVN LB uses connection hashing,
not health checks — no active health monitoring).
```bash
# Restart ha-web-01
incus start oc-node-01:ha-web-01
sleep 3
# All 3 backends serve again
for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done
```
Actual output after restart (all 3 backends serving again):
```
ha-web-01 on oc-node-01
ha-web-02 on oc-node-02
ha-web-02 on oc-node-02
ha-web-01 on oc-node-01
ha-web-01 on oc-node-01
ha-web-01 on oc-node-01
```
All 3 backends recovered.
### 6.6 Network Forward (Direct Backend Access)
```bash
incus network forward create oc-node-01:net-prod 192.168.103.202
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8080 10.10.10.2 80
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8081 10.10.10.3 80
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8082 10.10.10.4 80
```
Test direct access:
```bash
curl -s http://192.168.103.202:8080 # ha-web-01
curl -s http://192.168.103.202:8081 # ha-web-02
curl -s http://192.168.103.202:8082 # ha-web-03
```
Actual output:
```
ha-web-01 on oc-node-01
ha-web-02 on oc-node-02
ha-web-03 on oc-node-03
```
### 6.7 Final Instance State
```bash
incus list oc-node-01:
```
Actual output (key columns; full table includes IPv6, SNAPSHOTS):
```
+-------------+---------+-----------------------+-----------+------------+
| NAME | STATE | IPV4 | TYPE | LOCATION |
+-------------+---------+-----------------------+-----------+------------+
| ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 |
| ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 |
| ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 |
| ovn-central | RUNNING | 10.x.x.x (eth0) | CONTAINER | oc-node-01 |
+-------------+---------+-----------------------+-----------+------------+
```
The ovn-central container is on incusbr0 (random subnet per deployment).
The ha-web containers are on net-prod (10.10.10.0/24, fixed).
---
## Section 7: OC Management
### 7.1 Inventory Sync
OC inventory is **not real-time** — requires explicit `cluster resync`:
```bash
operations-center provisioning cluster resync oc-cluster
```
### 7.2 Inventory Browsing
```bash
operations-center inventory instance list --cluster oc-cluster
```
Actual output (UUIDs are deterministic based on name/project/cluster):
```
+--------------------------------------+-------------+---------+------------+------------+
| UUID | Name | Project | Cluster | Server |
+--------------------------------------+-------------+---------+------------+------------+
| cf27717e-5f87-5a22-9e82-cc6616fd665d | ha-web-01 | default | oc-cluster | oc-node-01 |
| 657447d3-7773-5005-af07-b62d9f789ffc | ha-web-02 | default | oc-cluster | oc-node-02 |
| 7b79cab9-37f9-50d8-9d4f-20daafd3d932 | ha-web-03 | default | oc-cluster | oc-node-03 |
| 17133a45-6318-5aa5-b6a8-fc5a5871f679 | ovn-central | default | oc-cluster | oc-node-01 |
+--------------------------------------+-------------+---------+------------+------------+
```
The full table also includes a `Last Updated` column.
```bash
operations-center inventory network list --cluster oc-cluster
```
Actual output:
```
+--------------------------------------+----------+----------+---------+------------+
| UUID | Name | Type | Project | Cluster |
+--------------------------------------+----------+----------+---------+------------+
| 628df562-c980-5c27-8268-2ba69211eb7f | UPLINK | physical | default | oc-cluster |
| e35ecd83-25c7-54ae-8fd9-66bc241a002f | incusbr0 | bridge | default | oc-cluster |
| 9cc58dd0-9821-5649-807a-88e0cbce8f5c | meshbr0 | bridge | default | oc-cluster |
| 9c3ee7cb-6b7e-5ea9-bf91-e8d44f0f04da | net-prod | ovn | default | oc-cluster |
+--------------------------------------+----------+----------+---------+------------+
```
The full table also includes a `Last Updated` column.
```bash
operations-center inventory network-load-balancer list
operations-center inventory network-forward list
operations-center inventory storage-pool list
```
All resources visible in OC inventory after resync.
### 7.3 Server Management
```bash
operations-center provisioning server list
```
Actual output (key columns):
```
+------------+-------------------+------------------------------+--------+----------------+
| Cluster | Name | Connection URL | Status | Update Status |
+------------+-------------------+------------------------------+--------+----------------+
| oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pending |
+------------+-------------------+------------------------------+--------+----------------+
```
Nodes now show their cluster membership. The OC server itself always shows
`update pending` (it cannot update itself via its own pipeline).
### 7.4 Update Management
```bash
operations-center provisioning update list
operations-center provisioning update show
operations-center provisioning update refresh # Check for new updates
operations-center provisioning update assign-channels --channel
```
**Channel management**: updates auto-link to channels based on upstream
`filter_expression`. Use `assign-channels` to explicitly control which
updates appear in which channels.
### 7.5 Web UI
Navigate to `https://192.168.102.120:8443/ui/` in Firefox (with client.pfx
imported). The web UI shows: servers, clusters, instances, networks, storage
pools, load balancers, network forwards, and update status.
---
## Section 8: Cluster Lifecycle
### 8.1 Evacuation & Restore
```bash
incus cluster evacuate oc-node-01:oc-node-01 --force
# OC still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
# Inventory shows workloads moved to other nodes
incus cluster restore oc-node-01:oc-node-01 --force
```
### 8.2 Node Failure Simulation
Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s).
Node auto-rejoins after Proxmox restart (~60s boot):
```bash
# OC does NOT detect node failures
# provisioning server show still reports "Status: ready" for offline nodes
```
### 8.3 Reboot Considerations
**WARNING**: `operations-center provisioning server system reboot` can break
OC-managed nodes on Proxmox. The OC agent pushes config via the IncusOS REST
API that gets persisted to `state.txt`. On reboot, invalid values crash the
daemon. See CLAUDE.md for full analysis.
**Safe maintenance procedure**:
1. `incus cluster evacuate : --force`
2. Proxmox stop the VM
3. Proxmox start the VM
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore : --force`
---
## Section 9: Cleanup
**Note**: this section documents the cleanup procedure but is NOT executed.
The lab is left fully running with all workloads, OVN networking, and load
balancer active for exploration via OC web UI and CLI.
```bash
# Delete workloads
incus delete oc-node-01:ha-web-01 oc-node-01:ha-web-02 oc-node-01:ha-web-03 --force
# Delete load balancer and forward
incus network load-balancer delete oc-node-01:net-prod 192.168.103.201
incus network forward delete oc-node-01:net-prod 192.168.103.202
# Delete OVN network
incus network delete oc-node-01:net-prod
# Delete OVN control plane
incus delete oc-node-01:ovn-central --force
# Delete UPLINK
incus network delete oc-node-01:UPLINK
# Disable OVN services on all nodes
for remote in oc-node-01 oc-node-02 oc-node-03; do
incus query ${remote}:/os/1.0/services/ovn --request PUT --data '{
"config": {"enabled": false}, "state": {}
}'
done
# Remove OVN chassis role
for node in oc-node-01 oc-node-02 oc-node-03; do
incus cluster role remove oc-node-01:${node} ovn-chassis
done
# Destroy infrastructure (--yes to skip confirmation prompts)
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml
# Remove OC remote
operations-center remote remove oc-lab
```
---
## Section 10: Verification Checklist
| # | Check | Status |
|---|-------|--------|
| 1 | OC server deployed and reachable | PASS |
| 2 | OC remote configured | PASS |
| 3 | Updates downloaded and ready | PASS |
| 4 | 3 nodes provisioned from OC ISO | PASS |
| 5 | All nodes auto-registered with OC | PASS |
| 6 | All nodes "up to date" (not "update pending") | PASS |
| 7 | No crontab bug (force_reboot not used) | PASS |
| 8 | Cluster formed via OC `provisioning cluster add` | PASS |
| 9 | All 3 nodes ONLINE + Fully operational | PASS |
| 10 | OVN cross-node connectivity (sub-ms) | PASS |
| 11 | 3 nginx instances on 3 nodes | PASS |
| 12 | LB distributes across all backends | PASS |
| 13 | HA: LB serves with one backend down | PASS |
| 14 | Network forward: direct backend access | PASS |
| 15 | OC inventory shows all resources | PASS |
| 16 | OC web UI accessible | PASS |
---
## Section 11: Quick Reference
### OC Provisioning Workflow
```
1. Deploy OC server (incusos-proxmox)
2. Add OC remote (operations-center remote add)
3. Wait for updates (provisioning update list -- at least one "ready")
4. Create token (provisioning token add)
5. Create token seed (provisioning token seed add) -- NO force_reboot
6. Create old-stable channel (provisioning channel add old-stable)
7. Assign older update to old-stable (provisioning update assign-channels)
8. Generate ISO from OLDER channel (token seed get-image --channel old-stable)
9. Deploy nodes (incusos-proxmox --iso)
10. Wait for auto-registration + update delivery
11. Form cluster (provisioning cluster add)
12. Set up OVN networking (OVN container + services + UPLINK + net-prod)
```
### Key OC CLI Commands
| Command | Description |
|---------|-------------|
| `remote add --auth-type tls` | Add OC remote |
| `remote switch ` | Set default remote |
| `admin os show` | IncusOS hostname, version, uptime |
| `provisioning token add --uses N` | Create provisioning token |
| `provisioning token seed add ` | Attach seed to token |
| `provisioning token seed get-image --channel ` | Generate ISO |
| `provisioning channel add --description ` | Create provisioning channel |
| `provisioning update list` | List available updates |
| `provisioning update refresh` | Check for new updates |
| `provisioning update assign-channels --channel ` | Assign update to channel |
| `provisioning server list` | List registered servers |
| `provisioning server resync ` | Resync server state |
| `provisioning cluster add --server-names ... -a ` | Form cluster |
| `provisioning cluster resync ` | Force inventory resync |
| `provisioning cluster list` | List clusters |
| `inventory instance list --cluster ` | List instances |
| `inventory network list --cluster ` | List networks |
| `inventory network-load-balancer list` | List load balancers |
| `inventory network-forward list` | List network forwards |
| `inventory storage-pool list` | List storage pools |
### Incus Cluster Commands (via remote)
| Command | Description |
|---------|-------------|
| `incus cluster list :` | List cluster members |
| `incus cluster role add : ovn-chassis` | Add OVN role |
| `incus cluster evacuate : --force` | Evacuate member |
| `incus cluster restore : --force` | Restore member |
| `incus network create : --type=physical --target= parent=mgmt` | Per-target network |
| `incus network create : --type=ovn network=UPLINK` | OVN network |
| `incus network load-balancer create : ` | Create LB |
| `incus network forward create : ` | Create forward |
---
## Section 12: Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| `needs_update: true` blocks clustering | Nodes deployed from ISO matching latest version | Deploy from older ISO (`--channel old-stable`), let OC push the update |
| Terraform "Certificate already in trust store" | Cert in both SEED_DATA and app-seed-config | Use empty `{}` for app-seed-config when cert is in SEED_DATA |
| Terraform "Network/Pool not in pending state" | `apply_defaults: true` pre-created resources | Use `apply_defaults: false` (recommended) |
| TLS cert error after clustering | Cluster cert regenerated | `incus remote remove/add` with `--accept-certificate` |
| `get-image` fails | No updates in `ready` state | Wait ~5 min for OC to download; `provisioning update list` |
| Node doesn't self-register | Bad ISO or token exhausted | Verify ISO from `get-image`; check `token list` |
| Token seed fields all `{}` | Flat YAML format | Use structured: `install:` section key |
| OC web UI 403 | Client cert not imported | Import client.pfx into browser |
| OC shows "ready" for offline node | OC doesn't detect failures | Check via `incus cluster list` instead |
| OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot |
| UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide |
| Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs |
| `assign-channels` "Not found" | Channel doesn't exist yet | Run `provisioning channel add old-stable` first |
| Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` |
---
## Section 13: Known Limitations (v0.3.0)
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
- **No cluster member state tracking**: OC always shows `ready` for all nodes
- **Stale server entries**: out-of-band changes create stale entries
- **needs_update tracking**: based on OC's delivery pipeline, not version comparison
- **OC reboot breaks Proxmox VMs**: OC agent boot failures on restart
- **No cluster resize**: cannot add/remove members from existing OC clusters
- **Token seeds require structured YAML**: section keys, not flat format
- **OVN LB has no health checks**: connection hashing distributes to dead backends
---
## References
- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
- [IncusOS GitHub](https://github.com/lxc/incus-os)
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
- [Clustering guide](clustering-guide.md) -- manual cluster formation
- [Networking guide](networking-guide.md) -- OVN overlay tutorial
- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA