# Operations Center Guide for IncusOS End-to-end guide for deploying an OC-managed Incus cluster with OVN overlay networking and an HA nginx workload. All commands and output captured from a real deployment on 2026-02-23 using Operations Center v0.3.0 and IncusOS build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel). --- ## Section 0: Architecture Overview ``` ┌──────────────────────────────────────────┐ │ Proxmox VE Host (pve) │ │ Intel i9-13900HK 64 GiB 20 cores │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐│ │ │oc-node-01│ │oc-node-02│ │oc-node-03││ │ │ VMID 400 │ │ VMID 401 │ │ VMID 402 ││ │ │ 4c/8G/64G│ │ 4c/8G/50G│ │ 4c/8G/50G││ │ │ .140 │ │ .141 │ │ .142 ││ │ │ │ │ │ │ ││ │ │ovn-central│ │ │ │ ││ │ │ha-web-01 │ │ha-web-02 │ │ha-web-03 ││ │ └──────────┘ └──────────┘ └──────────┘│ │ │ │ │ │ │ └────────────┼────────────┘ │ │ OVN Geneve tunnels │ │ net-prod 10.10.10.0/24 │ │ │ │ ┌──────────┐ │ │ │oc-server │ OC Management │ │ │ VMID 920 │ https://192.168.102.120 │ │ │ 2c/4G/50G│ Web UI + CLI + REST API │ │ │ .120 │ │ │ └──────────┘ │ │ │ │ vmbr0 (VLAN 69) ── 192.168.100.0/22 │ └──────────────────────────────────────────┘ │ │ gateway 192.168.100.1 │ OVN external IPs: 192.168.103.200-210 LB: 192.168.103.201 Fwd: 192.168.103.202 ``` ### Infrastructure | Component | VMID | IP | Cores | RAM | Disk | Role | |-----------|------|-----|-------|-----|------|------| | oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center | | oc-node-01 | 400 | 192.168.102.140/22 | 4 | 8 GiB | 64G | Cluster init + OVN central | | oc-node-02 | 401 | 192.168.102.141/22 | 4 | 8 GiB | 50G | Cluster member | | oc-node-03 | 402 | 192.168.102.142/22 | 4 | 8 GiB | 50G | Cluster member | **RAM budget**: 28 GiB of 64 GiB (44%). VLAN 69, gateway 192.168.100.1. ### Decision Rationale - **OC-managed cluster**: OC is the future default clustering path for Incus. Supports provisioning, update management, inventory, and centralized control. - **OVN overlay**: cross-node L2 connectivity for HA workloads. Sub-ms latency. - **HA nginx**: demonstrates load balancing, failover, and network forwards. - **Static IPs**: required for VLAN 69 (ARP-based detection doesn't work across VLANs). - **apply_defaults: false**: recommended for OC-managed nodes. OC's Terraform handles resource creation during cluster formation. ### Cross-References - [Clustering guide](clustering-guide.md) -- manual cluster formation reference - [Networking guide](networking-guide.md) -- OVN overlay tutorial - [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA --- ## Section 1: Prerequisites ### Required Tools ```bash # Operations Center CLI operations-center --version # operations-center version 0.3.0 # Incus client incus version # Client version: 6.12 # incusos-proxmox (from this repo) ./incusos/incusos-proxmox --doctor ``` ### Client Certificates OC uses the same TLS client certificate format as Incus: ```bash # Copy Incus certs for OC CLI mkdir -p ~/.config/operations-center cp ~/.config/incus/client.crt ~/.config/operations-center/ cp ~/.config/incus/client.key ~/.config/operations-center/ # Generate PKCS#12 for browser access to OC web UI openssl pkcs12 -export \ -inkey ~/.config/incus/client.key \ -in ~/.config/incus/client.crt \ -out ~/.config/incus/client.pfx # Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import ``` --- ## Section 2: Deploy OC Server ### Configuration File ```yaml # incusos/examples/lab-oc-deploy.yaml defaults: cores: 2 memory: 4096 disk: 50 start_vmid: 920 proxmox: gateway: 192.168.100.1 dns: 192.168.100.1 vms: - name: oc-server app: operations-center apply_defaults: true ip: 192.168.102.120/22 ``` ### Deploy ```bash ./incusos/incusos-proxmox --yes examples/lab-oc-deploy.yaml ``` Actual output (key lines): ``` [ok] VM 'oc-server' created (VMID 920) [ok] VM 'oc-server' installed and running at 192.168.102.120 ``` ### Set Up OC CLI Remote ```bash operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls operations-center remote switch oc-lab ``` ### Verify OC ```bash operations-center admin os show ``` Actual output: ``` environment: hostname: oc-server os_name: IncusOS os_version: "202602230420" os_version_next: "" uptime: 4956 ``` ### Wait for Updates OC downloads IncusOS update packages from upstream. At least one update must reach `ready` state before ISOs can be generated: ```bash # Poll until at least one update shows "ready" operations-center provisioning update list ``` Actual output (after ~5 minutes): ``` +--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ | UUID | Origin | Channels | Upstream Channels | Version | Sev | Status | +--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ | 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | stable, testing | 202602200553 | none | ready | | 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | stable, testing | 202602210344 | none | ready | | c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | stable, testing | 202602230420 | none | ready | +--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ ``` --- ## Section 3: Provision Nodes ### 3.1 Create Provisioning Token ```bash operations-center provisioning token add --uses 5 --description "OC-managed cluster v2" operations-center provisioning token list ``` Actual output: ``` +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ | UUID | Uses Remaining | Expire At | Channel | Description | +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ | 2ba12420-fee5-4d55-9f2f-40809e9ad5de | 5 | 2026-03-25 10:29:39 +0100 CET | stable | OC-managed cluster v2 | +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ ``` ### 3.2 Create Token Seed (No force_reboot) **Critical**: the token seed must NOT include `force_reboot`. On Proxmox, `incusos-proxmox` handles the install lifecycle externally (blockstat detection + media removal). `force_reboot` triggers SysRq-B which causes the ~50% crontab bug (see CLAUDE.md for full investigation). ```yaml # /tmp/oc-preseed.yaml install: version: "1" force_install: true ``` **Important**: use the structured format with section keys (`install:`). A flat format (`version: "1"` at root) maps fields to empty `{}` and they don't get assigned to any section. ```bash operations-center provisioning token seed add proxmox-preseed \ /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox" ``` ### 3.3 Generate OC-Provisioned ISO (Older Version) **Critical discovery**: nodes deployed from an ISO matching the latest OC update version are tracked as `needs_update: true` by OC because the OS was never delivered through OC's update pipeline. The fix: generate the ISO from an older channel so OC can push the real update after deployment. ```bash # Assign the second-latest update to a separate channel operations-center provisioning update assign-channels --channel old-stable # Generate ISO from the older channel operations-center provisioning token seed get-image proxmox-preseed \ /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable ``` Actual output: ``` Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso" ``` The ISO contains IncusOS 202602210344 (one version behind). OC will push the latest (202602230420) after nodes register. ### 3.4 Node Configuration ```yaml # incusos/examples/lab-oc-nodes.yaml defaults: cores: 4 memory: 8192 disk: 50 start_vmid: 400 proxmox: gateway: 192.168.100.1 dns: 192.168.100.1 vms: - name: oc-node-01 app: incus apply_defaults: false disk: 64 ip: 192.168.102.140/22 - name: oc-node-02 app: incus apply_defaults: false ip: 192.168.102.141/22 - name: oc-node-03 app: incus apply_defaults: false ip: 192.168.102.142/22 ``` ### 3.5 Deploy Nodes (Hybrid Approach) The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration (from the boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA (hostname, static IP), install monitoring, and media cleanup. ```bash ./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-oc-nodes.yaml ``` Actual output (key lines): ``` [ok] ISO uploaded: IncusOS-oc.iso [ok] VM 'oc-node-01' installed and running at 192.168.102.140 [ok] Remote 'oc-node-01' added (192.168.102.140) [ok] VM 'oc-node-02' installed and running at 192.168.102.141 [ok] Remote 'oc-node-02' added (192.168.102.141) [ok] VM 'oc-node-03' installed and running at 192.168.102.142 [ok] Remote 'oc-node-03' added (192.168.102.142) [ok] All post-deployment checks passed ``` All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug. ### 3.6 Verify Auto-Registration Nodes auto-register with OC within ~30 seconds of first boot. The update from 202602210344 to 202602230420 happens automatically: ```bash operations-center provisioning server list ``` Actual output: ``` +------------+-------------------+------------------------------+--------+-------------+ | Cluster | Name | Connection URL | Status | Update Stat | +------------+-------------------+------------------------------+--------+-------------+ | | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | | | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | | | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | | | operations-center | https://[::1]:8443 | ready | update pend | +------------+-------------------+------------------------------+--------+-------------+ ``` **Key**: all 3 nodes show **"up to date"** because OC delivered the 202602230420 update through its pipeline. This is what unlocks clustering. ### 3.7 Verify Scrub Schedules ```bash for node in oc-node-01 oc-node-02 oc-node-03; do incus query ${node}:/os/1.0/system/storage | python3 -c \ "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])" done ``` Actual output: ``` oc-node-01: 0 4 * * 0 oc-node-02: 0 4 * * 0 oc-node-03: 0 4 * * 0 ``` All healthy. No crontab bug (force_reboot was not used). --- ## Section 4: Form Cluster via Operations Center ### 4.1 The needs_update Blocker OC requires all nodes to show `needs_update: false` before clustering. Nodes deployed from an ISO matching the latest version are tracked as `needs_update: true` because the OS was never delivered through OC's update pipeline. The `needs_update` flag is server-side computed and cannot be overridden via REST API PUT. **Solution**: deploy from an older ISO version (Section 3.3). OC then pushes the real update to nodes through its pipeline, clearing the flag. ### 4.2 Form Cluster **Important**: if the client certificate was already injected via SEED_DATA, use an empty application seed config to avoid "Certificate already in trust store" Terraform errors: ```bash echo '{}' > /tmp/oc-app-config.yaml operations-center provisioning cluster add oc-cluster \ https://192.168.102.140:8443 \ --server-names oc-node-01,oc-node-02,oc-node-03 \ --server-type incus \ --application-seed-config /tmp/oc-app-config.yaml ``` OC orchestrates the full cluster formation: 1. Sets `core.https_address` to each node's specific IP 2. Enables clustering on oc-node-01 3. Joins oc-node-02 and oc-node-03 4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`) 5. Runs Terraform/OpenTofu for post-cluster configuration ### 4.3 Fix Remotes After Clustering Clustering regenerates TLS certificates. Re-add the remotes: ```bash incus remote remove oc-node-01 incus remote remove oc-node-02 incus remote remove oc-node-03 incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate ``` ### 4.4 Verify Cluster ```bash incus cluster list oc-node-01: ``` Actual output: ``` +------------+------------------------------+-----------------+--------------+--------+-------------------+ | NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE | +------------+------------------------------+-----------------+--------------+--------+-------------------+ | oc-node-01 | https://192.168.102.140:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | | | database-leader | | | | | | | database | | | | +------------+------------------------------+-----------------+--------------+--------+-------------------+ | oc-node-02 | https://192.168.102.141:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | | | database | | | | +------------+------------------------------+-----------------+--------------+--------+-------------------+ | oc-node-03 | https://192.168.102.142:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | | | database | | | | +------------+------------------------------+-----------------+--------------+--------+-------------------+ ``` All 3 nodes ONLINE and Fully operational. ### 4.5 Cluster Resources Created by OC ```bash incus storage list oc-node-01: incus network list oc-node-01: ``` Actual output: ``` +-------+--------+--------------------------------------+---------+---------+ | NAME | DRIVER | DESCRIPTION | USED BY | STATE | +-------+--------+--------------------------------------+---------+---------+ | local | zfs | Local storage pool (on system drive) | 8 | CREATED | +-------+--------+--------------------------------------+---------+---------+ +----------+--------+---------+----------------+------------------------------+---------+ | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | +----------+--------+---------+----------------+------------------------------+---------+ | incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | | meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | +----------+--------+---------+----------------+------------------------------+---------+ ``` OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and `meshbr0` (OC-specific mesh network for inter-node communication). --- ## Section 5: OVN Overlay Networking ### 5.1 Deploy OVN Control Plane Launch an `ovn-central` container on oc-node-01: ```bash incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01 incus exec oc-node-01:ovn-central -- bash -c \ "apt-get update -qq && apt-get install -y -qq ovn-central" ``` Configure OVN to listen on all interfaces: ```bash incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0 incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0 ``` Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP: ```bash incus config device add oc-node-01:ovn-central \ nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641 incus config device add oc-node-01:ovn-central \ sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642 ``` ### 5.2 Enable OVN on All IncusOS Nodes Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`). The `database` field is the **southbound** DB (port 6642), not northbound. ```bash for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')" incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{ \"config\": { \"database\": \"tcp:192.168.102.140:6642\", \"enabled\": true, \"tunnel_address\": \"${node_ip}\", \"tunnel_protocol\": \"geneve\" }, \"state\": {} }" done ``` ### 5.3 Configure Incus OVN Connection ```bash incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641 ``` ### 5.4 Assign OVN Chassis Role ```bash for node in oc-node-01 oc-node-02 oc-node-03; do incus cluster role add oc-node-01:${node} ovn-chassis done ``` ### 5.5 Create UPLINK Physical Network Uses `parent=mgmt` (IncusOS management NIC). Two-step cluster pattern: ```bash # Per-target (parent is member-specific) for node in oc-node-01 oc-node-02 oc-node-03; do incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt done # Cluster-wide config incus network create oc-node-01:UPLINK --type=physical \ ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \ ipv4.gateway=192.168.100.1/22 \ dns.nameservers=192.168.100.1 ``` ### 5.6 Create OVN Network (net-prod) ```bash incus network create oc-node-01:net-prod --type=ovn \ network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true ``` Actual output: ``` Network net-prod created ``` net-prod is assigned external IP `192.168.103.200` from the UPLINK range. ### 5.7 Verify Cross-Node OVN Connectivity ```bash incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3 ``` Actual output: ``` 64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.744 ms 64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.169 ms 64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.150 ms ``` Sub-millisecond cross-node latency via Geneve tunnels. Clean up test containers after verification. ### 5.8 Final Network State ```bash incus network list oc-node-01: ``` Actual output: ``` +----------+----------+---------+----------------+------------------------------+---------+ | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | +----------+----------+---------+----------------+------------------------------+---------+ | UPLINK | physical | YES | | | 1 | | incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | | meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | | net-prod | ovn | YES | 10.10.10.1/24 | | 3 | +----------+----------+---------+----------------+------------------------------+---------+ ``` --- ## Section 6: HA Nginx Workload ### 6.1 Launch Nginx Containers Launch 3 nginx containers across all nodes on net-prod: ```bash incus launch images:debian/12 oc-node-01:ha-web-01 --target oc-node-01 -n net-prod incus launch images:debian/12 oc-node-01:ha-web-02 --target oc-node-02 -n net-prod incus launch images:debian/12 oc-node-01:ha-web-03 --target oc-node-03 -n net-prod ``` ### 6.2 Install Nginx and Set Distinct Content ```bash for i in 01 02 03; do incus exec oc-node-01:ha-web-${i} -- bash -c \ "apt-get update -qq && apt-get install -y -qq nginx" incus exec oc-node-01:ha-web-${i} -- bash -c \ "echo '

ha-web-${i} on oc-node-${i}

' > /var/www/html/index.html" incus exec oc-node-01:ha-web-${i} -- systemctl start nginx done ``` ### 6.3 Create OVN Load Balancer ```bash incus network load-balancer create oc-node-01:net-prod 192.168.103.201 # Add backends (must use IP addresses, not instance names) incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \ ha-web-01 10.10.10.2 80 incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \ ha-web-02 10.10.10.3 80 incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \ ha-web-03 10.10.10.4 80 # Add port forwarding rule incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \ tcp 80 ha-web-01,ha-web-02,ha-web-03 ``` ### 6.4 Test Load Balancer ```bash for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done ``` Actual output: ```

ha-web-03 on oc-node-03

ha-web-02 on oc-node-02

ha-web-02 on oc-node-02

ha-web-01 on oc-node-01

ha-web-02 on oc-node-02

``` Traffic distributes across all 3 backends (connection-based hashing, not round-robin). ### 6.5 HA Failover Test ```bash # Stop ha-web-01 incus stop oc-node-01:ha-web-01 sleep 2 # Test with one backend down for i in 1 2 3 4 5; do curl -s --connect-timeout 5 http://192.168.103.201; done ``` Actual result: remaining backends (ha-web-02, ha-web-03) serve traffic. Some requests to the dead backend return empty (OVN LB uses connection hashing, not health checks — no active health monitoring). ```bash # Restart ha-web-01 incus start oc-node-01:ha-web-01 sleep 3 # All 3 backends serve again for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done ``` Actual output after restart: ```

ha-web-03 on oc-node-03

ha-web-01 on oc-node-01

ha-web-03 on oc-node-03

ha-web-02 on oc-node-02

ha-web-02 on oc-node-02

ha-web-02 on oc-node-02

``` All 3 backends recovered. ### 6.6 Network Forward (Direct Backend Access) ```bash incus network forward create oc-node-01:net-prod 192.168.103.202 incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8080 10.10.10.2 80 incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8081 10.10.10.3 80 incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8082 10.10.10.4 80 ``` Test direct access: ```bash curl -s http://192.168.103.202:8080 # ha-web-01 curl -s http://192.168.103.202:8081 # ha-web-02 curl -s http://192.168.103.202:8082 # ha-web-03 ``` Actual output: ```

ha-web-01 on oc-node-01

ha-web-02 on oc-node-02

ha-web-03 on oc-node-03

``` ### 6.7 Final Instance State ```bash incus list oc-node-01: ``` Actual output: ``` +-------------+---------+----------------------+-----------+------------+ | NAME | STATE | IPV4 | TYPE | LOCATION | +-------------+---------+----------------------+-----------+------------+ | ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 | | ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 | | ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 | | ovn-central | RUNNING | 10.149.76.198 (eth0) | CONTAINER | oc-node-01 | +-------------+---------+----------------------+-----------+------------+ ``` --- ## Section 7: OC Management ### 7.1 Inventory Sync OC inventory is **not real-time** — requires explicit `cluster resync`: ```bash operations-center provisioning cluster resync oc-cluster ``` ### 7.2 Inventory Browsing ```bash operations-center inventory instance list --cluster oc-cluster ``` Actual output: ``` +--------------------------------------+-------------+---------+------------+------------+ | UUID | Name | Project | Cluster | Server | +--------------------------------------+-------------+---------+------------+------------+ | cf27717e-5f87-5a22-9e82-cc6616fd665d | ha-web-01 | default | oc-cluster | oc-node-01 | | 657447d3-7773-5005-af07-b62d9f789ffc | ha-web-02 | default | oc-cluster | oc-node-02 | | 7b79cab9-37f9-50d8-9d4f-20daafd3d932 | ha-web-03 | default | oc-cluster | oc-node-03 | | 17133a45-6318-5aa5-b6a8-fc5a5871f679 | ovn-central | default | oc-cluster | oc-node-01 | +--------------------------------------+-------------+---------+------------+------------+ ``` ```bash operations-center inventory network list --cluster oc-cluster ``` Actual output: ``` +--------------------------------------+----------+----------+---------+------------+ | UUID | Name | Type | Project | Cluster | +--------------------------------------+----------+----------+---------+------------+ | 628df562-c980-5c27-8268-2ba69211eb7f | UPLINK | physical | default | oc-cluster | | e35ecd83-25c7-54ae-8fd9-66bc241a002f | incusbr0 | bridge | default | oc-cluster | | 9cc58dd0-9821-5649-807a-88e0cbce8f5c | meshbr0 | bridge | default | oc-cluster | | 9c3ee7cb-6b7e-5ea9-bf91-e8d44f0f04da | net-prod | ovn | default | oc-cluster | +--------------------------------------+----------+----------+---------+------------+ ``` ```bash operations-center inventory network-load-balancer list operations-center inventory network-forward list operations-center inventory storage-pool list ``` All resources visible in OC inventory after resync. ### 7.3 Server Management ```bash operations-center provisioning server list ``` Actual output: ``` +------------+-------------------+------------------------------+--------+-------------+ | Cluster | Name | Connection URL | Status | Update Stat | +------------+-------------------+------------------------------+--------+-------------+ | oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | | oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | | oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | | | operations-center | https://[::1]:8443 | ready | update pend | +------------+-------------------+------------------------------+--------+-------------+ ``` ### 7.4 Update Management ```bash operations-center provisioning update list operations-center provisioning update show operations-center provisioning update refresh # Check for new updates operations-center provisioning update assign-channels --channel ``` **Channel management**: updates auto-link to channels based on upstream `filter_expression`. Use `assign-channels` to explicitly control which updates appear in which channels. ### 7.5 Web UI Navigate to `https://192.168.102.120:8443/ui/` in Firefox (with client.pfx imported). The web UI shows: servers, clusters, instances, networks, storage pools, load balancers, network forwards, and update status. --- ## Section 8: Cluster Lifecycle ### 8.1 Evacuation & Restore ```bash incus cluster evacuate oc-node-01:oc-node-01 --force # OC still shows oc-node-01 as "ready" (doesn't track EVACUATED state) operations-center provisioning cluster resync oc-cluster # Inventory shows workloads moved to other nodes incus cluster restore oc-node-01:oc-node-01 --force ``` ### 8.2 Node Failure Simulation Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s). Node auto-rejoins after Proxmox restart (~60s boot): ```bash # OC does NOT detect node failures # provisioning server show still reports "Status: ready" for offline nodes ``` ### 8.3 Reboot Considerations **WARNING**: `operations-center provisioning server system reboot` can break OC-managed nodes on Proxmox. The OC agent pushes config via the IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid values crash the daemon. See CLAUDE.md for full analysis. **Safe maintenance procedure**: 1. `incus cluster evacuate : --force` 2. Proxmox stop the VM 3. Proxmox start the VM 4. Wait for IncusOS to boot (~60-90s) 5. `incus cluster restore : --force` --- ## Section 9: Cleanup **Note**: this section documents the cleanup procedure but is NOT executed. The lab is left fully running with all workloads, OVN networking, and load balancer active for exploration via OC web UI and CLI. ```bash # Delete workloads incus delete oc-node-01:ha-web-01 oc-node-01:ha-web-02 oc-node-01:ha-web-03 --force # Delete load balancer and forward incus network load-balancer delete oc-node-01:net-prod 192.168.103.201 incus network forward delete oc-node-01:net-prod 192.168.103.202 # Delete OVN network incus network delete oc-node-01:net-prod # Delete OVN control plane incus delete oc-node-01:ovn-central --force # Delete UPLINK incus network delete oc-node-01:UPLINK # Disable OVN services on all nodes for remote in oc-node-01 oc-node-02 oc-node-03; do incus query ${remote}:/os/1.0/services/ovn --request PUT --data '{ "config": {"enabled": false}, "state": {} }' done # Remove OVN chassis role for node in oc-node-01 oc-node-02 oc-node-03; do incus cluster role remove oc-node-01:${node} ovn-chassis done # Destroy infrastructure ./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-nodes.yaml ./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-deploy.yaml # Remove OC remote operations-center remote remove oc-lab ``` --- ## Section 10: Verification Checklist | # | Check | Status | |---|-------|--------| | 1 | OC server deployed and reachable | PASS | | 2 | OC remote configured | PASS | | 3 | Updates downloaded and ready | PASS | | 4 | 3 nodes provisioned from OC ISO | PASS | | 5 | All nodes auto-registered with OC | PASS | | 6 | All nodes "up to date" (not "update pending") | PASS | | 7 | No crontab bug (force_reboot not used) | PASS | | 8 | Cluster formed via OC `provisioning cluster add` | PASS | | 9 | All 3 nodes ONLINE + Fully operational | PASS | | 10 | OVN cross-node connectivity (sub-ms) | PASS | | 11 | 3 nginx instances on 3 nodes | PASS | | 12 | LB distributes across all backends | PASS | | 13 | HA: LB serves with one backend down | PASS | | 14 | Network forward: direct backend access | PASS | | 15 | OC inventory shows all resources | PASS | | 16 | OC web UI accessible | PASS | --- ## Section 11: Quick Reference ### OC Provisioning Workflow ``` 1. Deploy OC server (incusos-proxmox) 2. Add OC remote (operations-center remote add) 3. Wait for updates (provisioning update list) 4. Create token (provisioning token add) 5. Create token seed (provisioning token seed add) -- NO force_reboot 6. Generate ISO from OLDER channel (token seed get-image --channel old-stable) 7. Deploy nodes (incusos-proxmox --iso) 8. Wait for auto-registration + update delivery 9. Form cluster (provisioning cluster add) 10. Set up OVN networking (OVN container + services + UPLINK + net-prod) ``` ### Key OC CLI Commands | Command | Description | |---------|-------------| | `remote add --auth-type tls` | Add OC remote | | `remote switch ` | Set default remote | | `admin os show` | IncusOS hostname, version, uptime | | `provisioning token add --uses N` | Create provisioning token | | `provisioning token seed add ` | Attach seed to token | | `provisioning token seed get-image --channel ` | Generate ISO | | `provisioning update list` | List available updates | | `provisioning update refresh` | Check for new updates | | `provisioning update assign-channels --channel ` | Assign update to channel | | `provisioning server list` | List registered servers | | `provisioning server resync ` | Resync server state | | `provisioning cluster add --server-names ... -a ` | Form cluster | | `provisioning cluster resync ` | Force inventory resync | | `provisioning cluster list` | List clusters | | `inventory instance list --cluster ` | List instances | | `inventory network list --cluster ` | List networks | | `inventory network-load-balancer list` | List load balancers | | `inventory network-forward list` | List network forwards | | `inventory storage-pool list` | List storage pools | ### Incus Cluster Commands (via remote) | Command | Description | |---------|-------------| | `incus cluster list :` | List cluster members | | `incus cluster role add : ovn-chassis` | Add OVN role | | `incus cluster evacuate : --force` | Evacuate member | | `incus cluster restore : --force` | Restore member | | `incus network create : --type=physical --target= parent=mgmt` | Per-target network | | `incus network create : --type=ovn network=UPLINK` | OVN network | | `incus network load-balancer create : ` | Create LB | | `incus network forward create : ` | Create forward | --- ## Section 12: Troubleshooting | Symptom | Cause | Fix | |---------|-------|-----| | `needs_update: true` blocks clustering | Nodes deployed from ISO matching latest version | Deploy from older ISO (`--channel old-stable`), let OC push the update | | Terraform "Certificate already in trust store" | Cert in both SEED_DATA and app-seed-config | Use empty `{}` for app-seed-config when cert is in SEED_DATA | | Terraform "Network/Pool not in pending state" | `apply_defaults: true` pre-created resources | Use `apply_defaults: false` (recommended) | | TLS cert error after clustering | Cluster cert regenerated | `incus remote remove/add` with `--accept-certificate` | | `get-image` fails | No updates in `ready` state | Wait ~5 min for OC to download; `provisioning update list` | | Node doesn't self-register | Bad ISO or token exhausted | Verify ISO from `get-image`; check `token list` | | Token seed fields all `{}` | Flat YAML format | Use structured: `install:` section key | | OC web UI 403 | Client cert not imported | Import client.pfx into browser | | OC shows "ready" for offline node | OC doesn't detect failures | Check via `incus cluster list` instead | | OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot | | UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide | | Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs | | Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` | --- ## Section 13: Known Limitations (v0.3.0) - **No brownfield adoption**: nodes must boot from OC-provisioned ISO - **No real-time inventory sync**: requires explicit `cluster resync` - **No cluster member state tracking**: OC always shows `ready` for all nodes - **Stale server entries**: out-of-band changes create stale entries - **needs_update tracking**: based on OC's delivery pipeline, not version comparison - **OC reboot breaks Proxmox VMs**: OC agent boot failures on restart - **No cluster resize**: cannot add/remove members from existing OC clusters - **Token seeds require structured YAML**: section keys, not flat format - **OVN LB has no health checks**: connection hashing distributes to dead backends --- ## References - [Operations Center GitHub](https://github.com/FuturFusion/operations-center) - [Operations Center releases](https://github.com/FuturFusion/operations-center/releases) - [IncusOS GitHub](https://github.com/lxc/incus-os) - [Incus documentation](https://linuxcontainers.org/incus/docs/main/) - [Clustering guide](clustering-guide.md) -- manual cluster formation - [Networking guide](networking-guide.md) -- OVN overlay tutorial - [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA