1084 lines
37 KiB
Markdown
1084 lines
37 KiB
Markdown
# Operations Center Guide for IncusOS
|
|
|
|
End-to-end guide for deploying an OC-managed Incus cluster with OVN overlay
|
|
networking and an HA nginx workload. All commands and output captured from a
|
|
real deployment on 2026-02-23 using Operations Center v0.3.0 and IncusOS
|
|
build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).
|
|
|
|
---
|
|
|
|
## Section 0: Architecture Overview
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"]
|
|
subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"]
|
|
n1["oc-node-01<br/>VMID 400 · .140"]
|
|
n2["oc-node-02<br/>VMID 401 · .141"]
|
|
n3["oc-node-03<br/>VMID 402 · .142"]
|
|
end
|
|
oc["Operations Center<br/>VMID 920 · .120"]
|
|
end
|
|
|
|
vmbr0(("VLAN 69<br/>192.168.100.0/22"))
|
|
ext["OVN external IPs<br/>192.168.103.200-210"]
|
|
|
|
proxmox --- vmbr0
|
|
cluster -.->|"external gateway"| ext
|
|
|
|
classDef node fill:#009E73,color:#fff,stroke:#007a5e
|
|
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
|
|
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
|
|
|
|
class n1,n2,n3 node
|
|
class oc mgmt
|
|
class vmbr0,ext network
|
|
|
|
style proxmox fill:#f5f5f5,stroke:#999
|
|
style cluster fill:#e6f5f0,stroke:#009E73
|
|
```
|
|
|
|
### Infrastructure
|
|
|
|
| Component | VMID | IP | Cores | RAM | Disk | Role |
|
|
|-----------|------|-----|-------|-----|------|------|
|
|
| oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center |
|
|
| oc-node-01 | 400 | 192.168.102.140/22 | 4 | 8 GiB | 64G | Cluster init + OVN central |
|
|
| oc-node-02 | 401 | 192.168.102.141/22 | 4 | 8 GiB | 50G | Cluster member |
|
|
| oc-node-03 | 402 | 192.168.102.142/22 | 4 | 8 GiB | 50G | Cluster member |
|
|
|
|
**RAM budget**: 28 GiB of 64 GiB (44%). VLAN 69, gateway 192.168.100.1.
|
|
|
|
### Decision Rationale
|
|
|
|
- **OC-managed cluster**: OC is the future default clustering path for Incus.
|
|
Supports provisioning, update management, inventory, and centralized control.
|
|
- **OVN overlay**: cross-node L2 connectivity for HA workloads. Sub-ms latency.
|
|
- **HA nginx**: demonstrates load balancing, failover, and network forwards.
|
|
- **Static IPs**: required for VLAN 69 (ARP-based detection doesn't work
|
|
across VLANs).
|
|
- **apply_defaults: false**: recommended for OC-managed nodes. OC's Terraform
|
|
handles resource creation during cluster formation.
|
|
|
|
### Cross-References
|
|
|
|
- [Clustering guide](clustering-guide.md) -- manual cluster formation reference
|
|
- [Networking guide](networking-guide.md) -- OVN overlay tutorial
|
|
- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA
|
|
|
|
---
|
|
|
|
## Section 1: Prerequisites
|
|
|
|
### Required Tools
|
|
|
|
```bash
|
|
# Operations Center CLI
|
|
operations-center --version
|
|
# 0.3.0
|
|
|
|
# Incus client
|
|
incus version
|
|
# Client version: 6.21
|
|
|
|
# incusos-proxmox (from this repo)
|
|
./incusos/incusos-proxmox --doctor
|
|
```
|
|
|
|
### Client Certificates
|
|
|
|
OC uses the same TLS client certificate format as Incus:
|
|
|
|
```bash
|
|
# Copy Incus certs for OC CLI
|
|
mkdir -p ~/.config/operations-center
|
|
cp ~/.config/incus/client.crt ~/.config/operations-center/
|
|
cp ~/.config/incus/client.key ~/.config/operations-center/
|
|
|
|
# Generate PKCS#12 for browser access to OC web UI
|
|
openssl pkcs12 -export \
|
|
-inkey ~/.config/incus/client.key \
|
|
-in ~/.config/incus/client.crt \
|
|
-out ~/.config/incus/client.pfx
|
|
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
|
|
```
|
|
|
|
---
|
|
|
|
## Section 2: Deploy OC Server
|
|
|
|
### Configuration File
|
|
|
|
```yaml
|
|
# incusos/examples/lab-oc-deploy.yaml
|
|
defaults:
|
|
cores: 2
|
|
memory: 4096
|
|
disk: 50
|
|
start_vmid: 920
|
|
|
|
proxmox:
|
|
gateway: 192.168.100.1
|
|
dns: 192.168.100.1
|
|
|
|
vms:
|
|
- name: oc-server
|
|
app: operations-center
|
|
apply_defaults: true
|
|
ip: 192.168.102.120/22
|
|
```
|
|
|
|
### Deploy
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
|
|
```
|
|
|
|
Actual output (key lines):
|
|
|
|
```
|
|
[ok] VM 'oc-server' created (VMID 920)
|
|
[ok] VM 'oc-server' installed and running at 192.168.102.120
|
|
```
|
|
|
|
### Set Up OC CLI Remote
|
|
|
|
```bash
|
|
# Accept the TLS certificate when prompted
|
|
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
|
|
operations-center remote switch oc-lab
|
|
```
|
|
|
|
### Verify OC
|
|
|
|
```bash
|
|
operations-center admin os show
|
|
```
|
|
|
|
Actual output (uptime will vary):
|
|
|
|
```
|
|
WARNING: The IncusOS API and configuration is subject to change
|
|
|
|
environment:
|
|
hostname: oc-server
|
|
os_name: IncusOS
|
|
os_version: "202602230420"
|
|
os_version_next: ""
|
|
uptime: 63
|
|
```
|
|
|
|
### Wait for Updates
|
|
|
|
OC downloads IncusOS update packages from upstream. At least one update must
|
|
reach `ready` state before ISOs can be generated:
|
|
|
|
```bash
|
|
# Poll until at least one update shows "ready"
|
|
operations-center provisioning update list
|
|
```
|
|
|
|
Actual output (after ~8 minutes; UUIDs are stable across deployments):
|
|
|
|
```
|
|
+--------------------------------------+---------------------+----------+--------------+----------+--------+
|
|
| UUID | Origin | Channels | Version | Severity | Status |
|
|
+--------------------------------------+---------------------+----------+--------------+----------+--------+
|
|
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready |
|
|
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready |
|
|
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready |
|
|
+--------------------------------------+---------------------+----------+--------------+----------+--------+
|
|
```
|
|
|
|
The table also includes `Upstream Channels` and `Published At` columns (omitted
|
|
for width). All 3 updates may not be ready at the same time -- at least one
|
|
`ready` is sufficient to proceed. The latest version (202602230420) typically
|
|
reaches `ready` first.
|
|
|
|
---
|
|
|
|
## Section 3: Provision Nodes
|
|
|
|
### 3.1 Create Provisioning Token
|
|
|
|
```bash
|
|
operations-center provisioning token add --uses 5 --description "OC-managed cluster v2"
|
|
operations-center provisioning token list
|
|
```
|
|
|
|
Actual output (UUID changes every run):
|
|
|
|
```
|
|
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
|
|
| UUID | Uses Remaining | Expire At | Channel | Description |
|
|
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
|
|
| <token-UUID> | 5 | <30 days from now> | stable | OC-managed cluster v2 |
|
|
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+
|
|
```
|
|
|
|
Save the `<token-UUID>` -- you'll need it for the next steps.
|
|
|
|
### 3.2 Create Token Seed (No force_reboot)
|
|
|
|
**Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
|
|
`incusos-proxmox` handles the install lifecycle externally (blockstat
|
|
detection + media removal). `force_reboot` triggers SysRq-B which causes
|
|
the ~50% crontab bug (see CLAUDE.md for full investigation).
|
|
|
|
```yaml
|
|
# /tmp/oc-preseed.yaml
|
|
install:
|
|
version: "1"
|
|
force_install: true
|
|
```
|
|
|
|
**Important**: use the structured format with section keys (`install:`). A flat
|
|
format (`version: "1"` at root) maps fields to empty `{}` and they don't get
|
|
assigned to any section.
|
|
|
|
```bash
|
|
operations-center provisioning token seed add <token-UUID> proxmox-preseed \
|
|
/tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
|
|
```
|
|
|
|
### 3.3 Generate OC-Provisioned ISO (Older Version)
|
|
|
|
**Critical discovery**: nodes deployed from an ISO matching the latest OC
|
|
update version are tracked as `needs_update: true` by OC because the OS
|
|
was never delivered through OC's update pipeline. The fix: generate the ISO
|
|
from an older channel so OC can push the real update after deployment.
|
|
|
|
```bash
|
|
# Create the old-stable channel (must exist before assigning updates to it)
|
|
operations-center provisioning channel add old-stable \
|
|
--description "Older stable versions for initial provisioning"
|
|
|
|
# Assign the second-latest update to the old-stable channel
|
|
# (use the UUID for 202602210344 from `provisioning update list`)
|
|
operations-center provisioning update assign-channels <older-UUID> --channel old-stable
|
|
|
|
# Generate ISO from the older channel
|
|
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
|
|
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"
|
|
```
|
|
|
|
The ISO contains IncusOS 202602210344 (one version behind). OC will push
|
|
the latest (202602230420) after nodes register.
|
|
|
|
### 3.4 Node Configuration
|
|
|
|
```yaml
|
|
# incusos/examples/lab-oc-nodes.yaml
|
|
defaults:
|
|
cores: 4
|
|
memory: 8192
|
|
disk: 50
|
|
start_vmid: 400
|
|
|
|
proxmox:
|
|
gateway: 192.168.100.1
|
|
dns: 192.168.100.1
|
|
|
|
vms:
|
|
- name: oc-node-01
|
|
app: incus
|
|
apply_defaults: false
|
|
disk: 64
|
|
ip: 192.168.102.140/22
|
|
|
|
- name: oc-node-02
|
|
app: incus
|
|
apply_defaults: false
|
|
ip: 192.168.102.141/22
|
|
|
|
- name: oc-node-03
|
|
app: incus
|
|
apply_defaults: false
|
|
ip: 192.168.102.142/22
|
|
```
|
|
|
|
### 3.5 Deploy Nodes (Hybrid Approach)
|
|
|
|
The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration
|
|
(from the boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA
|
|
(hostname, static IP), install monitoring, and media cleanup.
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml
|
|
```
|
|
|
|
Actual output (key lines):
|
|
|
|
```
|
|
[ok] ISO uploaded: IncusOS-oc.iso
|
|
[ok] VM 'oc-node-01' installed and running at 192.168.102.140
|
|
[ok] Remote 'oc-node-01' added (192.168.102.140)
|
|
[ok] VM 'oc-node-02' installed and running at 192.168.102.141
|
|
[ok] Remote 'oc-node-02' added (192.168.102.141)
|
|
[ok] VM 'oc-node-03' installed and running at 192.168.102.142
|
|
[ok] Remote 'oc-node-03' added (192.168.102.142)
|
|
[ok] All post-deployment checks passed
|
|
```
|
|
|
|
All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.
|
|
|
|
### 3.6 Verify Auto-Registration
|
|
|
|
Nodes auto-register with OC within ~30 seconds of first boot. The update
|
|
from 202602210344 to 202602230420 happens automatically:
|
|
|
|
```bash
|
|
operations-center provisioning server list
|
|
```
|
|
|
|
Actual output (key columns; full table includes Type, Channel, Certificate
|
|
Fingerprint, Public Connection URL, Last Updated, Last Seen):
|
|
|
|
```
|
|
+---------+-------------------+------------------------------+--------+----------------+
|
|
| Cluster | Name | Connection URL | Status | Update Status |
|
|
+---------+-------------------+------------------------------+--------+----------------+
|
|
| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
|
|
| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
|
|
| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
|
|
| | operations-center | https://[::1]:8443 | ready | update pending |
|
|
+---------+-------------------+------------------------------+--------+----------------+
|
|
```
|
|
|
|
**Key**: all 3 nodes show **"up to date"** because OC delivered the
|
|
202602230420 update through its pipeline. This is what unlocks clustering.
|
|
Nodes may already be up to date by the time the last node finishes
|
|
deploying -- the update gets pushed while `incusos-proxmox` deploys
|
|
subsequent nodes sequentially.
|
|
|
|
### 3.7 Verify Scrub Schedules
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
incus query ${node}:/os/1.0/system/storage | python3 -c \
|
|
"import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
|
|
done
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
oc-node-01: 0 4 * * 0
|
|
oc-node-02: 0 4 * * 0
|
|
oc-node-03: 0 4 * * 0
|
|
```
|
|
|
|
All healthy. No crontab bug (force_reboot was not used).
|
|
|
|
---
|
|
|
|
## Section 4: Form Cluster via Operations Center
|
|
|
|
### 4.1 The needs_update Blocker
|
|
|
|
OC requires all nodes to show `needs_update: false` before clustering.
|
|
Nodes deployed from an ISO matching the latest version are tracked as
|
|
`needs_update: true` because the OS was never delivered through OC's update
|
|
pipeline. The `needs_update` flag is server-side computed and cannot be
|
|
overridden via REST API PUT.
|
|
|
|
**Solution**: deploy from an older ISO version (Section 3.3). OC then pushes
|
|
the real update to nodes through its pipeline, clearing the flag.
|
|
|
|
### 4.2 Form Cluster
|
|
|
|
**Important**: if the client certificate was already injected via SEED_DATA,
|
|
use an empty application seed config to avoid "Certificate already in trust
|
|
store" Terraform errors:
|
|
|
|
```bash
|
|
echo '{}' > /tmp/oc-app-config.yaml
|
|
|
|
operations-center provisioning cluster add oc-cluster \
|
|
https://192.168.102.140:8443 \
|
|
--server-names oc-node-01,oc-node-02,oc-node-03 \
|
|
--server-type incus \
|
|
--application-seed-config /tmp/oc-app-config.yaml
|
|
```
|
|
|
|
OC orchestrates the full cluster formation:
|
|
1. Sets `core.https_address` to each node's specific IP
|
|
2. Enables clustering on oc-node-01
|
|
3. Joins oc-node-02 and oc-node-03
|
|
4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`)
|
|
5. Runs Terraform/OpenTofu for post-cluster configuration
|
|
|
|
### 4.3 Fix Remotes After Clustering
|
|
|
|
Clustering regenerates TLS certificates. Re-add the remotes:
|
|
|
|
```bash
|
|
incus remote remove oc-node-01
|
|
incus remote remove oc-node-02
|
|
incus remote remove oc-node-03
|
|
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
|
|
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
|
|
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
|
|
```
|
|
|
|
### 4.4 Verify Cluster
|
|
|
|
```bash
|
|
incus cluster list oc-node-01:
|
|
```
|
|
|
|
Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):
|
|
|
|
```
|
|
+------------+------------------------------+-----------------+--------------+--------+-------------------+
|
|
| NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE |
|
|
+------------+------------------------------+-----------------+--------------+--------+-------------------+
|
|
| oc-node-01 | https://192.168.102.140:8443 | database-leader | x86_64 | ONLINE | Fully operational |
|
|
| | | database | | | |
|
|
+------------+------------------------------+-----------------+--------------+--------+-------------------+
|
|
| oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational |
|
|
+------------+------------------------------+-----------------+--------------+--------+-------------------+
|
|
| oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational |
|
|
+------------+------------------------------+-----------------+--------------+--------+-------------------+
|
|
```
|
|
|
|
All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added
|
|
later in Section 5.4.
|
|
|
|
### 4.5 Cluster Resources Created by OC
|
|
|
|
```bash
|
|
incus storage list oc-node-01:
|
|
incus network list oc-node-01:
|
|
```
|
|
|
|
Actual output (incusbr0 subnet varies per deployment):
|
|
|
|
```
|
|
+-------+--------+--------------------------------------+---------+---------+
|
|
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
|
|
+-------+--------+--------------------------------------+---------+---------+
|
|
| local | zfs | Local storage pool (on system drive) | 8 | CREATED |
|
|
+-------+--------+--------------------------------------+---------+---------+
|
|
|
|
+----------+--------+---------+-----------------+------------------------------+---------+
|
|
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|
|
+----------+--------+---------+-----------------+------------------------------+---------+
|
|
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 |
|
|
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
|
|
+----------+--------+---------+-----------------+------------------------------+---------+
|
|
```
|
|
|
|
OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
|
|
`meshbr0` (OC-specific mesh network for inter-node communication). The
|
|
table also includes IPv6 and STATE columns.
|
|
|
|
---
|
|
|
|
## Section 5: OVN Overlay Networking
|
|
|
|
### 5.1 Deploy OVN Control Plane
|
|
|
|
Launch an `ovn-central` container on oc-node-01:
|
|
|
|
```bash
|
|
incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
|
|
incus exec oc-node-01:ovn-central -- bash -c \
|
|
"apt-get update -qq && apt-get install -y -qq ovn-central"
|
|
```
|
|
|
|
Configure OVN to listen on all interfaces:
|
|
|
|
```bash
|
|
incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
|
|
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
|
|
```
|
|
|
|
Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:
|
|
|
|
```bash
|
|
incus config device add oc-node-01:ovn-central \
|
|
nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
|
|
incus config device add oc-node-01:ovn-central \
|
|
sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642
|
|
```
|
|
|
|
### 5.2 Enable OVN on All IncusOS Nodes
|
|
|
|
Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`).
|
|
The `database` field is the **southbound** DB (port 6642), not northbound.
|
|
|
|
```bash
|
|
for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
|
|
remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
|
|
incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
|
|
\"config\": {
|
|
\"database\": \"tcp:192.168.102.140:6642\",
|
|
\"enabled\": true,
|
|
\"tunnel_address\": \"${node_ip}\",
|
|
\"tunnel_protocol\": \"geneve\"
|
|
},
|
|
\"state\": {}
|
|
}"
|
|
done
|
|
```
|
|
|
|
### 5.3 Configure Incus OVN Connection
|
|
|
|
```bash
|
|
incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641
|
|
```
|
|
|
|
### 5.4 Assign OVN Chassis Role
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
incus cluster role add oc-node-01:${node} ovn-chassis
|
|
done
|
|
```
|
|
|
|
### 5.5 Create UPLINK Physical Network
|
|
|
|
Uses `parent=mgmt` (IncusOS management NIC). Two-step cluster pattern:
|
|
|
|
```bash
|
|
# Per-target (parent is member-specific)
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
|
|
done
|
|
|
|
# Cluster-wide config
|
|
incus network create oc-node-01:UPLINK --type=physical \
|
|
ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
|
|
ipv4.gateway=192.168.100.1/22 \
|
|
dns.nameservers=192.168.100.1
|
|
```
|
|
|
|
### 5.6 Create OVN Network (net-prod)
|
|
|
|
```bash
|
|
incus network create oc-node-01:net-prod --type=ovn \
|
|
network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
Network net-prod created
|
|
```
|
|
|
|
net-prod is assigned external IP `192.168.103.200` from the UPLINK range.
|
|
|
|
### 5.7 Verify Cross-Node OVN Connectivity
|
|
|
|
```bash
|
|
incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
|
|
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
|
|
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
|
|
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
|
|
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms
|
|
```
|
|
|
|
Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
|
|
containers after verification:
|
|
|
|
```bash
|
|
incus delete oc-node-01:test-1 oc-node-01:test-2 --force
|
|
```
|
|
|
|
### 5.8 Final Network State
|
|
|
|
```bash
|
|
incus network list oc-node-01:
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
+----------+----------+---------+----------------+------------------------------+---------+
|
|
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|
|
+----------+----------+---------+----------------+------------------------------+---------+
|
|
| UPLINK | physical | YES | | | 1 |
|
|
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 |
|
|
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
|
|
| net-prod | ovn | YES | 10.10.10.1/24 | | 0 |
|
|
+----------+----------+---------+----------------+------------------------------+---------+
|
|
```
|
|
|
|
The incusbr0 subnet is randomly assigned per deployment. The USED BY count
|
|
for net-prod is 0 at this point (test containers deleted); it increases as
|
|
workloads are added in Section 6.
|
|
|
|
---
|
|
|
|
## Section 6: HA Nginx Workload
|
|
|
|
### 6.1 Launch Nginx Containers
|
|
|
|
Launch 3 nginx containers across all nodes on net-prod:
|
|
|
|
```bash
|
|
incus launch images:debian/12 oc-node-01:ha-web-01 --target oc-node-01 -n net-prod
|
|
incus launch images:debian/12 oc-node-01:ha-web-02 --target oc-node-02 -n net-prod
|
|
incus launch images:debian/12 oc-node-01:ha-web-03 --target oc-node-03 -n net-prod
|
|
```
|
|
|
|
### 6.2 Install Nginx and Set Distinct Content
|
|
|
|
```bash
|
|
for i in 01 02 03; do
|
|
incus exec oc-node-01:ha-web-${i} -- bash -c \
|
|
"apt-get update -qq && apt-get install -y -qq nginx"
|
|
incus exec oc-node-01:ha-web-${i} -- bash -c \
|
|
"echo '<h1>ha-web-${i} on oc-node-${i}</h1>' > /var/www/html/index.html"
|
|
incus exec oc-node-01:ha-web-${i} -- systemctl start nginx
|
|
done
|
|
```
|
|
|
|
### 6.3 Create OVN Load Balancer
|
|
|
|
```bash
|
|
incus network load-balancer create oc-node-01:net-prod 192.168.103.201
|
|
|
|
# Add backends (must use IP addresses, not instance names)
|
|
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
|
|
ha-web-01 10.10.10.2 80
|
|
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
|
|
ha-web-02 10.10.10.3 80
|
|
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
|
|
ha-web-03 10.10.10.4 80
|
|
|
|
# Add port forwarding rule
|
|
incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \
|
|
tcp 80 ha-web-01,ha-web-02,ha-web-03
|
|
```
|
|
|
|
### 6.4 Test Load Balancer
|
|
|
|
```bash
|
|
for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done
|
|
```
|
|
|
|
Actual output (distribution varies per run due to connection-based hashing):
|
|
|
|
```
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-02 on oc-node-02</h1>
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-02 on oc-node-02</h1>
|
|
```
|
|
|
|
Traffic distributes across backends (connection-based hashing, not
|
|
round-robin). Not all backends may appear in every 5-request sample.
|
|
|
|
### 6.5 HA Failover Test
|
|
|
|
```bash
|
|
# Stop ha-web-01
|
|
incus stop oc-node-01:ha-web-01
|
|
sleep 2
|
|
|
|
# Test with one backend down
|
|
for i in 1 2 3 4 5; do curl -s --connect-timeout 5 http://192.168.103.201; done
|
|
```
|
|
|
|
Actual result: remaining backends (ha-web-02, ha-web-03) serve traffic. Some
|
|
requests to the dead backend return empty (OVN LB uses connection hashing,
|
|
not health checks — no active health monitoring).
|
|
|
|
```bash
|
|
# Restart ha-web-01
|
|
incus start oc-node-01:ha-web-01
|
|
sleep 3
|
|
|
|
# All 3 backends serve again
|
|
for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done
|
|
```
|
|
|
|
Actual output after restart (all 3 backends serving again):
|
|
|
|
```
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-02 on oc-node-02</h1>
|
|
<h1>ha-web-02 on oc-node-02</h1>
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
```
|
|
|
|
All 3 backends recovered.
|
|
|
|
### 6.6 Network Forward (Direct Backend Access)
|
|
|
|
```bash
|
|
incus network forward create oc-node-01:net-prod 192.168.103.202
|
|
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8080 10.10.10.2 80
|
|
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8081 10.10.10.3 80
|
|
incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8082 10.10.10.4 80
|
|
```
|
|
|
|
Test direct access:
|
|
|
|
```bash
|
|
curl -s http://192.168.103.202:8080 # ha-web-01
|
|
curl -s http://192.168.103.202:8081 # ha-web-02
|
|
curl -s http://192.168.103.202:8082 # ha-web-03
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
<h1>ha-web-01 on oc-node-01</h1>
|
|
<h1>ha-web-02 on oc-node-02</h1>
|
|
<h1>ha-web-03 on oc-node-03</h1>
|
|
```
|
|
|
|
### 6.7 Final Instance State
|
|
|
|
```bash
|
|
incus list oc-node-01:
|
|
```
|
|
|
|
Actual output (key columns; full table includes IPv6, SNAPSHOTS):
|
|
|
|
```
|
|
+-------------+---------+-----------------------+-----------+------------+
|
|
| NAME | STATE | IPV4 | TYPE | LOCATION |
|
|
+-------------+---------+-----------------------+-----------+------------+
|
|
| ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 |
|
|
| ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 |
|
|
| ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 |
|
|
| ovn-central | RUNNING | 10.x.x.x (eth0) | CONTAINER | oc-node-01 |
|
|
+-------------+---------+-----------------------+-----------+------------+
|
|
```
|
|
|
|
The ovn-central container is on incusbr0 (random subnet per deployment).
|
|
The ha-web containers are on net-prod (10.10.10.0/24, fixed).
|
|
|
|
---
|
|
|
|
## Section 7: OC Management
|
|
|
|
### 7.1 Inventory Sync
|
|
|
|
OC inventory is **not real-time** — requires explicit `cluster resync`:
|
|
|
|
```bash
|
|
operations-center provisioning cluster resync oc-cluster
|
|
```
|
|
|
|
### 7.2 Inventory Browsing
|
|
|
|
```bash
|
|
operations-center inventory instance list --cluster oc-cluster
|
|
```
|
|
|
|
Actual output (UUIDs are deterministic based on name/project/cluster):
|
|
|
|
```
|
|
+--------------------------------------+-------------+---------+------------+------------+
|
|
| UUID | Name | Project | Cluster | Server |
|
|
+--------------------------------------+-------------+---------+------------+------------+
|
|
| cf27717e-5f87-5a22-9e82-cc6616fd665d | ha-web-01 | default | oc-cluster | oc-node-01 |
|
|
| 657447d3-7773-5005-af07-b62d9f789ffc | ha-web-02 | default | oc-cluster | oc-node-02 |
|
|
| 7b79cab9-37f9-50d8-9d4f-20daafd3d932 | ha-web-03 | default | oc-cluster | oc-node-03 |
|
|
| 17133a45-6318-5aa5-b6a8-fc5a5871f679 | ovn-central | default | oc-cluster | oc-node-01 |
|
|
+--------------------------------------+-------------+---------+------------+------------+
|
|
```
|
|
|
|
The full table also includes a `Last Updated` column.
|
|
|
|
```bash
|
|
operations-center inventory network list --cluster oc-cluster
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
+--------------------------------------+----------+----------+---------+------------+
|
|
| UUID | Name | Type | Project | Cluster |
|
|
+--------------------------------------+----------+----------+---------+------------+
|
|
| 628df562-c980-5c27-8268-2ba69211eb7f | UPLINK | physical | default | oc-cluster |
|
|
| e35ecd83-25c7-54ae-8fd9-66bc241a002f | incusbr0 | bridge | default | oc-cluster |
|
|
| 9cc58dd0-9821-5649-807a-88e0cbce8f5c | meshbr0 | bridge | default | oc-cluster |
|
|
| 9c3ee7cb-6b7e-5ea9-bf91-e8d44f0f04da | net-prod | ovn | default | oc-cluster |
|
|
+--------------------------------------+----------+----------+---------+------------+
|
|
```
|
|
|
|
The full table also includes a `Last Updated` column.
|
|
|
|
```bash
|
|
operations-center inventory network-load-balancer list
|
|
operations-center inventory network-forward list
|
|
operations-center inventory storage-pool list
|
|
```
|
|
|
|
All resources visible in OC inventory after resync.
|
|
|
|
### 7.3 Server Management
|
|
|
|
```bash
|
|
operations-center provisioning server list
|
|
```
|
|
|
|
Actual output (key columns):
|
|
|
|
```
|
|
+------------+-------------------+------------------------------+--------+----------------+
|
|
| Cluster | Name | Connection URL | Status | Update Status |
|
|
+------------+-------------------+------------------------------+--------+----------------+
|
|
| oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
|
|
| oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
|
|
| oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
|
|
| | operations-center | https://[::1]:8443 | ready | update pending |
|
|
+------------+-------------------+------------------------------+--------+----------------+
|
|
```
|
|
|
|
Nodes now show their cluster membership. The OC server itself always shows
|
|
`update pending` (it cannot update itself via its own pipeline).
|
|
|
|
### 7.4 Update Management
|
|
|
|
```bash
|
|
operations-center provisioning update list
|
|
operations-center provisioning update show <UUID>
|
|
operations-center provisioning update refresh # Check for new updates
|
|
operations-center provisioning update assign-channels <UUID> --channel <name>
|
|
```
|
|
|
|
**Channel management**: updates auto-link to channels based on upstream
|
|
`filter_expression`. Use `assign-channels` to explicitly control which
|
|
updates appear in which channels.
|
|
|
|
### 7.5 Web UI
|
|
|
|
Navigate to `https://192.168.102.120:8443/ui/` in Firefox (with client.pfx
|
|
imported). The web UI shows: servers, clusters, instances, networks, storage
|
|
pools, load balancers, network forwards, and update status.
|
|
|
|
---
|
|
|
|
## Section 8: Cluster Lifecycle
|
|
|
|
### 8.1 Evacuation & Restore
|
|
|
|
```bash
|
|
incus cluster evacuate oc-node-01:oc-node-01 --force
|
|
# OC still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
|
|
operations-center provisioning cluster resync oc-cluster
|
|
# Inventory shows workloads moved to other nodes
|
|
|
|
incus cluster restore oc-node-01:oc-node-01 --force
|
|
```
|
|
|
|
### 8.2 Node Failure Simulation
|
|
|
|
Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s).
|
|
Node auto-rejoins after Proxmox restart (~60s boot):
|
|
|
|
```bash
|
|
# OC does NOT detect node failures
|
|
# provisioning server show still reports "Status: ready" for offline nodes
|
|
```
|
|
|
|
### 8.3 Reboot Considerations
|
|
|
|
**WARNING**: `operations-center provisioning server system reboot` can break
|
|
OC-managed nodes on Proxmox. The OC agent pushes config via the IncusOS REST
|
|
API that gets persisted to `state.txt`. On reboot, invalid values crash the
|
|
daemon. See CLAUDE.md for full analysis.
|
|
|
|
**Safe maintenance procedure**:
|
|
1. `incus cluster evacuate <remote>:<member> --force`
|
|
2. Proxmox stop the VM
|
|
3. Proxmox start the VM
|
|
4. Wait for IncusOS to boot (~60-90s)
|
|
5. `incus cluster restore <remote>:<member> --force`
|
|
|
|
---
|
|
|
|
## Section 9: Cleanup
|
|
|
|
**Note**: this section documents the cleanup procedure but is NOT executed.
|
|
The lab is left fully running with all workloads, OVN networking, and load
|
|
balancer active for exploration via OC web UI and CLI.
|
|
|
|
```bash
|
|
# Delete workloads
|
|
incus delete oc-node-01:ha-web-01 oc-node-01:ha-web-02 oc-node-01:ha-web-03 --force
|
|
|
|
# Delete load balancer and forward
|
|
incus network load-balancer delete oc-node-01:net-prod 192.168.103.201
|
|
incus network forward delete oc-node-01:net-prod 192.168.103.202
|
|
|
|
# Delete OVN network
|
|
incus network delete oc-node-01:net-prod
|
|
|
|
# Delete OVN control plane
|
|
incus delete oc-node-01:ovn-central --force
|
|
|
|
# Delete UPLINK
|
|
incus network delete oc-node-01:UPLINK
|
|
|
|
# Disable OVN services on all nodes
|
|
for remote in oc-node-01 oc-node-02 oc-node-03; do
|
|
incus query ${remote}:/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {"enabled": false}, "state": {}
|
|
}'
|
|
done
|
|
|
|
# Remove OVN chassis role
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
incus cluster role remove oc-node-01:${node} ovn-chassis
|
|
done
|
|
|
|
# Destroy infrastructure (--yes to skip confirmation prompts)
|
|
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
|
|
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml
|
|
|
|
# Remove OC remote
|
|
operations-center remote remove oc-lab
|
|
```
|
|
|
|
---
|
|
|
|
## Section 10: Verification Checklist
|
|
|
|
| # | Check | Status |
|
|
|---|-------|--------|
|
|
| 1 | OC server deployed and reachable | PASS |
|
|
| 2 | OC remote configured | PASS |
|
|
| 3 | Updates downloaded and ready | PASS |
|
|
| 4 | 3 nodes provisioned from OC ISO | PASS |
|
|
| 5 | All nodes auto-registered with OC | PASS |
|
|
| 6 | All nodes "up to date" (not "update pending") | PASS |
|
|
| 7 | No crontab bug (force_reboot not used) | PASS |
|
|
| 8 | Cluster formed via OC `provisioning cluster add` | PASS |
|
|
| 9 | All 3 nodes ONLINE + Fully operational | PASS |
|
|
| 10 | OVN cross-node connectivity (sub-ms) | PASS |
|
|
| 11 | 3 nginx instances on 3 nodes | PASS |
|
|
| 12 | LB distributes across all backends | PASS |
|
|
| 13 | HA: LB serves with one backend down | PASS |
|
|
| 14 | Network forward: direct backend access | PASS |
|
|
| 15 | OC inventory shows all resources | PASS |
|
|
| 16 | OC web UI accessible | PASS |
|
|
|
|
---
|
|
|
|
## Section 11: Quick Reference
|
|
|
|
### OC Provisioning Workflow
|
|
|
|
```
|
|
1. Deploy OC server (incusos-proxmox)
|
|
2. Add OC remote (operations-center remote add)
|
|
3. Wait for updates (provisioning update list -- at least one "ready")
|
|
4. Create token (provisioning token add)
|
|
5. Create token seed (provisioning token seed add) -- NO force_reboot
|
|
6. Create old-stable channel (provisioning channel add old-stable)
|
|
7. Assign older update to old-stable (provisioning update assign-channels)
|
|
8. Generate ISO from OLDER channel (token seed get-image --channel old-stable)
|
|
9. Deploy nodes (incusos-proxmox --iso)
|
|
10. Wait for auto-registration + update delivery
|
|
11. Form cluster (provisioning cluster add)
|
|
12. Set up OVN networking (OVN container + services + UPLINK + net-prod)
|
|
```
|
|
|
|
### Key OC CLI Commands
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `remote add <name> <url> --auth-type tls` | Add OC remote |
|
|
| `remote switch <name>` | Set default remote |
|
|
| `admin os show` | IncusOS hostname, version, uptime |
|
|
| `provisioning token add --uses N` | Create provisioning token |
|
|
| `provisioning token seed add <UUID> <name> <file>` | Attach seed to token |
|
|
| `provisioning token seed get-image <UUID> <seed> <file> --channel <ch>` | Generate ISO |
|
|
| `provisioning channel add <name> --description <desc>` | Create provisioning channel |
|
|
| `provisioning update list` | List available updates |
|
|
| `provisioning update refresh` | Check for new updates |
|
|
| `provisioning update assign-channels <UUID> --channel <name>` | Assign update to channel |
|
|
| `provisioning server list` | List registered servers |
|
|
| `provisioning server resync <name>` | Resync server state |
|
|
| `provisioning cluster add <name> <url> --server-names ... -a <seed>` | Form cluster |
|
|
| `provisioning cluster resync <name>` | Force inventory resync |
|
|
| `provisioning cluster list` | List clusters |
|
|
| `inventory instance list --cluster <name>` | List instances |
|
|
| `inventory network list --cluster <name>` | List networks |
|
|
| `inventory network-load-balancer list` | List load balancers |
|
|
| `inventory network-forward list` | List network forwards |
|
|
| `inventory storage-pool list` | List storage pools |
|
|
|
|
### Incus Cluster Commands (via remote)
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `incus cluster list <remote>:` | List cluster members |
|
|
| `incus cluster role add <remote>:<member> ovn-chassis` | Add OVN role |
|
|
| `incus cluster evacuate <remote>:<member> --force` | Evacuate member |
|
|
| `incus cluster restore <remote>:<member> --force` | Restore member |
|
|
| `incus network create <remote>:<net> --type=physical --target=<member> parent=mgmt` | Per-target network |
|
|
| `incus network create <remote>:<net> --type=ovn network=UPLINK` | OVN network |
|
|
| `incus network load-balancer create <remote>:<net> <ip>` | Create LB |
|
|
| `incus network forward create <remote>:<net> <ip>` | Create forward |
|
|
|
|
---
|
|
|
|
## Section 12: Troubleshooting
|
|
|
|
| Symptom | Cause | Fix |
|
|
|---------|-------|-----|
|
|
| `needs_update: true` blocks clustering | Nodes deployed from ISO matching latest version | Deploy from older ISO (`--channel old-stable`), let OC push the update |
|
|
| Terraform "Certificate already in trust store" | Cert in both SEED_DATA and app-seed-config | Use empty `{}` for app-seed-config when cert is in SEED_DATA |
|
|
| Terraform "Network/Pool not in pending state" | `apply_defaults: true` pre-created resources | Use `apply_defaults: false` (recommended) |
|
|
| TLS cert error after clustering | Cluster cert regenerated | `incus remote remove/add` with `--accept-certificate` |
|
|
| `get-image` fails | No updates in `ready` state | Wait ~5 min for OC to download; `provisioning update list` |
|
|
| Node doesn't self-register | Bad ISO or token exhausted | Verify ISO from `get-image`; check `token list` |
|
|
| Token seed fields all `{}` | Flat YAML format | Use structured: `install:` section key |
|
|
| OC web UI 403 | Client cert not imported | Import client.pfx into browser |
|
|
| OC shows "ready" for offline node | OC doesn't detect failures | Check via `incus cluster list` instead |
|
|
| OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot |
|
|
| UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide |
|
|
| Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs |
|
|
| `assign-channels` "Not found" | Channel doesn't exist yet | Run `provisioning channel add old-stable` first |
|
|
| Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` |
|
|
|
|
---
|
|
|
|
## Section 13: Known Limitations (v0.3.0)
|
|
|
|
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
|
|
- **No real-time inventory sync**: requires explicit `cluster resync`
|
|
- **No cluster member state tracking**: OC always shows `ready` for all nodes
|
|
- **Stale server entries**: out-of-band changes create stale entries
|
|
- **needs_update tracking**: based on OC's delivery pipeline, not version comparison
|
|
- **OC reboot breaks Proxmox VMs**: OC agent boot failures on restart
|
|
- **No cluster resize**: cannot add/remove members from existing OC clusters
|
|
- **Token seeds require structured YAML**: section keys, not flat format
|
|
- **OVN LB has no health checks**: connection hashing distributes to dead backends
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
|
|
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
|
|
- [IncusOS GitHub](https://github.com/lxc/incus-os)
|
|
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
|
|
- [Clustering guide](clustering-guide.md) -- manual cluster formation
|
|
- [Networking guide](networking-guide.md) -- OVN overlay tutorial
|
|
- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA
|