31 KiB

Raw Blame History

Operations Center Guide for IncusOS

This guide covers deploying and using Operations Center (OC) by FuturFusion -- a centralized management layer for IncusOS/Incus deployments. OC handles provisioning, clustering, inventory, and updates through a REST API, CLI, and web UI.

All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553 on Proxmox VE 9.1.5 (nested virtualization on Intel).

Overview

Operations Center is fundamentally different from managing Incus clusters manually via incus CLI remotes. Key differences:

	Manual (incus CLI)	Operations Center
Node provisioning	`incusos-proxmox` + seed archives	OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid)
Cluster formation	`incus cluster enable` + `incus cluster join`	OC orchestrates via CLI/API/web UI
Node registration	Manual remote add	Automatic self-registration on first boot
Inventory	`incus list` per remote	Centralized view across all clusters
Updates	Manual per-node	OC manages update channels and rollouts
Brownfield adoption	N/A (nodes are standalone)	Not supported -- nodes must be OC-provisioned

Key constraint: OC only manages nodes it has provisioned. There is no "adopt existing node" workflow. Nodes must boot from an OC-generated ISO containing a provisioning token, then self-register on first boot.

Prerequisites

Install OC CLI

# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center version

Set up OC client certificates

OC uses the same TLS client certificate format as Incus:

mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

Generate PKCS#12 cert for browser access

The OC web UI requires a client certificate for authentication:

openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import

Verify with doctor check

./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed

Part 1: Deploy Operations Center Server

Deployment config

Use incusos/examples/lab-oc-deploy.yaml -- deploys only the OC server (Incus nodes will be provisioned by OC itself):

defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true

Deploy

./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml

Note the OC IP address from --status output.

Set up OC CLI remote

operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list

Verify browser access

Navigate to https://<OC_IP>:8443 in Firefox (with client.pfx imported). The OC web UI should load and show the dashboard.

Wait for updates to download

OC automatically downloads IncusOS update packages from upstream. At least one update must be in ready state before provisioned ISOs can be generated:

operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot

Also via REST API:

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/provisioning/updates

Part 2: Provision Incus Node ISOs

Create a provisioning token

# CLI
operations-center provisioning token add \
    --description "Lab cluster nodes" \
    --uses 5

# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>

Via REST API:

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
     -d '{"description": "Lab cluster nodes (API)", "uses": 5}'

Create a token seed for Proxmox VMs (tested)

OC supports token seeds -- named, reusable pre-seed configs attached to a token. The YAML format uses section-level keys (install:, applications:, etc.) -- NOT the flat format from standard seed archives.

# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
  version: "1"
  force_install: true
  force_reboot: true

Important: a flat format (version: "1" at root) maps all fields to empty {} -- they don't get assigned to any section. The structured format with install: as the top-level key is required.

operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml \
    --description "Force reboot for Proxmox VMs"

operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed

Download the OC-provisioned ISO (tested)

Use token seed get-image for the cleanest approach (pre-seed embedded in ISO):

operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

ls -lh /tmp/IncusOS-oc.iso  # ~3.4 GB

Alternative without token seeds (inline pre-seed):

operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
    /tmp/oc-preseed.yaml --type iso --architecture x86_64

Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

The hybrid approach uses incusos-proxmox --iso to deploy nodes from an OC-provisioned ISO. This combines the best of both worlds:

OC: auto-registration token embedded in the boot ISO
incusos-proxmox: VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, media cleanup, IP detection, remote setup

The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides force_reboot, hostname, and app selection. IncusOS reads both sources and merges them -- they coexist without conflict.

Deployment config

# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400   # Use high VMIDs to avoid collisions with VMs outside pool

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: true

  - name: oc-node-02
    app: incus
    apply_defaults: true

  - name: oc-node-03
    app: incus
    apply_defaults: true

VMID note: if your Proxmox API token is scoped to a resource pool, it cannot see VMs outside that pool. Use start_vmid above existing VM ranges to avoid VMID collisions with invisible VMs.

Deploy (tested)

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml

incusos-proxmox handles everything: VM creation, ISO upload, SEED_DATA generation (force_reboot + hostname per node), install monitoring via blockstat polling, media removal, boot order switch, IP detection, and incus remote setup.

Verify OC auto-registration (tested)

After nodes boot from disk, they auto-register with OC within ~30 seconds:

operations-center provisioning server list

Nodes register with their hostname (set via SEED_DATA), so no renaming needed. Each node appears with type incus, status ready, and its connection URL.

Tested result: all 3 nodes auto-registered with correct hostnames, correct IPs, type incus, status ready. No manual intervention needed.

Part 4: Form Cluster via Operations Center

Prepare application config

Create an Incus preseed that injects the client certificate into the cluster (so the incus CLI can connect directly):

# /tmp/oc-app-config.yaml
certificates:
  - type: client
    name: lab-client-cert
    certificate: |-
      -----BEGIN CERTIFICATE-----
      <contents of ~/.config/incus/client.crt>
      -----END CERTIFICATE-----

Form cluster via OC CLI (tested)

operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC orchestrates the full cluster formation:

Sets core.https_address to each node's specific IP
Enables clustering on the first node
Joins remaining nodes (handles storage pool + network creation)
Applies the application seed (injects client cert)
Runs Terraform/OpenTofu for post-cluster configuration

apply_defaults conflict (tested finding)

When using apply_defaults: true in the SEED_DATA (via incusos-proxmox), each node already has a local ZFS storage pool, incusbr0 network bridge, and the client certificate. OC's Terraform post-clustering step then tries to create the same resources and fails with:

"Certificate already in trust store"
"Network is not in pending state"
"Storage pool is not in pending state"

The cluster itself forms successfully despite these Terraform errors. All 3 nodes join the cluster correctly. The Terraform errors are non-fatal because the resources already exist. However, the cluster artifact list will be empty since Terraform didn't complete.

Recommended approaches for future deployments:

Keep apply_defaults (current tested approach): cluster forms fine, ignore Terraform errors. Skip --application-seed-config since the cert is already injected by the seed.
Skip apply_defaults: deploy nodes without apply_defaults: true and without cert injection in the seed. Let OC handle everything via Terraform. This is the "clean" approach but hasn't been tested yet.

Verify cluster (tested)

# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01  # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:

Cluster artifacts

OC generates Terraform/OpenTofu artifacts on successful cluster formation:

operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)

Part 5: OC Management Capabilities (Tested)

Inventory browsing

OC inventory is not real-time -- it requires an explicit cluster resync to pick up changes made via the incus CLI:

# Force sync first
operations-center provisioning cluster resync oc-cluster

# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list

Tested: after creating workloads via incus launch, OC inventory was empty until cluster resync was run. After resync, all instances, storage pools, networks, profiles, projects, and images appeared correctly.

Inventory sync latency

Creating workloads via incus CLI: not visible in OC until resync
After resync: ~5 seconds for inventory to populate
Live migration: location change not reflected until resync
Evacuation: instance relocation not reflected until resync
OC inventory instance show <UUID> shows: name, project, cluster, server, timestamp

Server management (tested)

operations-center provisioning server list      # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name>    # WARNING: see below
operations-center provisioning server system poweroff <name>

WARNING -- OC reboot breaks IncusOS on Proxmox: the system reboot command sends a guest-level (ACPI) reboot. On Proxmox VMs with TPM2-based disk encryption, this corrupts the TPM PCR measurements. The IncusOS node becomes pingable but all services fail to start (port 8443 never opens). Even a Proxmox-level stop/start does not recover the node. The only fix is to destroy and redeploy the VM.

Safe reboot alternative: use Proxmox stop/start instead of OC reboot. The proper lifecycle for maintenance is:

incus cluster evacuate <remote>:<member> --force
Proxmox stop the VM
Proxmox start the VM
Wait for IncusOS to boot (~60-90s)
incus cluster restore <remote>:<member> --force

OC does not track Incus cluster member state: the server list always shows ready status even when the Incus cluster reports EVACUATED.

Server edit commands

Network and storage have edit subcommands (no show):

operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>

Update management (tested)

operations-center provisioning update list       # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh     # Check for new updates

Updates show comprehensive file lists including sysexts: incus, operations-center, gpu-support, incus-ceph, incus-linstor, migration-manager, debug.

System configuration (tested)

operations-center system settings show   # log_level
operations-center system security show   # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show    # Update source, filter expressions, signature CA

Note: system certificate only has set (no show). system network has show and edit.

Admin / IncusOS management (tested)

operations-center admin os show           # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list   # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log

Cluster operations (tested)

operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>

Cluster templates

operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>

Part 6: Cluster Lifecycle (Tested)

Live migration visibility through OC (tested)

# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01

# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03

# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # now shows new location

Tested: ~141 MB/s migration speed. OC inventory reflected the location change only after explicit cluster resync.

Cluster evacuation through OC (tested)

incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # shows workloads moved to other nodes

incus cluster restore oc-node-01:oc-node-01 --force

Node replacement lifecycle (tested)

Full procedure tested after OC reboot broke a node:

# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

# 2. Destroy broken VM via Proxmox (stop + delete)

# 3. Redeploy via incusos-proxmox --iso (single-node config)
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml

# 4. New node auto-registers with OC (same hostname, new IP)
#    NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
#    while OC thinks it's in a cluster. Resync doesn't fix stale entries.

# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# Remove apply_defaults resources
incus config unset oc-node-03: storage.backups_volume
incus config unset oc-node-03: storage.images_volume
incus storage volume delete oc-node-03:local backups
incus storage volume delete oc-node-03:local images
incus profile device remove oc-node-03:default root
incus profile device remove oc-node-03:default eth0
incus storage delete oc-node-03:local
incus network delete oc-node-03:incusbr0
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)

# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate

Factory reset (not tested)

# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>

OC CLI Command Reference

Remote management

Command	Description
`operations-center remote add <name> <url> --auth-type tls`	Add OC remote
`operations-center remote switch <name>`	Set default remote
`operations-center remote list`	List configured remotes
`operations-center remote remove <name>`	Remove a remote

Provisioning tokens

Command	Description
`provisioning token add --description "..." --uses N --lifetime 168h`	Create token
`provisioning token list`	List tokens
`provisioning token show <UUID>`	Token details
`provisioning token remove <UUID>`	Delete token
`provisioning token seed add <UUID> <name> <file> --description "..."`	Attach named seed
`provisioning token seed list <UUID>`	List token seeds
`provisioning token seed show <UUID> <name>`	Show seed details
`provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64`	ISO from seed
`provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64`	ISO from inline preseed

Provisioning servers

Command	Description
`provisioning server list`	List registered servers (includes OC itself)
`provisioning server show <name>`	Server details + certificate
`provisioning server rename <old> <new>`	Rename server
`provisioning server resync <name>`	Resync single server
`provisioning server remove <name>`	Remove (fails if in cluster)
`provisioning server edit <name>`	Edit server (interactive)
`provisioning server system reboot <name>`	Guest reboot (DANGEROUS on Proxmox)
`provisioning server system poweroff <name>`	Guest poweroff
`provisioning server system network edit <name>`	Edit network config
`provisioning server system storage edit <name>`	Edit storage config

Provisioning clusters

Command	Description
`provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>`	Form cluster
`provisioning cluster list`	List clusters
`provisioning cluster show <name>`	Cluster details + certificate
`provisioning cluster resync <name>`	Force inventory resync
`provisioning cluster rename <name> <new-name>`	Rename cluster
`provisioning cluster update <name> --connection-url <url>`	Update cluster URL
`provisioning cluster update-certificate <name> <cert> <key>`	Update cluster cert
`provisioning cluster remove <name>`	Delete cluster
`provisioning cluster factory-reset <name> <token-uuid>`	Factory reset
`provisioning cluster artifact list <name>`	List artifacts
`provisioning cluster artifact archive <name> <type> <output>`	Download artifact

Cluster templates

Command	Description
`provisioning cluster-template add <name> --description "..."`	Create template
`provisioning cluster-template list`	List templates
`provisioning cluster-template show <name>`	Show template
`provisioning cluster-template remove <name>`	Delete template

Inventory

Command	Description
`inventory instance list`	List all instances
`inventory instance show <UUID>`	Instance details (by UUID, not name)
`inventory storage-pool list`	List storage pools
`inventory network list`	List networks
`inventory profile list`	List profiles
`inventory project list`	List projects
`inventory image list`	List images

Updates

Command	Description
`provisioning update list`	List available IncusOS versions
`provisioning update show <UUID>`	Version details + file list
`provisioning update refresh`	Check for new updates

System

Command	Description
`system settings show`	System settings (log_level)
`system security show`	OIDC, OpenFGA, ACME, trusted certs
`system updates show`	Update source, filter, signature CA
`system network show`	Network config
`system certificate set <cert> <key>`	Set server certificate

Admin

Command	Description
`admin os show`	IncusOS hostname, version, uptime
`admin os application list`	List applications
`admin os application show <name>`	Application state
`admin os service list`	List services (iscsi, lvm, nvme, ovn, tailscale, ...)
`admin os service show <name>`	Service config
`admin os debug log`	Debug logs

All commands prefixed with operations-center.

REST API Reference

OC listens on port 8443 (same as Incus on IncusOS). All endpoints require TLS client certificate authentication.

Common curl pattern

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/<endpoint>

Key endpoints

Method	Endpoint	Description
GET	`/1.0`	Server info
GET	`/1.0/provisioning/updates`	List update packages
GET	`/1.0/provisioning/tokens`	List tokens
POST	`/1.0/provisioning/tokens`	Create token
GET	`/1.0/provisioning/servers`	List registered servers
GET	`/1.0/provisioning/clusters`	List clusters
POST	`/1.0/provisioning/clusters`	Form cluster
GET	`/1.0/inventory/instances`	List all instances

Proxmox-Specific Considerations

force_reboot is required

IncusOS does not auto-halt after installation. Without force_reboot: true in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at "please remove installation media" indefinitely.

Options for OC-provisioned ISOs on Proxmox:

Pre-seed file passed to get-image: include force_reboot: true
Fallback SEED_DATA ISO on ide3: minimal install.yaml with force_reboot: true only

VM creation settings

OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS deployments (see incusos-proxmox):

bios=ovmf, machine=q35 -- UEFI boot required
efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
tpmstate0: version=v2.0 -- required for disk encryption
cpu=host -- needed for x86_64_v3 instruction set requirement
scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
balloon=0 -- IncusOS manages memory internally
Minimum 50 GiB disk, minimum 4096 MiB RAM

Install media removal

After installation completes, you must stop the VM and delete ide2 (and ide3 if used) before booting from disk. IncusOS refuses to start if install media is still attached.

IP detection

IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to find node IPs:

# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
    -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")

# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'

Known Limitations (Tested Findings)

Confirmed limitations (v0.2.2)

No brownfield adoption: nodes must boot from OC-provisioned ISO
No real-time inventory sync: requires explicit cluster resync
No cluster member state tracking: OC always shows ready, even for EVACUATED or OFFLINE nodes
Stale server entries: out-of-band cluster changes (via incus CLI) create stale entries that resync doesn't fix
OC reboot breaks Proxmox VMs: guest reboot corrupts TPM encryption
Token seed format quirk: requires structured YAML with section keys, not flat format
Expired tokens not auto-cleaned: remain in list after expiry
No cluster resize: cannot add/remove members from existing clusters

Design constraints

OC runs on IncusOS (same immutable OS as Incus nodes)
Uses port 8443 for both API and web UI
Requires at least one trusted client certificate in seed
Provisioning tokens have configurable use count and lifetime
OC adds meshbr0 network to clusters (not present in manual clustering)
REST API endpoint: /1.0 (same pattern as Incus)
Auth methods: TLS client cert or OIDC

Troubleshooting (Tested)

Symptom	Likely cause	Fix
`get-image` fails	No updates in `ready` state	Wait ~3 min for OC to download packages; check `provisioning update list`
Node doesn't self-register	ISO not OC-provisioned, or token exhausted	Verify ISO from `get-image` with valid token; check `token list` for remaining uses
Node stuck at "remove media"	Missing `force_reboot` in seed	Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox`
Token seed fields all `{}`	Flat YAML format	Use structured format with section keys: `install:`, `applications:`, etc.
OC web UI returns 403	Client cert not imported	Import client.pfx into browser certificates
OC CLI "not authorized"	Wrong certs in `~/.config/operations-center/`	Copy `client.crt` and `client.key` from `~/.config/incus/`
Cluster Terraform errors	`apply_defaults: true` pre-created resources	Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors
`incus` TLS error after clustering	Cluster cert regenerated	`incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate`
OC reboot kills IncusOS node	TPM PCR corruption from guest reboot	Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot.
OC stale server entry after node replace	OC doesn't detect out-of-band cluster changes	`cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC.
OC shows old IP for replaced node	Re-registration blocked by existing entry	Remove stale entry (requires cluster dissociation first)
Token expired but still listed	Tokens don't auto-clean	Remove with `provisioning token remove <UUID>`
VMID collision (500/403 on create)	Pool-scoped token can't see VMs outside pool	Use high `start_vmid` (400+) to avoid collisions
`cluster resync` socket error	Transient OC internal issue	Retry after a few seconds -- usually succeeds on second attempt
Proxmox API empty response	Bash `!` in token username	Use `set +H` or store token in variable; avoid bash history expansion
`server remove` fails	Server is part of a cluster	Must remove cluster first, or force-remove member from Incus cluster
`cluster join` extra prompt	OC adds `meshbr0` network	Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins

Comparison: Manual vs OC Cluster Workflow (Tested)

Manual (incusos-proxmox + incus CLI)

Write YAML config with VM specs
incusos-proxmox --yes config.yaml creates VMs with SEED_DATA
Manually set core.https_address on each node
Delete storage pool/network on joining nodes
incus cluster enable + incus cluster join per node
Re-add remotes after certificate changes
~30 minutes for 3-node cluster

Hybrid (incusos-proxmox --iso + OC)

Deploy OC server via incusos-proxmox --yes lab-oc-deploy.yaml
Create provisioning token + token seed in OC
Download OC-provisioned ISO via token seed get-image
incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml
Nodes auto-register with OC (~30s each)
operations-center provisioning cluster add oc-cluster ...
~20 minutes for 3-node cluster (including OC deploy)

What OC automates that we do manually

Step	Manual	OC
Set `core.https_address`	Per-node via `incus config set`	Automatic during cluster add
Storage pool/network cleanup for join	Manual delete per joining node	Automatic (handles pending state)
Cluster enable + join	Multi-step per node	Single `cluster add` command
Client cert injection	Via SEED_DATA	Via `--application-seed-config`
`meshbr0` network	Not created	Auto-created by OC clustering
TLS cert re-add	Manual remote remove/add	Not needed (OC handles internally)

Trade-offs:

Manual: full control, no extra server, works with any Incus version
OC hybrid: auto-registration, single-command clustering, centralized inventory/updates, but requires OC server + OC-provisioned ISOs
OC reboot command is dangerous on Proxmox (breaks TPM-based encryption)
OC inventory requires manual resync (not real-time)
Out-of-band cluster changes (via incus CLI) create stale OC state

References

Operations Center GitHub
Operations Center releases
IncusOS GitHub
Incus documentation
Clustering guide (manual cluster formation reference)

31 KiB Raw Blame History

Operations Center Guide for IncusOS

Overview

Prerequisites

Install OC CLI

Set up OC client certificates

Generate PKCS#12 cert for browser access

Verify with doctor check

Part 1: Deploy Operations Center Server

Deployment config

Deploy

Set up OC CLI remote

Verify browser access

Wait for updates to download

Part 2: Provision Incus Node ISOs

Create a provisioning token

Create a token seed for Proxmox VMs (tested)

Download the OC-provisioned ISO (tested)

Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

Deployment config

Deploy (tested)

Verify OC auto-registration (tested)

Part 4: Form Cluster via Operations Center

Prepare application config

Form cluster via OC CLI (tested)

apply_defaults conflict (tested finding)

Verify cluster (tested)

Cluster artifacts

Part 5: OC Management Capabilities (Tested)

Inventory browsing

Inventory sync latency

Server management (tested)

Server edit commands

Update management (tested)

System configuration (tested)

Admin / IncusOS management (tested)

Cluster operations (tested)

Cluster templates

Part 6: Cluster Lifecycle (Tested)

Live migration visibility through OC (tested)

Cluster evacuation through OC (tested)

Node replacement lifecycle (tested)

Factory reset (not tested)

OC CLI Command Reference

Remote management

Provisioning tokens

Provisioning servers

Provisioning clusters

Cluster templates

Inventory

Updates

System

Admin

REST API Reference

Common curl pattern

Key endpoints

Proxmox-Specific Considerations

force_reboot is required

VM creation settings

Install media removal

IP detection

Known Limitations (Tested Findings)

Confirmed limitations (v0.2.2)

Design constraints

Troubleshooting (Tested)

Comparison: Manual vs OC Cluster Workflow (Tested)

Manual (incusos-proxmox + incus CLI)

Hybrid (incusos-proxmox --iso + OC)

What OC automates that we do manually

References

31 KiB

Raw Blame History