37 KiB

Raw Blame History

Operations Center Guide for IncusOS

This guide covers deploying and using Operations Center (OC) by FuturFusion -- a centralized management layer for IncusOS/Incus deployments. OC handles provisioning, clustering, inventory, and updates through a REST API, CLI, and web UI.

All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553 on Proxmox VE 9.1.5 (nested virtualization on Intel).

Overview

Operations Center is fundamentally different from managing Incus clusters manually via incus CLI remotes. Key differences:

	Manual (incus CLI)	Operations Center
Node provisioning	`incusos-proxmox` + seed archives	OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid)
Cluster formation	`incus cluster enable` + `incus cluster join`	OC orchestrates via CLI/API/web UI
Node registration	Manual remote add	Automatic self-registration on first boot
Inventory	`incus list` per remote	Centralized view across all clusters
Updates	Manual per-node	OC manages update channels and rollouts
Brownfield adoption	N/A (nodes are standalone)	Not supported -- nodes must be OC-provisioned

Key constraint: OC only manages nodes it has provisioned. There is no "adopt existing node" workflow. Nodes must boot from an OC-generated ISO containing a provisioning token, then self-register on first boot.

Prerequisites

Install OC CLI

# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center --version  # Note: use --version, not 'version' subcommand

Set up OC client certificates

OC uses the same TLS client certificate format as Incus:

mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

Generate PKCS#12 cert for browser access

The OC web UI requires a client certificate for authentication:

openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import

Verify with doctor check

./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed

Part 1: Deploy Operations Center Server

Deployment config

Use incusos/examples/lab-oc-deploy.yaml -- deploys only the OC server (Incus nodes will be provisioned by OC itself):

defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true

Deploy

./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml

Note the OC IP address from --status output.

Set up OC CLI remote

operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list

Verify browser access

Navigate to https://<OC_IP>:8443 in Firefox (with client.pfx imported). The OC web UI should load and show the dashboard.

Wait for updates to download

OC automatically downloads IncusOS update packages from upstream. At least one update must be in ready state before provisioned ISOs can be generated:

operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot

Also via REST API:

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/provisioning/updates

Part 2: Provision Incus Node ISOs

Create a provisioning token

# CLI
operations-center provisioning token add \
    --description "Lab cluster nodes" \
    --uses 5

# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>

Via REST API:

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
     -d '{"description": "Lab cluster nodes (API)", "uses": 5}'

Create a token seed for Proxmox VMs (tested)

OC supports token seeds -- named, reusable pre-seed configs attached to a token. The YAML format uses section-level keys (install:, applications:, etc.) -- NOT the flat format from standard seed archives.

# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
  version: "1"
  force_install: true
  force_reboot: true

Important: a flat format (version: "1" at root) maps all fields to empty {} -- they don't get assigned to any section. The structured format with install: as the top-level key is required.

operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml \
    --description "Force reboot for Proxmox VMs"

operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed

Download the OC-provisioned ISO (tested)

Use token seed get-image for the cleanest approach (pre-seed embedded in ISO):

operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

ls -lh /tmp/IncusOS-oc.iso  # ~3.4 GB

Alternative without token seeds (inline pre-seed):

operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
    /tmp/oc-preseed.yaml --type iso --architecture x86_64

Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

The hybrid approach uses incusos-proxmox --iso to deploy nodes from an OC-provisioned ISO. This combines the best of both worlds:

OC: auto-registration token embedded in the boot ISO
incusos-proxmox: VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, media cleanup, IP detection, remote setup

The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides force_reboot, hostname, and app selection. IncusOS reads both sources and merges them -- they coexist without conflict.

Deployment config

# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400   # Use high VMIDs to avoid collisions with VMs outside pool

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false   # OC handles storage/network via Terraform

  - name: oc-node-02
    app: incus
    apply_defaults: false

  - name: oc-node-03
    app: incus
    apply_defaults: false

VMID note: if your Proxmox API token is scoped to a resource pool, it cannot see VMs outside that pool. Use start_vmid above existing VM ranges to avoid VMID collisions with invisible VMs.

Deploy (tested)

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml

incusos-proxmox handles everything: VM creation, ISO upload, SEED_DATA generation (force_reboot + hostname per node), install monitoring via blockstat polling, media removal, boot order switch, IP detection, and incus remote setup.

Verify OC auto-registration (tested)

After nodes boot from disk, they auto-register with OC within ~30 seconds:

operations-center provisioning server list

Nodes register with their hostname (set via SEED_DATA), so no renaming needed. Each node appears with type incus, status ready, and its connection URL.

Tested result: all 3 nodes auto-registered with correct hostnames, correct IPs, type incus, status ready. No manual intervention needed.

Part 4: Form Cluster via Operations Center

Prepare application config

Create an Incus preseed that injects the client certificate into the cluster (so the incus CLI can connect directly):

# /tmp/oc-app-config.yaml
certificates:
  - type: client
    name: lab-client-cert
    certificate: |-
      -----BEGIN CERTIFICATE-----
      <contents of ~/.config/incus/client.crt>
      -----END CERTIFICATE-----

Form cluster via OC CLI (tested)

operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

OC orchestrates the full cluster formation:

Sets core.https_address to each node's specific IP
Enables clustering on the first node
Joins remaining nodes (handles storage pool + network creation)
Applies the application seed (injects client cert)
Runs Terraform/OpenTofu for post-cluster configuration

apply_defaults: false (tested, recommended)

Use apply_defaults: false for OC-managed nodes. This is now the tested and recommended approach. With apply_defaults: false:

Nodes still listen on port 8443 and trust preseed certificates
The underlying ZFS dataset (local/incus) still exists
But no Incus storage pool, network, or profile devices are created
OC's Terraform handles all resource creation cleanly during cluster formation

Tested result: cluster forms cleanly with apply_defaults: false. OC's Terraform creates the storage pool (local), networks (incusbr0, meshbr0), and injects the client certificate via --application-seed-config. The only Terraform error is "Certificate already in trust store" when the cert was also injected via SEED_DATA -- avoid this by only injecting the cert through the application seed config, not the SEED_DATA.

apply_defaults: true (also works, with caveats)

When using apply_defaults: true, each node already has a local ZFS storage pool, incusbr0 network bridge, and the client certificate. OC's Terraform then fails with:

"Certificate already in trust store"
"Network is not in pending state"
"Storage pool is not in pending state"

The cluster itself forms successfully despite these Terraform errors. However, cluster artifact list will be empty since Terraform didn't complete.

Verify cluster (tested)

# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01  # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:

Cluster artifacts

OC generates Terraform/OpenTofu artifacts on successful cluster formation:

operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)

Part 5: OC Management Capabilities (Tested)

Inventory browsing

OC inventory is not real-time -- it requires an explicit cluster resync to pick up changes made via the incus CLI:

# Force sync first
operations-center provisioning cluster resync oc-cluster

# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list

Tested: after creating workloads via incus launch, OC inventory was empty until cluster resync was run. After resync, all instances, storage pools, networks, profiles, projects, and images appeared correctly.

Inventory sync latency

Creating workloads via incus CLI: not visible in OC until resync
After resync: ~5 seconds for inventory to populate
Live migration: location change not reflected until resync
Evacuation: instance relocation not reflected until resync
OC inventory instance show <UUID> shows: name, project, cluster, server, timestamp

Server management (tested)

operations-center provisioning server list      # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name>    # WARNING: see below
operations-center provisioning server system poweroff <name>

WARNING -- OC reboot breaks OC-managed nodes on Proxmox: the system reboot command sends a guest-level (ACPI) reboot. On standalone IncusOS nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot, all recover in ~50s with containers and data intact). The failure is OC-specific: the OC agent runs on every boot and can fail with:

"failed to parse update frequency - invalid crontab expression" (OC pushed invalid update config)
"error from operations center: forbidden: failed creating server: constraint violation" (OC trying to re-register an existing server)
Incus stuck at "starting application name=incus" (cluster peers are in boot loops, so quorum cannot form)

These failures cause boot loops or hung services. Even Proxmox stop/start does not recover because the OC state on disk remains broken. The only fix is to destroy and redeploy the VM.

Safe reboot: Proxmox stop/start and guest reboot both work on standalone (non-OC) IncusOS nodes. Avoid OC system reboot on OC-managed nodes. The proper lifecycle for maintenance is:

incus cluster evacuate <remote>:<member> --force
Proxmox stop the VM
Proxmox start the VM
Wait for IncusOS to boot (~60-90s)
incus cluster restore <remote>:<member> --force

IncusOS scrub_schedule fix via API (tested)

IncusOS has an intermittent bug (~50% of deploys) where state.txt is written with an empty scrub_schedule field. The registerJobs() function calls gocron.IsValid("") which fails, crashing the daemon in a loop. The REST API is briefly reachable during each crash cycle (Incus starts before the scheduler).

Fix via IncusOS REST API (through the Incus proxy at /os/ prefix):

# Check current storage config
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<NODE_IP>:8443/os/1.0/system/storage

# If scrub_schedule is empty, fix it
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X PUT -H "Content-Type: application/json" \
     -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
     https://<NODE_IP>:8443/os/1.0/system/storage

Key details:

The /os/ prefix is required — this proxies to the IncusOS daemon API
PATCH returns 501 (not implemented); use PUT instead
incus admin os system storage edit fails because it reads the current (corrupt) config and validates it before allowing edits
incusos-proxmox now includes fix_scrub_schedule() which proactively fixes this on every deployed node (safe — returns early if already set)
Filed upstream as IncusOS issue #843

Node failure and recovery (tested)

Proxmox hard-stop simulates node crash. Incus cluster detects failure via heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):

# Simulate crash: Proxmox API stop
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...

# After ~40s: incus cluster list shows OFFLINE with heartbeat info

# Restart: Proxmox API start
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...

# After ~60s boot: node auto-rejoins cluster (ONLINE)
# Containers on the node auto-start after recovery

OC does not detect node failures: provisioning server show still shows Status: ready for an OFFLINE/crashed node. Last Seen timestamp is stale.

OC does not track Incus cluster member state: the server list always shows ready status even when the Incus cluster reports EVACUATED.

Server edit commands

Network and storage have edit subcommands (no show):

operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>

Update management (tested)

operations-center provisioning update list       # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh     # Check for new updates

Updates show comprehensive file lists including sysexts: incus, operations-center, gpu-support, incus-ceph, incus-linstor, migration-manager, debug.

System configuration (tested)

operations-center system settings show   # log_level
operations-center system security show   # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show    # Update source, filter expressions, signature CA

Note: system certificate only has set (no show). system network has show and edit.

Admin / IncusOS management (tested)

operations-center admin os show           # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list   # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log

Cluster operations (tested)

operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>

Cluster templates

operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>

Part 6: Cluster Lifecycle (Tested)

Live migration visibility through OC (tested)

# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01

# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03

# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # now shows new location

Tested: ~141 MB/s migration speed. OC inventory reflected the location change only after explicit cluster resync.

Cluster evacuation through OC (tested)

incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # shows workloads moved to other nodes

incus cluster restore oc-node-01:oc-node-01 --force

Node replacement lifecycle (tested)

Full procedure tested after OC reboot broke a node:

# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

# 2. Destroy broken VM via Proxmox (stop + delete)

# 3. Redeploy via incusos-proxmox --iso (single-node config)
#    Use apply_defaults: false for OC-managed nodes
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml

# 4. New node auto-registers with OC (same hostname, new IP)
#    NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
#    while OC thinks it's in a cluster. Resync doesn't fix stale entries.

# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# With apply_defaults: false, no cleanup needed (no pool/network to delete)
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)

# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate

Node failure simulation (tested)

Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):

# Simulate crash
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# incus cluster list shows OFFLINE after ~40s

# Restart
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# Node auto-rejoins after ~60s boot. Containers auto-start.
# OC still shows "ready" (does not detect node failures)

Factory reset (not tested)

# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>

OC CLI Command Reference

Remote management

Command	Description
`operations-center remote add <name> <url> --auth-type tls`	Add OC remote
`operations-center remote switch <name>`	Set default remote
`operations-center remote list`	List configured remotes
`operations-center remote remove <name>`	Remove a remote

Provisioning tokens

Command	Description
`provisioning token add --description "..." --uses N --lifetime 168h`	Create token
`provisioning token list`	List tokens
`provisioning token show <UUID>`	Token details
`provisioning token remove <UUID>`	Delete token
`provisioning token seed add <UUID> <name> <file> --description "..."`	Attach named seed
`provisioning token seed list <UUID>`	List token seeds
`provisioning token seed show <UUID> <name>`	Show seed details
`provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64`	ISO from seed
`provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64`	ISO from inline preseed

Provisioning servers

Command	Description
`provisioning server list`	List registered servers (includes OC itself)
`provisioning server show <name>`	Server details + certificate
`provisioning server rename <old> <new>`	Rename server
`provisioning server resync <name>`	Resync single server
`provisioning server remove <name>`	Remove (fails if in cluster)
`provisioning server edit <name>`	Edit server (interactive)
`provisioning server system reboot <name>`	Guest reboot (DANGEROUS on Proxmox)
`provisioning server system poweroff <name>`	Guest poweroff
`provisioning server system network edit <name>`	Edit network config
`provisioning server system storage edit <name>`	Edit storage config

Provisioning clusters

Command	Description
`provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>`	Form cluster
`provisioning cluster list`	List clusters
`provisioning cluster show <name>`	Cluster details + certificate
`provisioning cluster resync <name>`	Force inventory resync
`provisioning cluster rename <name> <new-name>`	Rename cluster
`provisioning cluster update <name> --connection-url <url>`	Update cluster URL
`provisioning cluster update-certificate <name> <cert> <key>`	Update cluster cert
`provisioning cluster remove <name>`	Delete cluster
`provisioning cluster factory-reset <name> <token-uuid>`	Factory reset
`provisioning cluster artifact list <name>`	List artifacts
`provisioning cluster artifact archive <name> <type> <output>`	Download artifact

Cluster templates

Command	Description
`provisioning cluster-template add <name> --description "..."`	Create template
`provisioning cluster-template list`	List templates
`provisioning cluster-template show <name>`	Show template
`provisioning cluster-template remove <name>`	Delete template

Inventory

Command	Description
`inventory instance list`	List all instances
`inventory instance show <UUID>`	Instance details (by UUID, not name)
`inventory storage-pool list`	List storage pools
`inventory storage-volume list`	List storage volumes
`inventory storage-bucket list`	List storage buckets
`inventory network list`	List networks
`inventory network-acl list`	List network ACLs
`inventory network-address-set list`	List network address sets
`inventory network-forward list`	List network forwards
`inventory network-integration list`	List network integrations
`inventory network-load-balancer list`	List network load balancers
`inventory network-peer list`	List network peers
`inventory network-zone list`	List network zones
`inventory profile list`	List profiles
`inventory project list`	List projects
`inventory image list`	List images
`inventory query --cluster <name>`	Cross-resource tree view (v0.3.0+)

Updates

Command	Description
`provisioning update list`	List available IncusOS versions
`provisioning update show <UUID>`	Version details + file list
`provisioning update refresh`	Check for new updates

System

Command	Description
`system settings show`	System settings (log_level)
`system security show`	OIDC, OpenFGA, ACME, trusted certs
`system updates show`	Update source, filter, signature CA
`system network show`	Network config
`system certificate set <cert> <key>`	Set server certificate

Admin

Command	Description
`admin os show`	IncusOS hostname, version, uptime
`admin os application list`	List applications
`admin os application show <name>`	Application state
`admin os service list`	List services (iscsi, lvm, nvme, ovn, tailscale, ...)
`admin os service show <name>`	Service config
`admin os debug log`	Debug logs
`admin os debug processes`	System process list
`admin os debug secureboot`	Secure Boot debug info

All commands prefixed with operations-center.

REST API Reference

OC listens on port 8443 (same as Incus on IncusOS). All endpoints require TLS client certificate authentication.

Common curl pattern

curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/<endpoint>

Key endpoints

Method	Endpoint	Description
GET	`/1.0`	Server info
GET	`/1.0/provisioning/updates`	List update packages
GET	`/1.0/provisioning/tokens`	List tokens
POST	`/1.0/provisioning/tokens`	Create token
GET	`/1.0/provisioning/servers`	List registered servers
GET	`/1.0/provisioning/clusters`	List clusters
POST	`/1.0/provisioning/clusters`	Form cluster
GET	`/1.0/inventory/instances`	List all instances
GET	`/1.0/inventory/instance/<UUID>`	Instance details
GET	`/1.0/provisioning/servers/<name>`	Server details + hardware inventory

IncusOS REST API (via Incus proxy)

IncusOS nodes expose their own REST API through the Incus proxy at the /os/ prefix on port 8443:

Method	Endpoint	Description
GET	`/os/1.0/system/storage`	Storage config (scrub_schedule)
PUT	`/os/1.0/system/storage`	Update storage config
GET	`/os/1.0`	IncusOS system info

Proxmox-Specific Considerations

force_reboot is required

IncusOS does not auto-halt after installation. Without force_reboot: true in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at "please remove installation media" indefinitely.

Options for OC-provisioned ISOs on Proxmox:

Pre-seed file passed to get-image: include force_reboot: true
Fallback SEED_DATA ISO on ide3: minimal install.yaml with force_reboot: true only

VM creation settings

OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS deployments (see incusos-proxmox):

bios=ovmf, machine=q35 -- UEFI boot required
efidisk0: pre-enrolled-keys=0 -- IncusOS enrolls its own Secure Boot keys
tpmstate0: version=v2.0 -- required for disk encryption
cpu=host -- needed for x86_64_v3 instruction set requirement
scsihw=virtio-scsi-pci + scsi0 -- VirtIO-blk is broken with IncusOS
balloon=0 -- IncusOS manages memory internally
Minimum 50 GiB disk, minimum 4096 MiB RAM

Install media removal

After installation completes, you must stop the VM and delete ide2 (and ide3 if used) before booting from disk. IncusOS refuses to start if install media is still attached.

IP detection

IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to find node IPs:

# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
    -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")

# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'

Known Limitations (Tested Findings)

Confirmed limitations (v0.3.0)

No brownfield adoption: nodes must boot from OC-provisioned ISO
No real-time inventory sync: requires explicit cluster resync
No cluster member state tracking: OC always shows ready, even for EVACUATED or OFFLINE nodes
Stale server entries: out-of-band cluster changes (via incus CLI) create stale entries that resync doesn't fix
OC reboot breaks OC-managed Proxmox VMs: OC agent boot failures cause boot loops or hung services (not a dqlite or TPM issue -- standalone nodes reboot fine)
Token seed format quirk: requires structured YAML with section keys, not flat format
Expired tokens not auto-cleaned: remain in list after expiry
No cluster resize: cannot add/remove members from existing clusters

Design constraints

OC runs on IncusOS (same immutable OS as Incus nodes)
Uses port 8443 for both API and web UI
Requires at least one trusted client certificate in seed
Provisioning tokens have configurable use count and lifetime
OC adds meshbr0 network to clusters (not present in manual clustering)
REST API endpoint: /1.0 (same pattern as Incus)
Auth methods: TLS client cert or OIDC

Troubleshooting (Tested)

Symptom	Likely cause	Fix
`get-image` fails	No updates in `ready` state	Wait ~3 min for OC to download packages; check `provisioning update list`
Node doesn't self-register	ISO not OC-provisioned, or token exhausted	Verify ISO from `get-image` with valid token; check `token list` for remaining uses
Node stuck at "remove media"	Missing `force_reboot` in seed	Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox`
Token seed fields all `{}`	Flat YAML format	Use structured format with section keys: `install:`, `applications:`, etc.
OC web UI returns 403	Client cert not imported	Import client.pfx into browser certificates
OC CLI "not authorized"	Wrong certs in `~/.config/operations-center/`	Copy `client.crt` and `client.key` from `~/.config/incus/`
Cluster Terraform errors	`apply_defaults: true` pre-created resources	Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors
`incus` TLS error after clustering	Cluster cert regenerated	`incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate`
OC reboot kills OC-managed node	OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops	Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes.
OC stale server entry after node replace	OC doesn't detect out-of-band cluster changes	`cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC.
OC shows old IP for replaced node	Re-registration blocked by existing entry	Remove stale entry (requires cluster dissociation first)
Token expired but still listed	Tokens don't auto-clean	Remove with `provisioning token remove <UUID>`
VMID collision (500/403 on create)	Pool-scoped token can't see VMs outside pool	Use high `start_vmid` (400+) to avoid collisions
`cluster resync` socket error	Transient OC internal issue	Retry after a few seconds -- usually succeeds on second attempt
Proxmox API empty response	Bash `!` in token username	Use `set +H` or store token in variable; avoid bash history expansion
`server remove` fails	Server is part of a cluster	Must remove cluster first, or force-remove member from Incus cluster
`cluster join` extra prompt	OC adds `meshbr0` network	Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins

Comparison: Manual vs OC Cluster Workflow (Tested)

Manual (incusos-proxmox + incus CLI)

Write YAML config with VM specs
incusos-proxmox --yes config.yaml creates VMs with SEED_DATA
Manually set core.https_address on each node
Delete storage pool/network on joining nodes
incus cluster enable + incus cluster join per node
Re-add remotes after certificate changes
~30 minutes for 3-node cluster

Hybrid (incusos-proxmox --iso + OC)

Deploy OC server via incusos-proxmox --yes lab-oc-deploy.yaml
Create provisioning token + token seed in OC
Download OC-provisioned ISO via token seed get-image
incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml
Nodes auto-register with OC (~30s each)
operations-center provisioning cluster add oc-cluster ...
~20 minutes for 3-node cluster (including OC deploy)

What OC automates that we do manually

Step	Manual	OC
Set `core.https_address`	Per-node via `incus config set`	Automatic during cluster add
Storage pool/network cleanup for join	Manual delete per joining node	Automatic (handles pending state)
Cluster enable + join	Multi-step per node	Single `cluster add` command
Client cert injection	Via SEED_DATA	Via `--application-seed-config`
`meshbr0` network	Not created	Auto-created by OC clustering
TLS cert re-add	Manual remote remove/add	Not needed (OC handles internally)

Trade-offs:

Manual: full control, no extra server, works with any Incus version
OC hybrid: auto-registration, single-command clustering, centralized inventory/updates, but requires OC server + OC-provisioned ISOs
OC reboot command is dangerous on Proxmox (OC agent fails on boot, causing loops -- standalone reboot is safe)
OC inventory requires manual resync (not real-time)
Out-of-band cluster changes (via incus CLI) create stale OC state

Open Research Items

OC reboot failure mechanism (resolved): guest reboot is safe on standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover in ~50s). The failure is OC-specific: the OC agent pushes config via the IncusOS REST API that gets persisted to state.txt. On reboot, invalid values (e.g., cron expression where Go duration is expected) crash the daemon. Root cause is NOT dqlite corruption or TPM issues.
apply_defaults conflict with OC Terraform (resolved): apply_defaults: false is now the tested and recommended approach. OC's Terraform handles storage pool, network, and cert creation cleanly when nodes start without defaults. apply_defaults: true also works but produces non-fatal Terraform errors.
IncusOS scrub_schedule bug (workaround found, upstream filed as #843): ~50% of deploys hit an empty scrub_schedule in state.txt. Root cause is likely a race condition between REST API startup and registerJobs(), or a zero-value encoding bug in encodeHelper(). Workaround: PUT /os/1.0/system/storage with {"config":{"scrub_schedule":"0 4 * * 0"}}. incusos-proxmox now auto-heals this proactively on every deploy.
Stale server entries after out-of-band changes: when cluster members are removed via incus cluster remove (not through OC), stale entries persist in OC's server list. cluster resync doesn't fix them. The entry can't be removed while OC considers it part of a cluster.
Inventory sync is manual: OC inventory requires explicit cluster resync to pick up changes made via the incus CLI. No webhook or event-driven sync. This applies to instance creation, migration, evacuation, and deletion.
Node failure detection: OC does not detect node failures. provisioning server show continues to report Status: ready for OFFLINE/crashed nodes. Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.

References

Operations Center GitHub
Operations Center releases
IncusOS GitHub
Incus documentation
Clustering guide (manual cluster formation reference)

37 KiB Raw Blame History

Operations Center Guide for IncusOS

Overview

Prerequisites

Install OC CLI

Set up OC client certificates

Generate PKCS#12 cert for browser access

Verify with doctor check

Part 1: Deploy Operations Center Server

Deployment config

Deploy

Set up OC CLI remote

Verify browser access

Wait for updates to download

Part 2: Provision Incus Node ISOs

Create a provisioning token

Create a token seed for Proxmox VMs (tested)

Download the OC-provisioned ISO (tested)

Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

Deployment config

Deploy (tested)

Verify OC auto-registration (tested)

Part 4: Form Cluster via Operations Center

Prepare application config

Form cluster via OC CLI (tested)

apply_defaults: false (tested, recommended)

apply_defaults: true (also works, with caveats)

Verify cluster (tested)

Cluster artifacts

Part 5: OC Management Capabilities (Tested)

Inventory browsing

Inventory sync latency

Server management (tested)

IncusOS scrub_schedule fix via API (tested)

Node failure and recovery (tested)

Server edit commands

Update management (tested)

System configuration (tested)

Admin / IncusOS management (tested)

Cluster operations (tested)

Cluster templates

Part 6: Cluster Lifecycle (Tested)

Live migration visibility through OC (tested)

Cluster evacuation through OC (tested)

Node replacement lifecycle (tested)

Node failure simulation (tested)

Factory reset (not tested)

OC CLI Command Reference

Remote management

Provisioning tokens

Provisioning servers

Provisioning clusters

Cluster templates

Inventory

Updates

System

Admin

REST API Reference

Common curl pattern

Key endpoints

IncusOS REST API (via Incus proxy)

Proxmox-Specific Considerations

force_reboot is required

VM creation settings

Install media removal

IP detection

Known Limitations (Tested Findings)

Confirmed limitations (v0.3.0)

Design constraints

Troubleshooting (Tested)

Comparison: Manual vs OC Cluster Workflow (Tested)

Manual (incusos-proxmox + incus CLI)

Hybrid (incusos-proxmox --iso + OC)

What OC automates that we do manually

Open Research Items

References

37 KiB

Raw Blame History