maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/operations-center-guide.md

863 lines
31 KiB
Markdown
Raw Blame History

# Operations Center Guide for IncusOS
This guide covers deploying and using Operations Center (OC) by FuturFusion --
a centralized management layer for IncusOS/Incus deployments. OC handles
provisioning, clustering, inventory, and updates through a REST API, CLI, and
web UI.
All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553
on Proxmox VE 9.1.5 (nested virtualization on Intel).
---
## Overview
Operations Center is fundamentally different from managing Incus clusters
manually via `incus` CLI remotes. Key differences:
| | Manual (incus CLI) | Operations Center |
|---|---|---|
| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) |
| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI |
| Node registration | Manual remote add | Automatic self-registration on first boot |
| Inventory | `incus list` per remote | Centralized view across all clusters |
| Updates | Manual per-node | OC manages update channels and rollouts |
| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned |
**Key constraint**: OC only manages nodes it has provisioned. There is no
"adopt existing node" workflow. Nodes must boot from an OC-generated ISO
containing a provisioning token, then self-register on first boot.
---
## Prerequisites
### Install OC CLI
```bash
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center version
```
### Set up OC client certificates
OC uses the same TLS client certificate format as Incus:
```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```
### Generate PKCS#12 cert for browser access
The OC web UI requires a client certificate for authentication:
```bash
openssl pkcs12 -export \
-inkey ~/.config/incus/client.key \
-in ~/.config/incus/client.crt \
-out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
```
### Verify with doctor check
```bash
./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed
```
---
## Part 1: Deploy Operations Center Server
### Deployment config
Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server
(Incus nodes will be provisioned by OC itself):
```yaml
defaults:
cores: 2
memory: 4096
disk: 50
start_vmid: 920
vms:
- name: oc-server
app: operations-center
apply_defaults: true
```
### Deploy
```bash
./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
```
Note the OC IP address from `--status` output.
### Set up OC CLI remote
```bash
operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list
```
### Verify browser access
Navigate to `https://<OC_IP>:8443` in Firefox (with client.pfx imported).
The OC web UI should load and show the dashboard.
### Wait for updates to download
OC automatically downloads IncusOS update packages from upstream. At least
one update must be in `ready` state before provisioned ISOs can be generated:
```bash
operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot
```
Also via REST API:
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<OC_IP>:8443/1.0/provisioning/updates
```
---
## Part 2: Provision Incus Node ISOs
### Create a provisioning token
```bash
# CLI
operations-center provisioning token add \
--description "Lab cluster nodes" \
--uses 5
# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>
```
Via REST API:
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
-X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
-d '{"description": "Lab cluster nodes (API)", "uses": 5}'
```
### Create a token seed for Proxmox VMs (tested)
OC supports **token seeds** -- named, reusable pre-seed configs attached to a
token. The YAML format uses section-level keys (`install:`, `applications:`,
etc.) -- NOT the flat format from standard seed archives.
```yaml
# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
version: "1"
force_install: true
force_reboot: true
```
**Important**: a flat format (`version: "1"` at root) maps all fields to empty
`{}` -- they don't get assigned to any section. The structured format with
`install:` as the top-level key is required.
```bash
operations-center provisioning token seed add <UUID> proxmox-preseed \
/tmp/oc-preseed.yaml \
--description "Force reboot for Proxmox VMs"
operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed
```
### Download the OC-provisioned ISO (tested)
Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO):
```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
/tmp/IncusOS-oc.iso --type iso --architecture x86_64
ls -lh /tmp/IncusOS-oc.iso # ~3.4 GB
```
Alternative without token seeds (inline pre-seed):
```bash
operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
/tmp/oc-preseed.yaml --type iso --architecture x86_64
```
---
## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)
**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an
OC-provisioned ISO. This combines the best of both worlds:
- **OC**: auto-registration token embedded in the boot ISO
- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot),
install monitoring, media cleanup, IP detection, remote setup
The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides
force_reboot, hostname, and app selection. IncusOS reads both sources and
merges them -- they coexist without conflict.
### Deployment config
```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
cores: 4
memory: 8192
disk: 50
start_vmid: 400 # Use high VMIDs to avoid collisions with VMs outside pool
vms:
- name: oc-node-01
app: incus
apply_defaults: true
- name: oc-node-02
app: incus
apply_defaults: true
- name: oc-node-03
app: incus
apply_defaults: true
```
**VMID note**: if your Proxmox API token is scoped to a resource pool, it
cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges
to avoid VMID collisions with invisible VMs.
### Deploy (tested)
```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```
`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA
generation (force_reboot + hostname per node), install monitoring via
blockstat polling, media removal, boot order switch, IP detection, and
incus remote setup.
### Verify OC auto-registration (tested)
After nodes boot from disk, they auto-register with OC within ~30 seconds:
```bash
operations-center provisioning server list
```
Nodes register with their hostname (set via SEED_DATA), so no renaming needed.
Each node appears with type `incus`, status `ready`, and its connection URL.
**Tested result**: all 3 nodes auto-registered with correct hostnames,
correct IPs, type `incus`, status `ready`. No manual intervention needed.
---
## Part 4: Form Cluster via Operations Center
### Prepare application config
Create an Incus preseed that injects the client certificate into the cluster
(so the `incus` CLI can connect directly):
```yaml
# /tmp/oc-app-config.yaml
certificates:
- type: client
name: lab-client-cert
certificate: |-
-----BEGIN CERTIFICATE-----
<contents of ~/.config/incus/client.crt>
-----END CERTIFICATE-----
```
### Form cluster via OC CLI (tested)
```bash
operations-center provisioning cluster add oc-cluster \
https://<NODE_01_IP>:8443 \
--server-names oc-node-01,oc-node-02,oc-node-03 \
--server-type incus \
--application-seed-config /tmp/oc-app-config.yaml
```
OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on the first node
3. Joins remaining nodes (handles storage pool + network creation)
4. Applies the application seed (injects client cert)
5. Runs Terraform/OpenTofu for post-cluster configuration
### apply_defaults conflict (tested finding)
When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`),
each node already has a `local` ZFS storage pool, `incusbr0` network bridge,
and the client certificate. OC's Terraform post-clustering step then tries to
create the same resources and fails with:
- "Certificate already in trust store"
- "Network is not in pending state"
- "Storage pool is not in pending state"
**The cluster itself forms successfully** despite these Terraform errors. All
3 nodes join the cluster correctly. The Terraform errors are non-fatal because
the resources already exist. However, the `cluster artifact list` will be empty
since Terraform didn't complete.
**Recommended approaches** for future deployments:
1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore
Terraform errors. Skip `--application-seed-config` since the cert is already
injected by the seed.
2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and
without cert injection in the seed. Let OC handle everything via Terraform.
This is the "clean" approach but hasn't been tested yet.
### Verify cluster (tested)
```bash
# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster
# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01 # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:
```
### Cluster artifacts
OC generates Terraform/OpenTofu artifacts on successful cluster formation:
```bash
operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)
```
---
## Part 5: OC Management Capabilities (Tested)
### Inventory browsing
OC inventory is **not real-time** -- it requires an explicit `cluster resync`
to pick up changes made via the `incus` CLI:
```bash
# Force sync first
operations-center provisioning cluster resync oc-cluster
# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list
```
**Tested**: after creating workloads via `incus launch`, OC inventory was
empty until `cluster resync` was run. After resync, all instances, storage
pools, networks, profiles, projects, and images appeared correctly.
### Inventory sync latency
- Creating workloads via `incus` CLI: not visible in OC until resync
- After resync: ~5 seconds for inventory to populate
- Live migration: location change not reflected until resync
- Evacuation: instance relocation not reflected until resync
- OC `inventory instance show <UUID>` shows: name, project, cluster, server, timestamp
### Server management (tested)
```bash
operations-center provisioning server list # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name> # WARNING: see below
operations-center provisioning server system poweroff <name>
```
**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot`
command sends a guest-level (ACPI) reboot. On Proxmox VMs with TPM2-based
disk encryption, this corrupts the TPM PCR measurements. The IncusOS node
becomes pingable but all services fail to start (port 8443 never opens).
Even a Proxmox-level stop/start does not recover the node. The only fix is
to destroy and redeploy the VM.
**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot.
The proper lifecycle for maintenance is:
1. `incus cluster evacuate <remote>:<member> --force`
2. Proxmox stop the VM
3. Proxmox start the VM
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore <remote>:<member> --force`
**OC does not track Incus cluster member state**: the server list always
shows `ready` status even when the Incus cluster reports `EVACUATED`.
### Server edit commands
Network and storage have `edit` subcommands (no `show`):
```bash
operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>
```
### Update management (tested)
```bash
operations-center provisioning update list # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh # Check for new updates
```
Updates show comprehensive file lists including sysexts: incus, operations-center,
gpu-support, incus-ceph, incus-linstor, migration-manager, debug.
### System configuration (tested)
```bash
operations-center system settings show # log_level
operations-center system security show # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show # Update source, filter expressions, signature CA
```
Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`.
### Admin / IncusOS management (tested)
```bash
operations-center admin os show # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log
```
### Cluster operations (tested)
```bash
operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>
```
### Cluster templates
```bash
operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>
```
---
## Part 6: Cluster Lifecycle (Tested)
### Live migration visibility through OC (tested)
```bash
# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01
# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03
# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list # now shows new location
```
**Tested**: ~141 MB/s migration speed. OC inventory reflected the location
change only after explicit `cluster resync`.
### Cluster evacuation through OC (tested)
```bash
incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list # shows workloads moved to other nodes
incus cluster restore oc-node-01:oc-node-01 --force
```
### Node replacement lifecycle (tested)
Full procedure tested after OC reboot broke a node:
```bash
# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
# 2. Destroy broken VM via Proxmox (stop + delete)
# 3. Redeploy via incusos-proxmox --iso (single-node config)
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
# 4. New node auto-registers with OC (same hostname, new IP)
# NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
# while OC thinks it's in a cluster. Resync doesn't fix stale entries.
# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# Remove apply_defaults resources
incus config unset oc-node-03: storage.backups_volume
incus config unset oc-node-03: storage.images_volume
incus storage volume delete oc-node-03:local backups
incus storage volume delete oc-node-03:local images
incus profile device remove oc-node-03:default root
incus profile device remove oc-node-03:default eth0
incus storage delete oc-node-03:local
incus network delete oc-node-03:incusbr0
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
```
### Factory reset (not tested)
```bash
# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
```
---
## OC CLI Command Reference
### Remote management
| Command | Description |
|---------|-------------|
| `operations-center remote add <name> <url> --auth-type tls` | Add OC remote |
| `operations-center remote switch <name>` | Set default remote |
| `operations-center remote list` | List configured remotes |
| `operations-center remote remove <name>` | Remove a remote |
### Provisioning tokens
| Command | Description |
|---------|-------------|
| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token |
| `provisioning token list` | List tokens |
| `provisioning token show <UUID>` | Token details |
| `provisioning token remove <UUID>` | Delete token |
| `provisioning token seed add <UUID> <name> <file> --description "..."` | Attach named seed |
| `provisioning token seed list <UUID>` | List token seeds |
| `provisioning token seed show <UUID> <name>` | Show seed details |
| `provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64` | ISO from seed |
| `provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64` | ISO from inline preseed |
### Provisioning servers
| Command | Description |
|---------|-------------|
| `provisioning server list` | List registered servers (includes OC itself) |
| `provisioning server show <name>` | Server details + certificate |
| `provisioning server rename <old> <new>` | Rename server |
| `provisioning server resync <name>` | Resync single server |
| `provisioning server remove <name>` | Remove (fails if in cluster) |
| `provisioning server edit <name>` | Edit server (interactive) |
| `provisioning server system reboot <name>` | Guest reboot (DANGEROUS on Proxmox) |
| `provisioning server system poweroff <name>` | Guest poweroff |
| `provisioning server system network edit <name>` | Edit network config |
| `provisioning server system storage edit <name>` | Edit storage config |
### Provisioning clusters
| Command | Description |
|---------|-------------|
| `provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>` | Form cluster |
| `provisioning cluster list` | List clusters |
| `provisioning cluster show <name>` | Cluster details + certificate |
| `provisioning cluster resync <name>` | Force inventory resync |
| `provisioning cluster rename <name> <new-name>` | Rename cluster |
| `provisioning cluster update <name> --connection-url <url>` | Update cluster URL |
| `provisioning cluster update-certificate <name> <cert> <key>` | Update cluster cert |
| `provisioning cluster remove <name>` | Delete cluster |
| `provisioning cluster factory-reset <name> <token-uuid>` | Factory reset |
| `provisioning cluster artifact list <name>` | List artifacts |
| `provisioning cluster artifact archive <name> <type> <output>` | Download artifact |
### Cluster templates
| Command | Description |
|---------|-------------|
| `provisioning cluster-template add <name> --description "..."` | Create template |
| `provisioning cluster-template list` | List templates |
| `provisioning cluster-template show <name>` | Show template |
| `provisioning cluster-template remove <name>` | Delete template |
### Inventory
| Command | Description |
|---------|-------------|
| `inventory instance list` | List all instances |
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
| `inventory storage-pool list` | List storage pools |
| `inventory network list` | List networks |
| `inventory profile list` | List profiles |
| `inventory project list` | List projects |
| `inventory image list` | List images |
### Updates
| Command | Description |
|---------|-------------|
| `provisioning update list` | List available IncusOS versions |
| `provisioning update show <UUID>` | Version details + file list |
| `provisioning update refresh` | Check for new updates |
### System
| Command | Description |
|---------|-------------|
| `system settings show` | System settings (log_level) |
| `system security show` | OIDC, OpenFGA, ACME, trusted certs |
| `system updates show` | Update source, filter, signature CA |
| `system network show` | Network config |
| `system certificate set <cert> <key>` | Set server certificate |
### Admin
| Command | Description |
|---------|-------------|
| `admin os show` | IncusOS hostname, version, uptime |
| `admin os application list` | List applications |
| `admin os application show <name>` | Application state |
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
| `admin os service show <name>` | Service config |
| `admin os debug log` | Debug logs |
All commands prefixed with `operations-center`.
---
## REST API Reference
OC listens on port **8443** (same as Incus on IncusOS). All endpoints require
TLS client certificate authentication.
### Common curl pattern
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<OC_IP>:8443/1.0/<endpoint>
```
### Key endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/1.0` | Server info |
| GET | `/1.0/provisioning/updates` | List update packages |
| GET | `/1.0/provisioning/tokens` | List tokens |
| POST | `/1.0/provisioning/tokens` | Create token |
| GET | `/1.0/provisioning/servers` | List registered servers |
| GET | `/1.0/provisioning/clusters` | List clusters |
| POST | `/1.0/provisioning/clusters` | Form cluster |
| GET | `/1.0/inventory/instances` | List all instances |
---
## Proxmox-Specific Considerations
### force_reboot is required
IncusOS does not auto-halt after installation. Without `force_reboot: true`
in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at
"please remove installation media" indefinitely.
Options for OC-provisioned ISOs on Proxmox:
1. **Pre-seed file** passed to `get-image`: include `force_reboot: true`
2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with
`force_reboot: true` only
### VM creation settings
OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS
deployments (see `incusos-proxmox`):
- `bios=ovmf`, `machine=q35` -- UEFI boot required
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
- `tpmstate0`: `version=v2.0` -- required for disk encryption
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally
- Minimum 50 GiB disk, minimum 4096 MiB RAM
### Install media removal
After installation completes, you **must** stop the VM and delete ide2 (and
ide3 if used) before booting from disk. IncusOS refuses to start if install
media is still attached.
### IP detection
IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to
find node IPs:
```bash
# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
-H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")
# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'
```
---
## Known Limitations (Tested Findings)
### Confirmed limitations (v0.2.2)
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
- **No cluster member state tracking**: OC always shows `ready`, even for
EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
create stale entries that `resync` doesn't fix
- **OC reboot breaks Proxmox VMs**: guest reboot corrupts TPM encryption
- **Token seed format quirk**: requires structured YAML with section keys,
not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry
- **No cluster resize**: cannot add/remove members from existing clusters
### Design constraints
- OC runs on IncusOS (same immutable OS as Incus nodes)
- Uses port 8443 for both API and web UI
- Requires at least one trusted client certificate in seed
- Provisioning tokens have configurable use count and lifetime
- OC adds `meshbr0` network to clusters (not present in manual clustering)
- REST API endpoint: `/1.0` (same pattern as Incus)
- Auth methods: TLS client cert or OIDC
---
## Troubleshooting (Tested)
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` |
| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. |
| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates |
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills IncusOS node | TPM PCR corruption from guest reboot | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions |
| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt |
| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion |
| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster |
| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins |
---
## Comparison: Manual vs OC Cluster Workflow (Tested)
### Manual (incusos-proxmox + incus CLI)
1. Write YAML config with VM specs
2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA
3. Manually set `core.https_address` on each node
4. Delete storage pool/network on joining nodes
5. `incus cluster enable` + `incus cluster join` per node
6. Re-add remotes after certificate changes
7. ~30 minutes for 3-node cluster
### Hybrid (incusos-proxmox --iso + OC)
1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml`
2. Create provisioning token + token seed in OC
3. Download OC-provisioned ISO via `token seed get-image`
4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml`
5. Nodes auto-register with OC (~30s each)
6. `operations-center provisioning cluster add oc-cluster ...`
7. ~20 minutes for 3-node cluster (including OC deploy)
### What OC automates that we do manually
| Step | Manual | OC |
|------|--------|-----|
| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add |
| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) |
| Cluster enable + join | Multi-step per node | Single `cluster add` command |
| Client cert injection | Via SEED_DATA | Via `--application-seed-config` |
| `meshbr0` network | Not created | Auto-created by OC clustering |
| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) |
**Trade-offs**:
- Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized
inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (breaks TPM-based encryption)
- OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state
---
## References
- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
- [IncusOS GitHub](https://github.com/lxc/incus-os)
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
- [Clustering guide](clustering-guide.md) (manual cluster formation reference)
Reference in New Issue View Git Blame Copy Permalink