maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/operations-center-guide.md

1005 lines
37 KiB
Markdown
Raw Blame History

# Operations Center Guide for IncusOS
This guide covers deploying and using Operations Center (OC) by FuturFusion --
a centralized management layer for IncusOS/Incus deployments. OC handles
provisioning, clustering, inventory, and updates through a REST API, CLI, and
web UI.
All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
on Proxmox VE 9.1.5 (nested virtualization on Intel).
---
## Overview
Operations Center is fundamentally different from managing Incus clusters
manually via `incus` CLI remotes. Key differences:
| | Manual (incus CLI) | Operations Center |
|---|---|---|
| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) |
| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI |
| Node registration | Manual remote add | Automatic self-registration on first boot |
| Inventory | `incus list` per remote | Centralized view across all clusters |
| Updates | Manual per-node | OC manages update channels and rollouts |
| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned |
**Key constraint**: OC only manages nodes it has provisioned. There is no
"adopt existing node" workflow. Nodes must boot from an OC-generated ISO
containing a provisioning token, then self-register on first boot.
---
## Prerequisites
### Install OC CLI
```bash
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center --version # Note: use --version, not 'version' subcommand
```
### Set up OC client certificates
OC uses the same TLS client certificate format as Incus:
```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```
### Generate PKCS#12 cert for browser access
The OC web UI requires a client certificate for authentication:
```bash
openssl pkcs12 -export \
-inkey ~/.config/incus/client.key \
-in ~/.config/incus/client.crt \
-out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
```
### Verify with doctor check
```bash
./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed
```
---
## Part 1: Deploy Operations Center Server
### Deployment config
Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server
(Incus nodes will be provisioned by OC itself):
```yaml
defaults:
cores: 2
memory: 4096
disk: 50
start_vmid: 920
vms:
- name: oc-server
app: operations-center
apply_defaults: true
```
### Deploy
```bash
./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
```
Note the OC IP address from `--status` output.
### Set up OC CLI remote
```bash
operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list
```
### Verify browser access
Navigate to `https://<OC_IP>:8443` in Firefox (with client.pfx imported).
The OC web UI should load and show the dashboard.
### Wait for updates to download
OC automatically downloads IncusOS update packages from upstream. At least
one update must be in `ready` state before provisioned ISOs can be generated:
```bash
operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot
```
Also via REST API:
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<OC_IP>:8443/1.0/provisioning/updates
```
---
## Part 2: Provision Incus Node ISOs
### Create a provisioning token
```bash
# CLI
operations-center provisioning token add \
--description "Lab cluster nodes" \
--uses 5
# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>
```
Via REST API:
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
-X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
-d '{"description": "Lab cluster nodes (API)", "uses": 5}'
```
### Create a token seed for Proxmox VMs (tested)
OC supports **token seeds** -- named, reusable pre-seed configs attached to a
token. The YAML format uses section-level keys (`install:`, `applications:`,
etc.) -- NOT the flat format from standard seed archives.
```yaml
# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
version: "1"
force_install: true
force_reboot: true
```
**Important**: a flat format (`version: "1"` at root) maps all fields to empty
`{}` -- they don't get assigned to any section. The structured format with
`install:` as the top-level key is required.
```bash
operations-center provisioning token seed add <UUID> proxmox-preseed \
/tmp/oc-preseed.yaml \
--description "Force reboot for Proxmox VMs"
operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed
```
### Download the OC-provisioned ISO (tested)
Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO):
```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
/tmp/IncusOS-oc.iso --type iso --architecture x86_64
ls -lh /tmp/IncusOS-oc.iso # ~3.4 GB
```
Alternative without token seeds (inline pre-seed):
```bash
operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
/tmp/oc-preseed.yaml --type iso --architecture x86_64
```
---
## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)
**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an
OC-provisioned ISO. This combines the best of both worlds:
- **OC**: auto-registration token embedded in the boot ISO
- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot),
install monitoring, media cleanup, IP detection, remote setup
The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides
force_reboot, hostname, and app selection. IncusOS reads both sources and
merges them -- they coexist without conflict.
### Deployment config
```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
cores: 4
memory: 8192
disk: 50
start_vmid: 400 # Use high VMIDs to avoid collisions with VMs outside pool
vms:
- name: oc-node-01
app: incus
apply_defaults: false # OC handles storage/network via Terraform
- name: oc-node-02
app: incus
apply_defaults: false
- name: oc-node-03
app: incus
apply_defaults: false
```
**VMID note**: if your Proxmox API token is scoped to a resource pool, it
cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges
to avoid VMID collisions with invisible VMs.
### Deploy (tested)
```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```
`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA
generation (force_reboot + hostname per node), install monitoring via
blockstat polling, media removal, boot order switch, IP detection, and
incus remote setup.
### Verify OC auto-registration (tested)
After nodes boot from disk, they auto-register with OC within ~30 seconds:
```bash
operations-center provisioning server list
```
Nodes register with their hostname (set via SEED_DATA), so no renaming needed.
Each node appears with type `incus`, status `ready`, and its connection URL.
**Tested result**: all 3 nodes auto-registered with correct hostnames,
correct IPs, type `incus`, status `ready`. No manual intervention needed.
---
## Part 4: Form Cluster via Operations Center
### Prepare application config
Create an Incus preseed that injects the client certificate into the cluster
(so the `incus` CLI can connect directly):
```yaml
# /tmp/oc-app-config.yaml
certificates:
- type: client
name: lab-client-cert
certificate: |-
-----BEGIN CERTIFICATE-----
<contents of ~/.config/incus/client.crt>
-----END CERTIFICATE-----
```
### Form cluster via OC CLI (tested)
```bash
operations-center provisioning cluster add oc-cluster \
https://<NODE_01_IP>:8443 \
--server-names oc-node-01,oc-node-02,oc-node-03 \
--server-type incus \
--application-seed-config /tmp/oc-app-config.yaml
```
OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on the first node
3. Joins remaining nodes (handles storage pool + network creation)
4. Applies the application seed (injects client cert)
5. Runs Terraform/OpenTofu for post-cluster configuration
### apply_defaults: false (tested, recommended)
**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
and recommended approach. With `apply_defaults: false`:
- Nodes still listen on port 8443 and trust preseed certificates
- The underlying ZFS dataset (`local/incus`) still exists
- But no Incus storage pool, network, or profile devices are created
- OC's Terraform handles all resource creation cleanly during cluster formation
**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
and injects the client certificate via `--application-seed-config`. The only
Terraform error is "Certificate already in trust store" when the cert was also
injected via SEED_DATA -- avoid this by only injecting the cert through the
application seed config, not the SEED_DATA.
### apply_defaults: true (also works, with caveats)
When using `apply_defaults: true`, each node already has a `local` ZFS storage
pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
then fails with:
- "Certificate already in trust store"
- "Network is not in pending state"
- "Storage pool is not in pending state"
**The cluster itself forms successfully** despite these Terraform errors.
However, `cluster artifact list` will be empty since Terraform didn't complete.
### Verify cluster (tested)
```bash
# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster
# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01 # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:
```
### Cluster artifacts
OC generates Terraform/OpenTofu artifacts on successful cluster formation:
```bash
operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)
```
---
## Part 5: OC Management Capabilities (Tested)
### Inventory browsing
OC inventory is **not real-time** -- it requires an explicit `cluster resync`
to pick up changes made via the `incus` CLI:
```bash
# Force sync first
operations-center provisioning cluster resync oc-cluster
# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list
```
**Tested**: after creating workloads via `incus launch`, OC inventory was
empty until `cluster resync` was run. After resync, all instances, storage
pools, networks, profiles, projects, and images appeared correctly.
### Inventory sync latency
- Creating workloads via `incus` CLI: not visible in OC until resync
- After resync: ~5 seconds for inventory to populate
- Live migration: location change not reflected until resync
- Evacuation: instance relocation not reflected until resync
- OC `inventory instance show <UUID>` shows: name, project, cluster, server, timestamp
### Server management (tested)
```bash
operations-center provisioning server list # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name> # WARNING: see below
operations-center provisioning server system poweroff <name>
```
**WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system
reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS
nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot,
all recover in ~50s with containers and data intact). The failure is
**OC-specific**: the OC agent runs on every boot and can fail with:
- "failed to parse update frequency - invalid crontab expression" (OC
pushed invalid update config)
- "error from operations center: forbidden: failed creating server:
constraint violation" (OC trying to re-register an existing server)
- Incus stuck at "starting application name=incus" (cluster peers are in
boot loops, so quorum cannot form)
These failures cause boot loops or hung services. Even Proxmox stop/start
does not recover because the OC state on disk remains broken. The only
fix is to destroy and redeploy the VM.
**Safe reboot**: Proxmox stop/start and guest reboot both work on
standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on
OC-managed nodes.
The proper lifecycle for maintenance is:
1. `incus cluster evacuate <remote>:<member> --force`
2. Proxmox stop the VM
3. Proxmox start the VM
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore <remote>:<member> --force`
### IncusOS scrub_schedule fix via API (tested)
IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
with an empty `scrub_schedule` field. The `registerJobs()` function calls
`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
is briefly reachable during each crash cycle (Incus starts before the scheduler).
**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):
```bash
# Check current storage config
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<NODE_IP>:8443/os/1.0/system/storage
# If scrub_schedule is empty, fix it
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
-X PUT -H "Content-Type: application/json" \
-d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
https://<NODE_IP>:8443/os/1.0/system/storage
```
**Key details**:
- The `/os/` prefix is required — this proxies to the IncusOS daemon API
- PATCH returns 501 (not implemented); use PUT instead
- `incus admin os system storage edit` fails because it reads the current
(corrupt) config and validates it before allowing edits
- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
fixes this on every deployed node (safe — returns early if already set)
- Filed upstream as IncusOS issue #843
### Node failure and recovery (tested)
Proxmox hard-stop simulates node crash. Incus cluster detects failure via
heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):
```bash
# Simulate crash: Proxmox API stop
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# After ~40s: incus cluster list shows OFFLINE with heartbeat info
# Restart: Proxmox API start
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# After ~60s boot: node auto-rejoins cluster (ONLINE)
# Containers on the node auto-start after recovery
```
**OC does not detect node failures**: `provisioning server show` still shows
`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.
**OC does not track Incus cluster member state**: the server list always
shows `ready` status even when the Incus cluster reports `EVACUATED`.
### Server edit commands
Network and storage have `edit` subcommands (no `show`):
```bash
operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>
```
### Update management (tested)
```bash
operations-center provisioning update list # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh # Check for new updates
```
Updates show comprehensive file lists including sysexts: incus, operations-center,
gpu-support, incus-ceph, incus-linstor, migration-manager, debug.
### System configuration (tested)
```bash
operations-center system settings show # log_level
operations-center system security show # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show # Update source, filter expressions, signature CA
```
Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`.
### Admin / IncusOS management (tested)
```bash
operations-center admin os show # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log
```
### Cluster operations (tested)
```bash
operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>
```
### Cluster templates
```bash
operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>
```
---
## Part 6: Cluster Lifecycle (Tested)
### Live migration visibility through OC (tested)
```bash
# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01
# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03
# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list # now shows new location
```
**Tested**: ~141 MB/s migration speed. OC inventory reflected the location
change only after explicit `cluster resync`.
### Cluster evacuation through OC (tested)
```bash
incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list # shows workloads moved to other nodes
incus cluster restore oc-node-01:oc-node-01 --force
```
### Node replacement lifecycle (tested)
Full procedure tested after OC reboot broke a node:
```bash
# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
# 2. Destroy broken VM via Proxmox (stop + delete)
# 3. Redeploy via incusos-proxmox --iso (single-node config)
# Use apply_defaults: false for OC-managed nodes
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
# 4. New node auto-registers with OC (same hostname, new IP)
# NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
# while OC thinks it's in a cluster. Resync doesn't fix stale entries.
# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# With apply_defaults: false, no cleanup needed (no pool/network to delete)
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
```
### Node failure simulation (tested)
Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):
```bash
# Simulate crash
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# incus cluster list shows OFFLINE after ~40s
# Restart
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# Node auto-rejoins after ~60s boot. Containers auto-start.
# OC still shows "ready" (does not detect node failures)
```
### Factory reset (not tested)
```bash
# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
```
---
## OC CLI Command Reference
### Remote management
| Command | Description |
|---------|-------------|
| `operations-center remote add <name> <url> --auth-type tls` | Add OC remote |
| `operations-center remote switch <name>` | Set default remote |
| `operations-center remote list` | List configured remotes |
| `operations-center remote remove <name>` | Remove a remote |
### Provisioning tokens
| Command | Description |
|---------|-------------|
| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token |
| `provisioning token list` | List tokens |
| `provisioning token show <UUID>` | Token details |
| `provisioning token remove <UUID>` | Delete token |
| `provisioning token seed add <UUID> <name> <file> --description "..."` | Attach named seed |
| `provisioning token seed list <UUID>` | List token seeds |
| `provisioning token seed show <UUID> <name>` | Show seed details |
| `provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64` | ISO from seed |
| `provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64` | ISO from inline preseed |
### Provisioning servers
| Command | Description |
|---------|-------------|
| `provisioning server list` | List registered servers (includes OC itself) |
| `provisioning server show <name>` | Server details + certificate |
| `provisioning server rename <old> <new>` | Rename server |
| `provisioning server resync <name>` | Resync single server |
| `provisioning server remove <name>` | Remove (fails if in cluster) |
| `provisioning server edit <name>` | Edit server (interactive) |
| `provisioning server system reboot <name>` | Guest reboot (DANGEROUS on Proxmox) |
| `provisioning server system poweroff <name>` | Guest poweroff |
| `provisioning server system network edit <name>` | Edit network config |
| `provisioning server system storage edit <name>` | Edit storage config |
### Provisioning clusters
| Command | Description |
|---------|-------------|
| `provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>` | Form cluster |
| `provisioning cluster list` | List clusters |
| `provisioning cluster show <name>` | Cluster details + certificate |
| `provisioning cluster resync <name>` | Force inventory resync |
| `provisioning cluster rename <name> <new-name>` | Rename cluster |
| `provisioning cluster update <name> --connection-url <url>` | Update cluster URL |
| `provisioning cluster update-certificate <name> <cert> <key>` | Update cluster cert |
| `provisioning cluster remove <name>` | Delete cluster |
| `provisioning cluster factory-reset <name> <token-uuid>` | Factory reset |
| `provisioning cluster artifact list <name>` | List artifacts |
| `provisioning cluster artifact archive <name> <type> <output>` | Download artifact |
### Cluster templates
| Command | Description |
|---------|-------------|
| `provisioning cluster-template add <name> --description "..."` | Create template |
| `provisioning cluster-template list` | List templates |
| `provisioning cluster-template show <name>` | Show template |
| `provisioning cluster-template remove <name>` | Delete template |
### Inventory
| Command | Description |
|---------|-------------|
| `inventory instance list` | List all instances |
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
| `inventory storage-pool list` | List storage pools |
| `inventory storage-volume list` | List storage volumes |
| `inventory storage-bucket list` | List storage buckets |
| `inventory network list` | List networks |
| `inventory network-acl list` | List network ACLs |
| `inventory network-address-set list` | List network address sets |
| `inventory network-forward list` | List network forwards |
| `inventory network-integration list` | List network integrations |
| `inventory network-load-balancer list` | List network load balancers |
| `inventory network-peer list` | List network peers |
| `inventory network-zone list` | List network zones |
| `inventory profile list` | List profiles |
| `inventory project list` | List projects |
| `inventory image list` | List images |
| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |
### Updates
| Command | Description |
|---------|-------------|
| `provisioning update list` | List available IncusOS versions |
| `provisioning update show <UUID>` | Version details + file list |
| `provisioning update refresh` | Check for new updates |
### System
| Command | Description |
|---------|-------------|
| `system settings show` | System settings (log_level) |
| `system security show` | OIDC, OpenFGA, ACME, trusted certs |
| `system updates show` | Update source, filter, signature CA |
| `system network show` | Network config |
| `system certificate set <cert> <key>` | Set server certificate |
### Admin
| Command | Description |
|---------|-------------|
| `admin os show` | IncusOS hostname, version, uptime |
| `admin os application list` | List applications |
| `admin os application show <name>` | Application state |
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
| `admin os service show <name>` | Service config |
| `admin os debug log` | Debug logs |
| `admin os debug processes` | System process list |
| `admin os debug secureboot` | Secure Boot debug info |
All commands prefixed with `operations-center`.
---
## REST API Reference
OC listens on port **8443** (same as Incus on IncusOS). All endpoints require
TLS client certificate authentication.
### Common curl pattern
```bash
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<OC_IP>:8443/1.0/<endpoint>
```
### Key endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/1.0` | Server info |
| GET | `/1.0/provisioning/updates` | List update packages |
| GET | `/1.0/provisioning/tokens` | List tokens |
| POST | `/1.0/provisioning/tokens` | Create token |
| GET | `/1.0/provisioning/servers` | List registered servers |
| GET | `/1.0/provisioning/clusters` | List clusters |
| POST | `/1.0/provisioning/clusters` | Form cluster |
| GET | `/1.0/inventory/instances` | List all instances |
| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |
### IncusOS REST API (via Incus proxy)
IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
prefix on port 8443:
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
| PUT | `/os/1.0/system/storage` | Update storage config |
| GET | `/os/1.0` | IncusOS system info |
---
## Proxmox-Specific Considerations
### force_reboot is required
IncusOS does not auto-halt after installation. Without `force_reboot: true`
in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at
"please remove installation media" indefinitely.
Options for OC-provisioned ISOs on Proxmox:
1. **Pre-seed file** passed to `get-image`: include `force_reboot: true`
2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with
`force_reboot: true` only
### VM creation settings
OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS
deployments (see `incusos-proxmox`):
- `bios=ovmf`, `machine=q35` -- UEFI boot required
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
- `tpmstate0`: `version=v2.0` -- required for disk encryption
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally
- Minimum 50 GiB disk, minimum 4096 MiB RAM
### Install media removal
After installation completes, you **must** stop the VM and delete ide2 (and
ide3 if used) before booting from disk. IncusOS refuses to start if install
media is still attached.
### IP detection
IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to
find node IPs:
```bash
# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
-H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")
# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'
```
---
## Known Limitations (Tested Findings)
### Confirmed limitations (v0.3.0)
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
- **No cluster member state tracking**: OC always shows `ready`, even for
EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
create stale entries that `resync` doesn't fix
- **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause
boot loops or hung services (not a dqlite or TPM issue -- standalone nodes
reboot fine)
- **Token seed format quirk**: requires structured YAML with section keys,
not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry
- **No cluster resize**: cannot add/remove members from existing clusters
### Design constraints
- OC runs on IncusOS (same immutable OS as Incus nodes)
- Uses port 8443 for both API and web UI
- Requires at least one trusted client certificate in seed
- Provisioning tokens have configurable use count and lifetime
- OC adds `meshbr0` network to clusters (not present in manual clustering)
- REST API endpoint: `/1.0` (same pattern as Incus)
- Auth methods: TLS client cert or OIDC
---
## Troubleshooting (Tested)
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` |
| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. |
| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates |
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions |
| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt |
| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion |
| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster |
| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins |
---
## Comparison: Manual vs OC Cluster Workflow (Tested)
### Manual (incusos-proxmox + incus CLI)
1. Write YAML config with VM specs
2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA
3. Manually set `core.https_address` on each node
4. Delete storage pool/network on joining nodes
5. `incus cluster enable` + `incus cluster join` per node
6. Re-add remotes after certificate changes
7. ~30 minutes for 3-node cluster
### Hybrid (incusos-proxmox --iso + OC)
1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml`
2. Create provisioning token + token seed in OC
3. Download OC-provisioned ISO via `token seed get-image`
4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml`
5. Nodes auto-register with OC (~30s each)
6. `operations-center provisioning cluster add oc-cluster ...`
7. ~20 minutes for 3-node cluster (including OC deploy)
### What OC automates that we do manually
| Step | Manual | OC |
|------|--------|-----|
| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add |
| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) |
| Cluster enable + join | Multi-step per node | Single `cluster add` command |
| Client cert injection | Via SEED_DATA | Via `--application-seed-config` |
| `meshbr0` network | Not created | Auto-created by OC clustering |
| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) |
**Trade-offs**:
- Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized
inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing
loops -- standalone reboot is safe)
- OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state
---
## Open Research Items
- **OC reboot failure mechanism** (resolved): guest reboot is safe on
standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
in ~50s). The failure is OC-specific: the OC agent pushes config via the
IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
values (e.g., cron expression where Go duration is expected) crash the
daemon. Root cause is NOT dqlite corruption or TPM issues.
- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
is now the tested and recommended approach. OC's Terraform handles storage pool,
network, and cert creation cleanly when nodes start without defaults.
`apply_defaults: true` also works but produces non-fatal Terraform errors.
- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
likely a race condition between REST API startup and `registerJobs()`, or a
zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
auto-heals this proactively on every deploy.
- **Stale server entries after out-of-band changes**: when cluster members are
removed via `incus cluster remove` (not through OC), stale entries persist in
OC's server list. `cluster resync` doesn't fix them. The entry can't be
removed while OC considers it part of a cluster.
- **Inventory sync is manual**: OC inventory requires explicit `cluster resync`
to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
This applies to instance creation, migration, evacuation, and deletion.
- **Node failure detection**: OC does not detect node failures. `provisioning
server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.
---
## References
- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
- [IncusOS GitHub](https://github.com/lxc/incus-os)
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
- [Clustering guide](clustering-guide.md) (manual cluster formation reference)
Reference in New Issue View Git Blame Copy Permalink