# Operations Center Guide for IncusOS

This guide covers deploying and using Operations Center (OC) by FuturFusion --
a centralized management layer for IncusOS/Incus deployments. OC handles
provisioning, clustering, inventory, and updates through a REST API, CLI, and
web UI.

All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553
on Proxmox VE 9.1.5 (nested virtualization on Intel).

---

## Overview

Operations Center is fundamentally different from managing Incus clusters
manually via `incus` CLI remotes. Key differences:

| | Manual (incus CLI) | Operations Center |
|---|---|---|
| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) |
| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI |
| Node registration | Manual remote add | Automatic self-registration on first boot |
| Inventory | `incus list` per remote | Centralized view across all clusters |
| Updates | Manual per-node | OC manages update channels and rollouts |
| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned |

**Key constraint**: OC only manages nodes it has provisioned. There is no
"adopt existing node" workflow. Nodes must boot from an OC-generated ISO
containing a provisioning token, then self-register on first boot.

---

## Prerequisites

### Install OC CLI

```bash
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center version
```

### Set up OC client certificates

OC uses the same TLS client certificate format as Incus:

```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```

### Generate PKCS#12 cert for browser access

The OC web UI requires a client certificate for authentication:

```bash
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
```

### Verify with doctor check

```bash
./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed
```

---

## Part 1: Deploy Operations Center Server

### Deployment config

Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server
(Incus nodes will be provisioned by OC itself):

```yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
```

### Deploy

```bash
./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
```

Note the OC IP address from `--status` output.

### Set up OC CLI remote

```bash
operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list
```

### Verify browser access

Navigate to `https://<OC_IP>:8443` in Firefox (with client.pfx imported).
The OC web UI should load and show the dashboard.

### Wait for updates to download

OC automatically downloads IncusOS update packages from upstream. At least
one update must be in `ready` state before provisioned ISOs can be generated:

```bash
operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot
```

Also via REST API:

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/provisioning/updates
```

---

## Part 2: Provision Incus Node ISOs

### Create a provisioning token

```bash
# CLI
operations-center provisioning token add \
    --description "Lab cluster nodes" \
    --uses 5

# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>
```

Via REST API:

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
     -d '{"description": "Lab cluster nodes (API)", "uses": 5}'
```

### Create a token seed for Proxmox VMs (tested)

OC supports **token seeds** -- named, reusable pre-seed configs attached to a
token. The YAML format uses section-level keys (`install:`, `applications:`,
etc.) -- NOT the flat format from standard seed archives.

```yaml
# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
  version: "1"
  force_install: true
  force_reboot: true
```

**Important**: a flat format (`version: "1"` at root) maps all fields to empty
`{}` -- they don't get assigned to any section. The structured format with
`install:` as the top-level key is required.

```bash
operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml \
    --description "Force reboot for Proxmox VMs"

operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed
```

### Download the OC-provisioned ISO (tested)

Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO):

```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

ls -lh /tmp/IncusOS-oc.iso  # ~3.4 GB
```

Alternative without token seeds (inline pre-seed):

```bash
operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
    /tmp/oc-preseed.yaml --type iso --architecture x86_64
```

---

## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an
OC-provisioned ISO. This combines the best of both worlds:
- **OC**: auto-registration token embedded in the boot ISO
- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot),
  install monitoring, media cleanup, IP detection, remote setup

The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides
force_reboot, hostname, and app selection. IncusOS reads both sources and
merges them -- they coexist without conflict.

### Deployment config

```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400   # Use high VMIDs to avoid collisions with VMs outside pool

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: true

  - name: oc-node-02
    app: incus
    apply_defaults: true

  - name: oc-node-03
    app: incus
    apply_defaults: true
```

**VMID note**: if your Proxmox API token is scoped to a resource pool, it
cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges
to avoid VMID collisions with invisible VMs.

### Deploy (tested)

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```

`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA
generation (force_reboot + hostname per node), install monitoring via
blockstat polling, media removal, boot order switch, IP detection, and
incus remote setup.

### Verify OC auto-registration (tested)

After nodes boot from disk, they auto-register with OC within ~30 seconds:

```bash
operations-center provisioning server list
```

Nodes register with their hostname (set via SEED_DATA), so no renaming needed.
Each node appears with type `incus`, status `ready`, and its connection URL.

**Tested result**: all 3 nodes auto-registered with correct hostnames,
correct IPs, type `incus`, status `ready`. No manual intervention needed.

---

## Part 4: Form Cluster via Operations Center

### Prepare application config

Create an Incus preseed that injects the client certificate into the cluster
(so the `incus` CLI can connect directly):

```yaml
# /tmp/oc-app-config.yaml
certificates:
  - type: client
    name: lab-client-cert
    certificate: |-
      -----BEGIN CERTIFICATE-----
      <contents of ~/.config/incus/client.crt>
      -----END CERTIFICATE-----
```

### Form cluster via OC CLI (tested)

```bash
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on the first node
3. Joins remaining nodes (handles storage pool + network creation)
4. Applies the application seed (injects client cert)
5. Runs Terraform/OpenTofu for post-cluster configuration

### apply_defaults conflict (tested finding)

When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`),
each node already has a `local` ZFS storage pool, `incusbr0` network bridge,
and the client certificate. OC's Terraform post-clustering step then tries to
create the same resources and fails with:

- "Certificate already in trust store"
- "Network is not in pending state"
- "Storage pool is not in pending state"

**The cluster itself forms successfully** despite these Terraform errors. All
3 nodes join the cluster correctly. The Terraform errors are non-fatal because
the resources already exist. However, the `cluster artifact list` will be empty
since Terraform didn't complete.

**Recommended approaches** for future deployments:
1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore
   Terraform errors. Skip `--application-seed-config` since the cert is already
   injected by the seed.
2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and
   without cert injection in the seed. Let OC handle everything via Terraform.
   This is the "clean" approach but hasn't been tested yet.

### Verify cluster (tested)

```bash
# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01  # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:
```

### Cluster artifacts

OC generates Terraform/OpenTofu artifacts on successful cluster formation:

```bash
operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)
```

---

## Part 5: OC Management Capabilities (Tested)

### Inventory browsing

OC inventory is **not real-time** -- it requires an explicit `cluster resync`
to pick up changes made via the `incus` CLI:

```bash
# Force sync first
operations-center provisioning cluster resync oc-cluster

# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list
```

**Tested**: after creating workloads via `incus launch`, OC inventory was
empty until `cluster resync` was run. After resync, all instances, storage
pools, networks, profiles, projects, and images appeared correctly.

### Inventory sync latency

- Creating workloads via `incus` CLI: not visible in OC until resync
- After resync: ~5 seconds for inventory to populate
- Live migration: location change not reflected until resync
- Evacuation: instance relocation not reflected until resync
- OC `inventory instance show <UUID>` shows: name, project, cluster, server, timestamp

### Server management (tested)

```bash
operations-center provisioning server list      # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name>    # WARNING: see below
operations-center provisioning server system poweroff <name>
```

**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot`
command sends a guest-level (ACPI) reboot. On Proxmox VMs, this likely
corrupts the dqlite/Incus database through unclean shutdown. The IncusOS
node becomes pingable (TPM disk decryption succeeds, OS boots) but the
Incus daemon never starts (port 8443 never opens). Even a Proxmox-level
stop/start does not recover the node. The only fix is to destroy and
redeploy the VM. **Note**: this is an open research item -- the exact
failure mechanism is unconfirmed.

**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot.
The proper lifecycle for maintenance is:
1. `incus cluster evacuate <remote>:<member> --force`
2. Proxmox stop the VM
3. Proxmox start the VM
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore <remote>:<member> --force`

**OC does not track Incus cluster member state**: the server list always
shows `ready` status even when the Incus cluster reports `EVACUATED`.

### Server edit commands

Network and storage have `edit` subcommands (no `show`):

```bash
operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>
```

### Update management (tested)

```bash
operations-center provisioning update list       # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh     # Check for new updates
```

Updates show comprehensive file lists including sysexts: incus, operations-center,
gpu-support, incus-ceph, incus-linstor, migration-manager, debug.

### System configuration (tested)

```bash
operations-center system settings show   # log_level
operations-center system security show   # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show    # Update source, filter expressions, signature CA
```

Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`.

### Admin / IncusOS management (tested)

```bash
operations-center admin os show           # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list   # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log
```

### Cluster operations (tested)

```bash
operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>
```

### Cluster templates

```bash
operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>
```

---

## Part 6: Cluster Lifecycle (Tested)

### Live migration visibility through OC (tested)

```bash
# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01

# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03

# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # now shows new location
```

**Tested**: ~141 MB/s migration speed. OC inventory reflected the location
change only after explicit `cluster resync`.

### Cluster evacuation through OC (tested)

```bash
incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # shows workloads moved to other nodes

incus cluster restore oc-node-01:oc-node-01 --force
```

### Node replacement lifecycle (tested)

Full procedure tested after OC reboot broke a node:

```bash
# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

# 2. Destroy broken VM via Proxmox (stop + delete)

# 3. Redeploy via incusos-proxmox --iso (single-node config)
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml

# 4. New node auto-registers with OC (same hostname, new IP)
#    NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
#    while OC thinks it's in a cluster. Resync doesn't fix stale entries.

# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# Remove apply_defaults resources
incus config unset oc-node-03: storage.backups_volume
incus config unset oc-node-03: storage.images_volume
incus storage volume delete oc-node-03:local backups
incus storage volume delete oc-node-03:local images
incus profile device remove oc-node-03:default root
incus profile device remove oc-node-03:default eth0
incus storage delete oc-node-03:local
incus network delete oc-node-03:incusbr0
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)

# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
```

### Factory reset (not tested)

```bash
# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
```

---

## OC CLI Command Reference

### Remote management

| Command | Description |
|---------|-------------|
| `operations-center remote add <name> <url> --auth-type tls` | Add OC remote |
| `operations-center remote switch <name>` | Set default remote |
| `operations-center remote list` | List configured remotes |
| `operations-center remote remove <name>` | Remove a remote |

### Provisioning tokens

| Command | Description |
|---------|-------------|
| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token |
| `provisioning token list` | List tokens |
| `provisioning token show <UUID>` | Token details |
| `provisioning token remove <UUID>` | Delete token |
| `provisioning token seed add <UUID> <name> <file> --description "..."` | Attach named seed |
| `provisioning token seed list <UUID>` | List token seeds |
| `provisioning token seed show <UUID> <name>` | Show seed details |
| `provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64` | ISO from seed |
| `provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64` | ISO from inline preseed |

### Provisioning servers

| Command | Description |
|---------|-------------|
| `provisioning server list` | List registered servers (includes OC itself) |
| `provisioning server show <name>` | Server details + certificate |
| `provisioning server rename <old> <new>` | Rename server |
| `provisioning server resync <name>` | Resync single server |
| `provisioning server remove <name>` | Remove (fails if in cluster) |
| `provisioning server edit <name>` | Edit server (interactive) |
| `provisioning server system reboot <name>` | Guest reboot (DANGEROUS on Proxmox) |
| `provisioning server system poweroff <name>` | Guest poweroff |
| `provisioning server system network edit <name>` | Edit network config |
| `provisioning server system storage edit <name>` | Edit storage config |

### Provisioning clusters

| Command | Description |
|---------|-------------|
| `provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>` | Form cluster |
| `provisioning cluster list` | List clusters |
| `provisioning cluster show <name>` | Cluster details + certificate |
| `provisioning cluster resync <name>` | Force inventory resync |
| `provisioning cluster rename <name> <new-name>` | Rename cluster |
| `provisioning cluster update <name> --connection-url <url>` | Update cluster URL |
| `provisioning cluster update-certificate <name> <cert> <key>` | Update cluster cert |
| `provisioning cluster remove <name>` | Delete cluster |
| `provisioning cluster factory-reset <name> <token-uuid>` | Factory reset |
| `provisioning cluster artifact list <name>` | List artifacts |
| `provisioning cluster artifact archive <name> <type> <output>` | Download artifact |

### Cluster templates

| Command | Description |
|---------|-------------|
| `provisioning cluster-template add <name> --description "..."` | Create template |
| `provisioning cluster-template list` | List templates |
| `provisioning cluster-template show <name>` | Show template |
| `provisioning cluster-template remove <name>` | Delete template |

### Inventory

| Command | Description |
|---------|-------------|
| `inventory instance list` | List all instances |
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
| `inventory storage-pool list` | List storage pools |
| `inventory network list` | List networks |
| `inventory profile list` | List profiles |
| `inventory project list` | List projects |
| `inventory image list` | List images |

### Updates

| Command | Description |
|---------|-------------|
| `provisioning update list` | List available IncusOS versions |
| `provisioning update show <UUID>` | Version details + file list |
| `provisioning update refresh` | Check for new updates |

### System

| Command | Description |
|---------|-------------|
| `system settings show` | System settings (log_level) |
| `system security show` | OIDC, OpenFGA, ACME, trusted certs |
| `system updates show` | Update source, filter, signature CA |
| `system network show` | Network config |
| `system certificate set <cert> <key>` | Set server certificate |

### Admin

| Command | Description |
|---------|-------------|
| `admin os show` | IncusOS hostname, version, uptime |
| `admin os application list` | List applications |
| `admin os application show <name>` | Application state |
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
| `admin os service show <name>` | Service config |
| `admin os debug log` | Debug logs |

All commands prefixed with `operations-center`.

---

## REST API Reference

OC listens on port **8443** (same as Incus on IncusOS). All endpoints require
TLS client certificate authentication.

### Common curl pattern

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/<endpoint>
```

### Key endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/1.0` | Server info |
| GET | `/1.0/provisioning/updates` | List update packages |
| GET | `/1.0/provisioning/tokens` | List tokens |
| POST | `/1.0/provisioning/tokens` | Create token |
| GET | `/1.0/provisioning/servers` | List registered servers |
| GET | `/1.0/provisioning/clusters` | List clusters |
| POST | `/1.0/provisioning/clusters` | Form cluster |
| GET | `/1.0/inventory/instances` | List all instances |

---

## Proxmox-Specific Considerations

### force_reboot is required

IncusOS does not auto-halt after installation. Without `force_reboot: true`
in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at
"please remove installation media" indefinitely.

Options for OC-provisioned ISOs on Proxmox:
1. **Pre-seed file** passed to `get-image`: include `force_reboot: true`
2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with
   `force_reboot: true` only

### VM creation settings

OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS
deployments (see `incusos-proxmox`):

- `bios=ovmf`, `machine=q35` -- UEFI boot required
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
- `tpmstate0`: `version=v2.0` -- required for disk encryption
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally
- Minimum 50 GiB disk, minimum 4096 MiB RAM

### Install media removal

After installation completes, you **must** stop the VM and delete ide2 (and
ide3 if used) before booting from disk. IncusOS refuses to start if install
media is still attached.

### IP detection

IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to
find node IPs:

```bash
# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
    -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")

# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'
```

---

## Known Limitations (Tested Findings)

### Confirmed limitations (v0.2.2)

- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
- **No cluster member state tracking**: OC always shows `ready`, even for
  EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
  create stale entries that `resync` doesn't fix
- **OC reboot breaks Proxmox VMs**: guest reboot likely corrupts dqlite/Incus
  database (open research item -- OS boots but Incus daemon doesn't start)
- **Token seed format quirk**: requires structured YAML with section keys,
  not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry
- **No cluster resize**: cannot add/remove members from existing clusters

### Design constraints

- OC runs on IncusOS (same immutable OS as Incus nodes)
- Uses port 8443 for both API and web UI
- Requires at least one trusted client certificate in seed
- Provisioning tokens have configurable use count and lifetime
- OC adds `meshbr0` network to clusters (not present in manual clustering)
- REST API endpoint: `/1.0` (same pattern as Incus)
- Auth methods: TLS client cert or OIDC

---

## Troubleshooting (Tested)

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` |
| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. |
| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates |
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills IncusOS node | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions |
| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt |
| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion |
| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster |
| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins |

---

## Comparison: Manual vs OC Cluster Workflow (Tested)

### Manual (incusos-proxmox + incus CLI)

1. Write YAML config with VM specs
2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA
3. Manually set `core.https_address` on each node
4. Delete storage pool/network on joining nodes
5. `incus cluster enable` + `incus cluster join` per node
6. Re-add remotes after certificate changes
7. ~30 minutes for 3-node cluster

### Hybrid (incusos-proxmox --iso + OC)

1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml`
2. Create provisioning token + token seed in OC
3. Download OC-provisioned ISO via `token seed get-image`
4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml`
5. Nodes auto-register with OC (~30s each)
6. `operations-center provisioning cluster add oc-cluster ...`
7. ~20 minutes for 3-node cluster (including OC deploy)

### What OC automates that we do manually

| Step | Manual | OC |
|------|--------|-----|
| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add |
| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) |
| Cluster enable + join | Multi-step per node | Single `cluster add` command |
| Client cert injection | Via SEED_DATA | Via `--application-seed-config` |
| `meshbr0` network | Not created | Auto-created by OC clustering |
| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) |

**Trade-offs**:
- Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized
  inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (likely corrupts dqlite database)
- OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state

---

## Open Research Items

These are observed behaviors whose root cause is not yet confirmed:

- **OC reboot failure mechanism**: guest-level (ACPI) reboot on Proxmox VMs
  causes the Incus daemon to never start. The OS boots (node is pingable,
  TPM disk decryption succeeds) but port 8443 never opens. Likely cause:
  dqlite database corruption from unclean shutdown. Proxmox stop/start does
  not recover the node. Needs investigation with IncusOS debug logs.

- **apply_defaults conflict with OC Terraform**: when nodes have
  `apply_defaults: true`, OC's post-clustering Terraform step fails with
  "already in trust store" / "not in pending state" errors. The cluster forms
  successfully despite these errors. Unclear whether skipping `apply_defaults`
  would allow clean Terraform completion (untested).

- **Stale server entries after out-of-band changes**: when cluster members are
  removed via `incus cluster remove` (not through OC), stale entries persist in
  OC's server list. `cluster resync` doesn't fix them. The entry can't be
  removed while OC considers it part of a cluster.

- **Inventory sync is manual**: OC inventory requires explicit `cluster resync`
  to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
  This applies to instance creation, migration, evacuation, and deletion.

---

## References

- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
- [IncusOS GitHub](https://github.com/lxc/incus-os)
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
- [Clustering guide](clustering-guide.md) (manual cluster formation reference)