# Operations Center Guide for IncusOS

This guide covers deploying and using Operations Center (OC) by FuturFusion --
a centralized management layer for IncusOS/Incus deployments. OC handles
provisioning, clustering, inventory, and updates through a REST API, CLI, and
web UI.

All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
on Proxmox VE 9.1.5 (nested virtualization on Intel).

---

## Overview

Operations Center is fundamentally different from managing Incus clusters
manually via `incus` CLI remotes. Key differences:

| | Manual (incus CLI) | Operations Center |
|---|---|---|
| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) |
| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI |
| Node registration | Manual remote add | Automatic self-registration on first boot |
| Inventory | `incus list` per remote | Centralized view across all clusters |
| Updates | Manual per-node | OC manages update channels and rollouts |
| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned |

**Key constraint**: OC only manages nodes it has provisioned. There is no
"adopt existing node" workflow. Nodes must boot from an OC-generated ISO
containing a provisioning token, then self-register on first boot.

---

## Prerequisites

### Install OC CLI

```bash
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center --version  # Note: use --version, not 'version' subcommand
```

### Set up OC client certificates

OC uses the same TLS client certificate format as Incus:

```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```

### Generate PKCS#12 cert for browser access

The OC web UI requires a client certificate for authentication:

```bash
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx
# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
```

### Verify with doctor check

```bash
./incusos/incusos-proxmox --doctor
# Should report OC CLI as installed
```

---

## Part 1: Deploy Operations Center Server

### Deployment config

Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server
(Incus nodes will be provisioned by OC itself):

```yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
```

### Deploy

```bash
./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
```

Note the OC IP address from `--status` output.

### Set up OC CLI remote

```bash
operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
operations-center remote list
```

### Verify browser access

Navigate to `https://<OC_IP>:8443` in Firefox (with client.pfx imported).
The OC web UI should load and show the dashboard.

### Wait for updates to download

OC automatically downloads IncusOS update packages from upstream. At least
one update must be in `ready` state before provisioned ISOs can be generated:

```bash
operations-center provisioning update list
# Wait until at least one update shows "ready" status
# This may take several minutes on first boot
```

Also via REST API:

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/provisioning/updates
```

---

## Part 2: Provision Incus Node ISOs

### Create a provisioning token

```bash
# CLI
operations-center provisioning token add \
    --description "Lab cluster nodes" \
    --uses 5

# Note the token UUID from output
operations-center provisioning token list
operations-center provisioning token show <UUID>
```

Via REST API:

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
     -d '{"description": "Lab cluster nodes (API)", "uses": 5}'
```

### Create a token seed for Proxmox VMs (tested)

OC supports **token seeds** -- named, reusable pre-seed configs attached to a
token. The YAML format uses section-level keys (`install:`, `applications:`,
etc.) -- NOT the flat format from standard seed archives.

```yaml
# /tmp/oc-preseed.yaml -- structured format required for token seeds
install:
  version: "1"
  force_install: true
  force_reboot: true
```

**Important**: a flat format (`version: "1"` at root) maps all fields to empty
`{}` -- they don't get assigned to any section. The structured format with
`install:` as the top-level key is required.

```bash
operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml \
    --description "Force reboot for Proxmox VMs"

operations-center provisioning token seed list <UUID>
operations-center provisioning token seed show <UUID> proxmox-preseed
```

### Download the OC-provisioned ISO (tested)

Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO):

```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64

ls -lh /tmp/IncusOS-oc.iso  # ~3.4 GB
```

Alternative without token seeds (inline pre-seed):

```bash
operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
    /tmp/oc-preseed.yaml --type iso --architecture x86_64
```

---

## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)

**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an
OC-provisioned ISO. This combines the best of both worlds:
- **OC**: auto-registration token embedded in the boot ISO
- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot),
  install monitoring, media cleanup, IP detection, remote setup

The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides
force_reboot, hostname, and app selection. IncusOS reads both sources and
merges them -- they coexist without conflict.

### Deployment config

```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 400   # Use high VMIDs to avoid collisions with VMs outside pool

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false   # OC handles storage/network via Terraform

  - name: oc-node-02
    app: incus
    apply_defaults: false

  - name: oc-node-03
    app: incus
    apply_defaults: false
```

**VMID note**: if your Proxmox API token is scoped to a resource pool, it
cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges
to avoid VMID collisions with invisible VMs.

### Deploy (tested)

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml

./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```

`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA
generation (force_reboot + hostname per node), install monitoring via
blockstat polling, media removal, boot order switch, IP detection, and
incus remote setup.

### Verify OC auto-registration (tested)

After nodes boot from disk, they auto-register with OC within ~30 seconds:

```bash
operations-center provisioning server list
```

Nodes register with their hostname (set via SEED_DATA), so no renaming needed.
Each node appears with type `incus`, status `ready`, and its connection URL.

**Tested result**: all 3 nodes auto-registered with correct hostnames,
correct IPs, type `incus`, status `ready`. No manual intervention needed.

---

## Part 4: Form Cluster via Operations Center

### Prepare application config

Create an Incus preseed that injects the client certificate into the cluster
(so the `incus` CLI can connect directly):

```yaml
# /tmp/oc-app-config.yaml
certificates:
  - type: client
    name: lab-client-cert
    certificate: |-
      -----BEGIN CERTIFICATE-----
      <contents of ~/.config/incus/client.crt>
      -----END CERTIFICATE-----
```

### Form cluster via OC CLI (tested)

```bash
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on the first node
3. Joins remaining nodes (handles storage pool + network creation)
4. Applies the application seed (injects client cert)
5. Runs Terraform/OpenTofu for post-cluster configuration

### apply_defaults: false (tested, recommended)

**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
and recommended approach. With `apply_defaults: false`:

- Nodes still listen on port 8443 and trust preseed certificates
- The underlying ZFS dataset (`local/incus`) still exists
- But no Incus storage pool, network, or profile devices are created
- OC's Terraform handles all resource creation cleanly during cluster formation

**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
and injects the client certificate via `--application-seed-config`. The only
Terraform error is "Certificate already in trust store" when the cert was also
injected via SEED_DATA -- avoid this by only injecting the cert through the
application seed config, not the SEED_DATA.

### apply_defaults: true (also works, with caveats)

When using `apply_defaults: true`, each node already has a `local` ZFS storage
pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
then fails with:

- "Certificate already in trust store"
- "Network is not in pending state"
- "Storage pool is not in pending state"

**The cluster itself forms successfully** despite these Terraform errors.
However, `cluster artifact list` will be empty since Terraform didn't complete.

### Verify cluster (tested)

```bash
# Via OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Via Incus (direct cluster access)
# NOTE: clustering regenerates TLS certificates, so re-add remotes:
incus remote remove oc-node-01  # if existing remote has wrong cert
incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
incus cluster list oc-node-01:
```

### Cluster artifacts

OC generates Terraform/OpenTofu artifacts on successful cluster formation:

```bash
operations-center provisioning cluster artifact list oc-cluster
# Empty if Terraform failed (see apply_defaults conflict above)
```

---

## Part 5: OC Management Capabilities (Tested)

### Inventory browsing

OC inventory is **not real-time** -- it requires an explicit `cluster resync`
to pick up changes made via the `incus` CLI:

```bash
# Force sync first
operations-center provisioning cluster resync oc-cluster

# Then query (use UUIDs for instance show, not names)
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
operations-center inventory profile list
operations-center inventory project list
operations-center inventory image list
```

**Tested**: after creating workloads via `incus launch`, OC inventory was
empty until `cluster resync` was run. After resync, all instances, storage
pools, networks, profiles, projects, and images appeared correctly.

### Inventory sync latency

- Creating workloads via `incus` CLI: not visible in OC until resync
- After resync: ~5 seconds for inventory to populate
- Live migration: location change not reflected until resync
- Evacuation: instance relocation not reflected until resync
- OC `inventory instance show <UUID>` shows: name, project, cluster, server, timestamp

### Server management (tested)

```bash
operations-center provisioning server list      # Lists all servers + OC itself
operations-center provisioning server show <name>
operations-center provisioning server resync <name>
operations-center provisioning server system reboot <name>    # WARNING: see below
operations-center provisioning server system poweroff <name>
```

**WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system
reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS
nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot,
all recover in ~50s with containers and data intact). The failure is
**OC-specific**: the OC agent runs on every boot and can fail with:
- "failed to parse update frequency - invalid crontab expression" (OC
  pushed invalid update config)
- "error from operations center: forbidden: failed creating server:
  constraint violation" (OC trying to re-register an existing server)
- Incus stuck at "starting application name=incus" (cluster peers are in
  boot loops, so quorum cannot form)

These failures cause boot loops or hung services. Even Proxmox stop/start
does not recover because the OC state on disk remains broken. The only
fix is to destroy and redeploy the VM.

**Safe reboot**: Proxmox stop/start and guest reboot both work on
standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on
OC-managed nodes.
The proper lifecycle for maintenance is:
1. `incus cluster evacuate <remote>:<member> --force`
2. Proxmox stop the VM
3. Proxmox start the VM
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore <remote>:<member> --force`

### IncusOS scrub_schedule fix via API (tested)

IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
with an empty `scrub_schedule` field. The `registerJobs()` function calls
`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
is briefly reachable during each crash cycle (Incus starts before the scheduler).

**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):

```bash
# Check current storage config
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<NODE_IP>:8443/os/1.0/system/storage

# If scrub_schedule is empty, fix it
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     -X PUT -H "Content-Type: application/json" \
     -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
     https://<NODE_IP>:8443/os/1.0/system/storage
```

**Key details**:
- The `/os/` prefix is required — this proxies to the IncusOS daemon API
- PATCH returns 501 (not implemented); use PUT instead
- `incus admin os system storage edit` fails because it reads the current
  (corrupt) config and validates it before allowing edits
- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
  fixes this on every deployed node (safe — returns early if already set)
- Filed upstream as IncusOS issue #843

### Node failure and recovery (tested)

Proxmox hard-stop simulates node crash. Incus cluster detects failure via
heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):

```bash
# Simulate crash: Proxmox API stop
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...

# After ~40s: incus cluster list shows OFFLINE with heartbeat info

# Restart: Proxmox API start
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...

# After ~60s boot: node auto-rejoins cluster (ONLINE)
# Containers on the node auto-start after recovery
```

**OC does not detect node failures**: `provisioning server show` still shows
`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.

**OC does not track Incus cluster member state**: the server list always
shows `ready` status even when the Incus cluster reports `EVACUATED`.

### Server edit commands

Network and storage have `edit` subcommands (no `show`):

```bash
operations-center provisioning server system network edit <name>
operations-center provisioning server system storage edit <name>
```

### Update management (tested)

```bash
operations-center provisioning update list       # Shows all available IncusOS versions
operations-center provisioning update show <UUID> # Full file list with sizes
operations-center provisioning update refresh     # Check for new updates
```

Updates show comprehensive file lists including sysexts: incus, operations-center,
gpu-support, incus-ceph, incus-linstor, migration-manager, debug.

### System configuration (tested)

```bash
operations-center system settings show   # log_level
operations-center system security show   # OIDC, OpenFGA, ACME, trusted certs
operations-center system updates show    # Update source, filter expressions, signature CA
```

Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`.

### Admin / IncusOS management (tested)

```bash
operations-center admin os show           # hostname, os_version, uptime
operations-center admin os application list
operations-center admin os application show <name>
operations-center admin os service list   # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
operations-center admin os debug log
```

### Cluster operations (tested)

```bash
operations-center provisioning cluster resync <name>
operations-center provisioning cluster rename <name> <new-name>
operations-center provisioning cluster update --connection-url <url>
operations-center provisioning cluster update-certificate <name> <cert> <key>
operations-center provisioning cluster artifact list <name>
operations-center provisioning cluster factory-reset <name> <token-uuid>
```

### Cluster templates

```bash
operations-center provisioning cluster-template add <name> --description "..."
operations-center provisioning cluster-template list
operations-center provisioning cluster-template show <name>
operations-center provisioning cluster-template remove <name>
```

---

## Part 6: Cluster Lifecycle (Tested)

### Live migration visibility through OC (tested)

```bash
# Prepare VM for live migration
incus stop oc-node-01:test-vm-01
incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
incus start oc-node-01:test-vm-01

# Migrate
incus move oc-node-01:test-vm-01 --target oc-node-03

# OC does NOT auto-update -- resync required
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # now shows new location
```

**Tested**: ~141 MB/s migration speed. OC inventory reflected the location
change only after explicit `cluster resync`.

### Cluster evacuation through OC (tested)

```bash
incus cluster evacuate oc-node-01:oc-node-01 --force
# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list  # shows workloads moved to other nodes

incus cluster restore oc-node-01:oc-node-01 --force
```

### Node replacement lifecycle (tested)

Full procedure tested after OC reboot broke a node:

```bash
# 1. Force-remove dead node from Incus cluster
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force

# 2. Destroy broken VM via Proxmox (stop + delete)

# 3. Redeploy via incusos-proxmox --iso (single-node config)
#    Use apply_defaults: false for OC-managed nodes
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml

# 4. New node auto-registers with OC (same hostname, new IP)
#    NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
#    while OC thinks it's in a cluster. Resync doesn't fix stale entries.

# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# With apply_defaults: false, no cleanup needed (no pool/network to delete)
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)

# 6. Fix remote after cert change
incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
```

### Node failure simulation (tested)

Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):

```bash
# Simulate crash
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# incus cluster list shows OFFLINE after ~40s

# Restart
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# Node auto-rejoins after ~60s boot. Containers auto-start.
# OC still shows "ready" (does not detect node failures)
```

### Factory reset (not tested)

```bash
# WARNING: destructive
operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
```

---

## OC CLI Command Reference

### Remote management

| Command | Description |
|---------|-------------|
| `operations-center remote add <name> <url> --auth-type tls` | Add OC remote |
| `operations-center remote switch <name>` | Set default remote |
| `operations-center remote list` | List configured remotes |
| `operations-center remote remove <name>` | Remove a remote |

### Provisioning tokens

| Command | Description |
|---------|-------------|
| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token |
| `provisioning token list` | List tokens |
| `provisioning token show <UUID>` | Token details |
| `provisioning token remove <UUID>` | Delete token |
| `provisioning token seed add <UUID> <name> <file> --description "..."` | Attach named seed |
| `provisioning token seed list <UUID>` | List token seeds |
| `provisioning token seed show <UUID> <name>` | Show seed details |
| `provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64` | ISO from seed |
| `provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64` | ISO from inline preseed |

### Provisioning servers

| Command | Description |
|---------|-------------|
| `provisioning server list` | List registered servers (includes OC itself) |
| `provisioning server show <name>` | Server details + certificate |
| `provisioning server rename <old> <new>` | Rename server |
| `provisioning server resync <name>` | Resync single server |
| `provisioning server remove <name>` | Remove (fails if in cluster) |
| `provisioning server edit <name>` | Edit server (interactive) |
| `provisioning server system reboot <name>` | Guest reboot (DANGEROUS on Proxmox) |
| `provisioning server system poweroff <name>` | Guest poweroff |
| `provisioning server system network edit <name>` | Edit network config |
| `provisioning server system storage edit <name>` | Edit storage config |

### Provisioning clusters

| Command | Description |
|---------|-------------|
| `provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>` | Form cluster |
| `provisioning cluster list` | List clusters |
| `provisioning cluster show <name>` | Cluster details + certificate |
| `provisioning cluster resync <name>` | Force inventory resync |
| `provisioning cluster rename <name> <new-name>` | Rename cluster |
| `provisioning cluster update <name> --connection-url <url>` | Update cluster URL |
| `provisioning cluster update-certificate <name> <cert> <key>` | Update cluster cert |
| `provisioning cluster remove <name>` | Delete cluster |
| `provisioning cluster factory-reset <name> <token-uuid>` | Factory reset |
| `provisioning cluster artifact list <name>` | List artifacts |
| `provisioning cluster artifact archive <name> <type> <output>` | Download artifact |

### Cluster templates

| Command | Description |
|---------|-------------|
| `provisioning cluster-template add <name> --description "..."` | Create template |
| `provisioning cluster-template list` | List templates |
| `provisioning cluster-template show <name>` | Show template |
| `provisioning cluster-template remove <name>` | Delete template |

### Inventory

| Command | Description |
|---------|-------------|
| `inventory instance list` | List all instances |
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
| `inventory storage-pool list` | List storage pools |
| `inventory storage-volume list` | List storage volumes |
| `inventory storage-bucket list` | List storage buckets |
| `inventory network list` | List networks |
| `inventory network-acl list` | List network ACLs |
| `inventory network-address-set list` | List network address sets |
| `inventory network-forward list` | List network forwards |
| `inventory network-integration list` | List network integrations |
| `inventory network-load-balancer list` | List network load balancers |
| `inventory network-peer list` | List network peers |
| `inventory network-zone list` | List network zones |
| `inventory profile list` | List profiles |
| `inventory project list` | List projects |
| `inventory image list` | List images |
| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |

### Updates

| Command | Description |
|---------|-------------|
| `provisioning update list` | List available IncusOS versions |
| `provisioning update show <UUID>` | Version details + file list |
| `provisioning update refresh` | Check for new updates |

### System

| Command | Description |
|---------|-------------|
| `system settings show` | System settings (log_level) |
| `system security show` | OIDC, OpenFGA, ACME, trusted certs |
| `system updates show` | Update source, filter, signature CA |
| `system network show` | Network config |
| `system certificate set <cert> <key>` | Set server certificate |

### Admin

| Command | Description |
|---------|-------------|
| `admin os show` | IncusOS hostname, version, uptime |
| `admin os application list` | List applications |
| `admin os application show <name>` | Application state |
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
| `admin os service show <name>` | Service config |
| `admin os debug log` | Debug logs |
| `admin os debug processes` | System process list |
| `admin os debug secureboot` | Secure Boot debug info |

All commands prefixed with `operations-center`.

---

## REST API Reference

OC listens on port **8443** (same as Incus on IncusOS). All endpoints require
TLS client certificate authentication.

### Common curl pattern

```bash
curl -sk --cert ~/.config/incus/client.crt \
     --key ~/.config/incus/client.key \
     https://<OC_IP>:8443/1.0/<endpoint>
```

### Key endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/1.0` | Server info |
| GET | `/1.0/provisioning/updates` | List update packages |
| GET | `/1.0/provisioning/tokens` | List tokens |
| POST | `/1.0/provisioning/tokens` | Create token |
| GET | `/1.0/provisioning/servers` | List registered servers |
| GET | `/1.0/provisioning/clusters` | List clusters |
| POST | `/1.0/provisioning/clusters` | Form cluster |
| GET | `/1.0/inventory/instances` | List all instances |
| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |

### IncusOS REST API (via Incus proxy)

IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
prefix on port 8443:

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
| PUT | `/os/1.0/system/storage` | Update storage config |
| GET | `/os/1.0` | IncusOS system info |

---

## Proxmox-Specific Considerations

### force_reboot is required

IncusOS does not auto-halt after installation. Without `force_reboot: true`
in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at
"please remove installation media" indefinitely.

Options for OC-provisioned ISOs on Proxmox:
1. **Pre-seed file** passed to `get-image`: include `force_reboot: true`
2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with
   `force_reboot: true` only

### VM creation settings

OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS
deployments (see `incusos-proxmox`):

- `bios=ovmf`, `machine=q35` -- UEFI boot required
- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
- `tpmstate0`: `version=v2.0` -- required for disk encryption
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally
- Minimum 50 GiB disk, minimum 4096 MiB RAM

### Install media removal

After installation completes, you **must** stop the VM and delete ide2 (and
ide3 if used) before booting from disk. IncusOS refuses to start if install
media is still attached.

### IP detection

IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to
find node IPs:

```bash
# Get MAC from Proxmox VM config
MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
    -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print(d.get('net0', '').split('=')[1].split(',')[0])")

# ARP lookup
ip neigh flush dev <bridge>
ping -b -c 3 <broadcast_ip> 2>/dev/null
ip neigh show | grep -i "$MAC" | awk '{print $1}'
```

---

## Known Limitations (Tested Findings)

### Confirmed limitations (v0.3.0)

- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
- **No cluster member state tracking**: OC always shows `ready`, even for
  EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
  create stale entries that `resync` doesn't fix
- **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause
  boot loops or hung services (not a dqlite or TPM issue -- standalone nodes
  reboot fine)
- **Token seed format quirk**: requires structured YAML with section keys,
  not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry
- **No cluster resize**: cannot add/remove members from existing clusters

### Design constraints

- OC runs on IncusOS (same immutable OS as Incus nodes)
- Uses port 8443 for both API and web UI
- Requires at least one trusted client certificate in seed
- Provisioning tokens have configurable use count and lifetime
- OC adds `meshbr0` network to clusters (not present in manual clustering)
- REST API endpoint: `/1.0` (same pattern as Incus)
- Auth methods: TLS client cert or OIDC

---

## Troubleshooting (Tested)

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` |
| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. |
| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates |
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions |
| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt |
| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion |
| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster |
| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins |

---

## Comparison: Manual vs OC Cluster Workflow (Tested)

### Manual (incusos-proxmox + incus CLI)

1. Write YAML config with VM specs
2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA
3. Manually set `core.https_address` on each node
4. Delete storage pool/network on joining nodes
5. `incus cluster enable` + `incus cluster join` per node
6. Re-add remotes after certificate changes
7. ~30 minutes for 3-node cluster

### Hybrid (incusos-proxmox --iso + OC)

1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml`
2. Create provisioning token + token seed in OC
3. Download OC-provisioned ISO via `token seed get-image`
4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml`
5. Nodes auto-register with OC (~30s each)
6. `operations-center provisioning cluster add oc-cluster ...`
7. ~20 minutes for 3-node cluster (including OC deploy)

### What OC automates that we do manually

| Step | Manual | OC |
|------|--------|-----|
| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add |
| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) |
| Cluster enable + join | Multi-step per node | Single `cluster add` command |
| Client cert injection | Via SEED_DATA | Via `--application-seed-config` |
| `meshbr0` network | Not created | Auto-created by OC clustering |
| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) |

**Trade-offs**:
- Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized
  inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing
  loops -- standalone reboot is safe)
- OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state

---

## Open Research Items

- **OC reboot failure mechanism** (resolved): guest reboot is safe on
  standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
  in ~50s). The failure is OC-specific: the OC agent pushes config via the
  IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
  values (e.g., cron expression where Go duration is expected) crash the
  daemon. Root cause is NOT dqlite corruption or TPM issues.

- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
  is now the tested and recommended approach. OC's Terraform handles storage pool,
  network, and cert creation cleanly when nodes start without defaults.
  `apply_defaults: true` also works but produces non-fatal Terraform errors.

- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
  ~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
  likely a race condition between REST API startup and `registerJobs()`, or a
  zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
  with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
  auto-heals this proactively on every deploy.

- **Stale server entries after out-of-band changes**: when cluster members are
  removed via `incus cluster remove` (not through OC), stale entries persist in
  OC's server list. `cluster resync` doesn't fix them. The entry can't be
  removed while OC considers it part of a cluster.

- **Inventory sync is manual**: OC inventory requires explicit `cluster resync`
  to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
  This applies to instance creation, migration, evacuation, and deletion.

- **Node failure detection**: OC does not detect node failures. `provisioning
  server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
  Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.

---

## References

- [Operations Center GitHub](https://github.com/FuturFusion/operations-center)
- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
- [IncusOS GitHub](https://github.com/lxc/incus-os)
- [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
- [Clustering guide](clustering-guide.md) (manual cluster formation reference)