diff --git a/.gitignore b/.gitignore index 0160dc5..250da7a 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,6 @@ flasher-tool # Proxmox connection config (contains credentials) proxmox.yaml + +# Environment secrets +env diff --git a/CLAUDE.md b/CLAUDE.md index 12d36b8..9860136 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,8 @@ incu-contrib/ │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material - └── clustering-guide.md # Detailed Incus clustering walkthrough + ├── clustering-guide.md # Detailed Incus clustering walkthrough + └── operations-center-guide.md # Operations Center provisioning & management ``` ## Key technical context @@ -356,12 +357,58 @@ incu-contrib/ built from source at `github.com/FuturFusion/operations-center`). - **Config directory:** `~/.config/operations-center/` (uses same cert format as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). -- **Provisioning workflow:** create token → get provisioned ISO → deploy nodes - with that ISO → nodes auto-register with OC → form cluster via OC. -- **OC is under active development** (v0.2.2+). Commands and APIs may change. - The `--doctor` command on `incusos-proxmox` reports whether the CLI is +- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI. +- **Browser access**: requires PKCS#12 client certificate (`client.pfx`) + imported into the browser. +- **OC is under active development** (v0.2.2). Commands and APIs may change. +- The `--doctor` command on `incusos-proxmox` reports whether the CLI is installed. +#### Provisioning workflow (tested: token → seed → ISO → deploy → register → cluster) + +- **No brownfield adoption**: nodes must boot from an OC-provisioned ISO. +- **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML + must use structured format with section keys (`install:`, not flat). + ```bash + operations-center provisioning token seed add proxmox-preseed \ + /tmp/preseed.yaml --description "Force reboot for Proxmox" + operations-center provisioning token seed get-image proxmox-preseed \ + /tmp/IncusOS-oc.iso --type iso --architecture x86_64 + ``` +- **Hybrid deployment** (tested, recommended): + `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml` + combines OC auto-registration (from boot ISO token) with `incusos-proxmox` + VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring, + and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist. +- **Self-registration**: nodes auto-register with OC within ~30s of first + boot. Hostname from SEED_DATA is used as the server name. +- **Cluster formation** (tested): + ```bash + operations-center provisioning cluster add oc-cluster \ + https://:8443 \ + --server-names oc-node-01,oc-node-02,oc-node-03 \ + --server-type incus \ + --application-seed-config /tmp/oc-app-config.yaml + ``` + OC handles: `core.https_address` → cluster enable → joins → seed + application → Terraform config. Adds `meshbr0` network. +- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`, + nodes already have storage pool/network/cert. OC's Terraform fails with + "already exists" errors but the cluster forms successfully. + +#### Tested limitations + +- **Inventory is NOT real-time** -- requires explicit `cluster resync` +- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot corrupts TPM + encryption. Node becomes unreachable. Only fix: destroy and redeploy. + Use Proxmox stop/start instead. +- **No cluster member state tracking** -- OC always shows `ready` even for + EVACUATED/OFFLINE nodes +- **Stale entries** from out-of-band cluster changes persist after resync +- **Server removal blocked** if server is part of an OC cluster + +- See `notes/operations-center-guide.md` for full tested OC reference. + ### incusos-proxmox doctor and cleanup - **`--doctor`**: standalone environment check. No config file required. diff --git a/incusos/TESTING.md b/incusos/TESTING.md index 864a493..ade2697 100644 --- a/incusos/TESTING.md +++ b/incusos/TESTING.md @@ -644,58 +644,196 @@ and detailed test results. --- -## Operations Center Lab +## Operations Center Lab (Tested) -### Deploy +### Overview + +Operations Center (OC) manages IncusOS nodes through a provisioning pipeline: +tokens → provisioned ISOs → self-registration → cluster formation. + +**Hybrid approach** (tested, recommended): use `incusos-proxmox --iso` to +deploy nodes from an OC-provisioned ISO. Combines OC auto-registration with +`incusos-proxmox` deployment automation (VM creation, SEED_DATA, install +monitoring, media cleanup, IP detection). + +### Prerequisites ```bash -./incusos/incusos-proxmox --yes examples/lab-oc.yaml -./incusos/incusos-proxmox --status examples/lab-oc.yaml -``` +# 1. Install OC CLI +wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64 +chmod +x operations-center_linux_x86_64 +sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center -### Access the Web UI - -```bash -# Export client certificate for browser -openssl pkcs12 -export \ - -inkey ~/.config/incus/client.key \ - -in ~/.config/incus/client.crt \ - -out client.pfx - -# Import client.pfx into Firefox, navigate to https://:8443 -``` - -### OC CLI Setup - -```bash -# Copy incus client certs for OC CLI +# 2. Set up OC client certs mkdir -p ~/.config/operations-center cp ~/.config/incus/client.crt ~/.config/operations-center/ cp ~/.config/incus/client.key ~/.config/operations-center/ -# Add OC remote -operations-center remote add lab-oc https://:8443 -operations-center remote switch lab-oc +# 3. Browser cert (for web UI) +openssl pkcs12 -export \ + -inkey ~/.config/incus/client.key \ + -in ~/.config/incus/client.crt \ + -out ~/.config/incus/client.pfx -# Check for provisioning updates -operations-center provisioning update list +# 4. Verify +./incusos/incusos-proxmox --doctor ``` -### Provisioning Workflow +### Step 1: Deploy OC Server ```bash -# Create a provisioning token -operations-center provisioning token add --description "lab nodes" --uses 10 - -# Get provisioned ISO -operations-center provisioning token get-image ~/Downloads/IncusOS-provisioned.iso - -# Redeploy nodes with provisioned ISO, or check if manual adoption works +./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml +./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml +# Note OC_IP from output ``` -**Note:** Operations Center is under active development (v0.2.2+). Commands and -APIs may change between versions. The `--doctor` command reports whether the OC -CLI is installed. +### Step 2: Connect OC CLI + +```bash +printf "yes\n" | operations-center remote add oc-lab https://:8443 --auth-type tls +operations-center remote switch oc-lab +``` + +### Step 3: Wait for Updates (~3 min) + +```bash +operations-center provisioning update list +# Wait until at least one shows "ready" +``` + +### Step 4: Create Token + Seed + +```bash +# Create token +operations-center provisioning token add --description "Lab cluster" --uses 5 --lifetime 168h +operations-center provisioning token list # Note UUID + +# Create structured pre-seed (MUST use section-level keys) +cat > /tmp/oc-preseed.yaml << 'EOF' +install: + version: "1" + force_install: true + force_reboot: true +EOF + +# Attach as named seed +operations-center provisioning token seed add proxmox-preseed \ + /tmp/oc-preseed.yaml --description "Force reboot for Proxmox VMs" +``` + +### Step 5: Download OC-Provisioned ISO + +```bash +operations-center provisioning token seed get-image proxmox-preseed \ + /tmp/IncusOS-oc.iso --type iso --architecture x86_64 +ls -lh /tmp/IncusOS-oc.iso # ~3.4 GB +``` + +### Step 6: Deploy Nodes (Hybrid) + +```bash +./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \ + incusos/examples/lab-oc-nodes.yaml +./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml +``` + +`incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node +SEED_DATA on ide3 (hostname + force_reboot + app selection + cert), monitors +install, removes media, boots from disk, and sets up incus remotes. + +### Step 7: Verify Auto-Registration + +```bash +# Nodes register within ~30s of first boot +operations-center provisioning server list +# Should show 3 nodes + OC itself, all "ready" +``` + +### Step 8: Form Cluster + +```bash +# Prepare application seed (client cert for incus CLI access) +cat > /tmp/oc-app-config.yaml << EOF +certificates: + - type: client + name: lab-client + certificate: |- +$(sed 's/^/ /' ~/.config/incus/client.crt) +EOF + +# Form cluster +operations-center provisioning cluster add oc-cluster \ + https://:8443 \ + --server-names oc-node-01,oc-node-02,oc-node-03 \ + --server-type incus \ + --application-seed-config /tmp/oc-app-config.yaml +``` + +**Note**: with `apply_defaults: true` in SEED_DATA, Terraform post-config +errors are expected ("already in trust store", "not in pending state"). The +cluster forms successfully despite these. + +### Step 9: Verify + +```bash +# OC +operations-center provisioning cluster list +operations-center provisioning cluster show oc-cluster + +# Incus (re-add remotes after cert change from clustering) +incus remote remove oc-node-01 && incus remote add oc-node-01 https://:8443 --accept-certificate +incus cluster list oc-node-01: + +# Inventory (requires resync) +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list +operations-center inventory storage-pool list +operations-center inventory network list +``` + +### Step 10: Test Features + +```bash +# Workload sync: create via incus, resync, verify in OC +incus launch images:debian/12 oc-node-01:test-ct --target oc-node-01 +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list + +# Live migration visibility +incus move oc-node-01:test-vm --target oc-node-03 +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list # shows new location + +# Evacuation +incus cluster evacuate oc-node-01:oc-node-01 --force +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list # shows relocated workloads +incus cluster restore oc-node-01:oc-node-01 --force +``` + +**WARNING**: do NOT use `operations-center provisioning server system reboot`. +It sends a guest-level reboot that corrupts TPM disk encryption on Proxmox VMs. +The node becomes unreachable and must be destroyed/redeployed. + +### Cleanup + +```bash +# Test workloads +incus delete oc-node-01:test-ct --force 2>/dev/null +incus delete oc-node-01:test-vm --force 2>/dev/null + +# Nodes +./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml + +# OC server +./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml + +# Remotes +operations-center remote remove oc-lab 2>/dev/null +``` + +**Note:** Operations Center is under active development (v0.2.2+). See +`notes/operations-center-guide.md` for the full tested reference. --- @@ -722,3 +860,15 @@ CLI is installed. | `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start | | `certificate is valid for 127.0.0.1, ::1, not ` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` | | `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide | +| OC `get-image` fails | No updates in `ready` state yet | Wait ~3 min for OC to download packages; check `provisioning update list` | +| Node doesn't appear in OC server list | ISO not OC-provisioned or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses | +| OC web UI returns 403 | Client cert not imported in browser | Import `client.pfx` into browser certificates | +| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable | +| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root | +| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` | +| OC reboot kills IncusOS node (port 8443 closed forever) | Guest reboot corrupts TPM PCR measurements | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot | +| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync ` | +| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist | +| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management | +| `cluster join` prompts for `tunnel.mesh.interface` | OC adds `meshbr0` network | Add extra `\n` for meshbr0 prompt: `printf '\n\nyes\n\nlocal/incus\nlocal/incus\n'` | +| VMID collision (500/403 on VM create) | Pool-scoped API token can't see VMs outside pool | Use `start_vmid: 400` or higher in config to avoid collisions | diff --git a/incusos/examples/lab-oc-deploy.yaml b/incusos/examples/lab-oc-deploy.yaml new file mode 100644 index 0000000..3e25288 --- /dev/null +++ b/incusos/examples/lab-oc-deploy.yaml @@ -0,0 +1,39 @@ +# lab-oc-deploy.yaml - Operations Center server only +# +# Deploys ONLY the OC server node. Incus nodes are provisioned through OC's +# own provisioning pipeline (token -> ISO -> self-registration), not by +# incusos-proxmox. +# +# Workflow: +# 1. Deploy OC server with incusos-proxmox +# 2. Set up OC CLI remote + browser access (import client.pfx) +# 3. Wait for OC to download IncusOS update packages +# 4. Create provisioning token in OC +# 5. Download OC-provisioned IncusOS ISO +# 6. Create Proxmox VMs manually with OC-provisioned ISO +# 7. Nodes self-register with OC after first boot +# 8. Form cluster via OC CLI, API, or web UI +# +# Usage: +# incusos-proxmox --dry-run examples/lab-oc-deploy.yaml +# incusos-proxmox --yes examples/lab-oc-deploy.yaml +# incusos-proxmox --status examples/lab-oc-deploy.yaml +# incusos-proxmox --cleanup --deep examples/lab-oc-deploy.yaml +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. +# +# Note: Uses VMID 920 to avoid collisions with lab-cluster (900+) and +# lab-advanced/lab-oc (910+). Incus nodes deployed via OC provisioning +# use VMIDs 921-923 (created manually via Proxmox API). + +defaults: + cores: 2 + memory: 4096 + disk: 50 + start_vmid: 920 + +vms: + - name: oc-server + app: operations-center + apply_defaults: true diff --git a/incusos/examples/lab-oc-nodes.yaml b/incusos/examples/lab-oc-nodes.yaml new file mode 100644 index 0000000..08b079d --- /dev/null +++ b/incusos/examples/lab-oc-nodes.yaml @@ -0,0 +1,34 @@ +# lab-oc-nodes.yaml - Incus nodes for OC-managed cluster +# +# Deploys 3 Incus nodes using an OC-provisioned ISO. Nodes auto-register +# with Operations Center after installation using the token embedded in +# the ISO. +# +# Usage: +# # First: download OC-provisioned ISO via OC CLI +# # Then deploy with --iso pointing to it: +# incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run examples/lab-oc-nodes.yaml +# incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-oc-nodes.yaml +# incusos-proxmox --status examples/lab-oc-nodes.yaml +# incusos-proxmox --cleanup --deep examples/lab-oc-nodes.yaml +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). + +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 400 + +vms: + - name: oc-node-01 + app: incus + apply_defaults: true + + - name: oc-node-02 + app: incus + apply_defaults: true + + - name: oc-node-03 + app: incus + apply_defaults: true diff --git a/notes/operations-center-guide.md b/notes/operations-center-guide.md new file mode 100644 index 0000000..b38de77 --- /dev/null +++ b/notes/operations-center-guide.md @@ -0,0 +1,862 @@ +# Operations Center Guide for IncusOS + +This guide covers deploying and using Operations Center (OC) by FuturFusion -- +a centralized management layer for IncusOS/Incus deployments. OC handles +provisioning, clustering, inventory, and updates through a REST API, CLI, and +web UI. + +All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553 +on Proxmox VE 9.1.5 (nested virtualization on Intel). + +--- + +## Overview + +Operations Center is fundamentally different from managing Incus clusters +manually via `incus` CLI remotes. Key differences: + +| | Manual (incus CLI) | Operations Center | +|---|---|---| +| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) | +| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI | +| Node registration | Manual remote add | Automatic self-registration on first boot | +| Inventory | `incus list` per remote | Centralized view across all clusters | +| Updates | Manual per-node | OC manages update channels and rollouts | +| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned | + +**Key constraint**: OC only manages nodes it has provisioned. There is no +"adopt existing node" workflow. Nodes must boot from an OC-generated ISO +containing a provisioning token, then self-register on first boot. + +--- + +## Prerequisites + +### Install OC CLI + +```bash +# Check for latest release at https://github.com/FuturFusion/operations-center/releases +wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64 +chmod +x operations-center_linux_x86_64 +sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center +operations-center version +``` + +### Set up OC client certificates + +OC uses the same TLS client certificate format as Incus: + +```bash +mkdir -p ~/.config/operations-center +cp ~/.config/incus/client.crt ~/.config/operations-center/ +cp ~/.config/incus/client.key ~/.config/operations-center/ +``` + +### Generate PKCS#12 cert for browser access + +The OC web UI requires a client certificate for authentication: + +```bash +openssl pkcs12 -export \ + -inkey ~/.config/incus/client.key \ + -in ~/.config/incus/client.crt \ + -out ~/.config/incus/client.pfx +# Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import +``` + +### Verify with doctor check + +```bash +./incusos/incusos-proxmox --doctor +# Should report OC CLI as installed +``` + +--- + +## Part 1: Deploy Operations Center Server + +### Deployment config + +Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server +(Incus nodes will be provisioned by OC itself): + +```yaml +defaults: + cores: 2 + memory: 4096 + disk: 50 + start_vmid: 920 + +vms: + - name: oc-server + app: operations-center + apply_defaults: true +``` + +### Deploy + +```bash +./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml +./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml +./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml +``` + +Note the OC IP address from `--status` output. + +### Set up OC CLI remote + +```bash +operations-center remote add oc-lab https://:8443 --auth-type tls +operations-center remote switch oc-lab +operations-center remote list +``` + +### Verify browser access + +Navigate to `https://:8443` in Firefox (with client.pfx imported). +The OC web UI should load and show the dashboard. + +### Wait for updates to download + +OC automatically downloads IncusOS update packages from upstream. At least +one update must be in `ready` state before provisioned ISOs can be generated: + +```bash +operations-center provisioning update list +# Wait until at least one update shows "ready" status +# This may take several minutes on first boot +``` + +Also via REST API: + +```bash +curl -sk --cert ~/.config/incus/client.crt \ + --key ~/.config/incus/client.key \ + https://:8443/1.0/provisioning/updates +``` + +--- + +## Part 2: Provision Incus Node ISOs + +### Create a provisioning token + +```bash +# CLI +operations-center provisioning token add \ + --description "Lab cluster nodes" \ + --uses 5 + +# Note the token UUID from output +operations-center provisioning token list +operations-center provisioning token show +``` + +Via REST API: + +```bash +curl -sk --cert ~/.config/incus/client.crt \ + --key ~/.config/incus/client.key \ + -X POST https://:8443/1.0/provisioning/tokens \ + -d '{"description": "Lab cluster nodes (API)", "uses": 5}' +``` + +### Create a token seed for Proxmox VMs (tested) + +OC supports **token seeds** -- named, reusable pre-seed configs attached to a +token. The YAML format uses section-level keys (`install:`, `applications:`, +etc.) -- NOT the flat format from standard seed archives. + +```yaml +# /tmp/oc-preseed.yaml -- structured format required for token seeds +install: + version: "1" + force_install: true + force_reboot: true +``` + +**Important**: a flat format (`version: "1"` at root) maps all fields to empty +`{}` -- they don't get assigned to any section. The structured format with +`install:` as the top-level key is required. + +```bash +operations-center provisioning token seed add proxmox-preseed \ + /tmp/oc-preseed.yaml \ + --description "Force reboot for Proxmox VMs" + +operations-center provisioning token seed list +operations-center provisioning token seed show proxmox-preseed +``` + +### Download the OC-provisioned ISO (tested) + +Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO): + +```bash +operations-center provisioning token seed get-image proxmox-preseed \ + /tmp/IncusOS-oc.iso --type iso --architecture x86_64 + +ls -lh /tmp/IncusOS-oc.iso # ~3.4 GB +``` + +Alternative without token seeds (inline pre-seed): + +```bash +operations-center provisioning token get-image /tmp/IncusOS-oc.iso \ + /tmp/oc-preseed.yaml --type iso --architecture x86_64 +``` + +--- + +## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested) + +**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an +OC-provisioned ISO. This combines the best of both worlds: +- **OC**: auto-registration token embedded in the boot ISO +- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot), + install monitoring, media cleanup, IP detection, remote setup + +The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides +force_reboot, hostname, and app selection. IncusOS reads both sources and +merges them -- they coexist without conflict. + +### Deployment config + +```yaml +# incusos/examples/lab-oc-nodes.yaml +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 400 # Use high VMIDs to avoid collisions with VMs outside pool + +vms: + - name: oc-node-01 + app: incus + apply_defaults: true + + - name: oc-node-02 + app: incus + apply_defaults: true + + - name: oc-node-03 + app: incus + apply_defaults: true +``` + +**VMID note**: if your Proxmox API token is scoped to a resource pool, it +cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges +to avoid VMID collisions with invisible VMs. + +### Deploy (tested) + +```bash +./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \ + incusos/examples/lab-oc-nodes.yaml + +./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \ + incusos/examples/lab-oc-nodes.yaml + +./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml +``` + +`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA +generation (force_reboot + hostname per node), install monitoring via +blockstat polling, media removal, boot order switch, IP detection, and +incus remote setup. + +### Verify OC auto-registration (tested) + +After nodes boot from disk, they auto-register with OC within ~30 seconds: + +```bash +operations-center provisioning server list +``` + +Nodes register with their hostname (set via SEED_DATA), so no renaming needed. +Each node appears with type `incus`, status `ready`, and its connection URL. + +**Tested result**: all 3 nodes auto-registered with correct hostnames, +correct IPs, type `incus`, status `ready`. No manual intervention needed. + +--- + +## Part 4: Form Cluster via Operations Center + +### Prepare application config + +Create an Incus preseed that injects the client certificate into the cluster +(so the `incus` CLI can connect directly): + +```yaml +# /tmp/oc-app-config.yaml +certificates: + - type: client + name: lab-client-cert + certificate: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +### Form cluster via OC CLI (tested) + +```bash +operations-center provisioning cluster add oc-cluster \ + https://:8443 \ + --server-names oc-node-01,oc-node-02,oc-node-03 \ + --server-type incus \ + --application-seed-config /tmp/oc-app-config.yaml +``` + +OC orchestrates the full cluster formation: +1. Sets `core.https_address` to each node's specific IP +2. Enables clustering on the first node +3. Joins remaining nodes (handles storage pool + network creation) +4. Applies the application seed (injects client cert) +5. Runs Terraform/OpenTofu for post-cluster configuration + +### apply_defaults conflict (tested finding) + +When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`), +each node already has a `local` ZFS storage pool, `incusbr0` network bridge, +and the client certificate. OC's Terraform post-clustering step then tries to +create the same resources and fails with: + +- "Certificate already in trust store" +- "Network is not in pending state" +- "Storage pool is not in pending state" + +**The cluster itself forms successfully** despite these Terraform errors. All +3 nodes join the cluster correctly. The Terraform errors are non-fatal because +the resources already exist. However, the `cluster artifact list` will be empty +since Terraform didn't complete. + +**Recommended approaches** for future deployments: +1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore + Terraform errors. Skip `--application-seed-config` since the cert is already + injected by the seed. +2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and + without cert injection in the seed. Let OC handle everything via Terraform. + This is the "clean" approach but hasn't been tested yet. + +### Verify cluster (tested) + +```bash +# Via OC +operations-center provisioning cluster list +operations-center provisioning cluster show oc-cluster + +# Via Incus (direct cluster access) +# NOTE: clustering regenerates TLS certificates, so re-add remotes: +incus remote remove oc-node-01 # if existing remote has wrong cert +incus remote add oc-node-01 https://:8443 --accept-certificate +incus cluster list oc-node-01: +``` + +### Cluster artifacts + +OC generates Terraform/OpenTofu artifacts on successful cluster formation: + +```bash +operations-center provisioning cluster artifact list oc-cluster +# Empty if Terraform failed (see apply_defaults conflict above) +``` + +--- + +## Part 5: OC Management Capabilities (Tested) + +### Inventory browsing + +OC inventory is **not real-time** -- it requires an explicit `cluster resync` +to pick up changes made via the `incus` CLI: + +```bash +# Force sync first +operations-center provisioning cluster resync oc-cluster + +# Then query (use UUIDs for instance show, not names) +operations-center inventory instance list +operations-center inventory storage-pool list +operations-center inventory network list +operations-center inventory profile list +operations-center inventory project list +operations-center inventory image list +``` + +**Tested**: after creating workloads via `incus launch`, OC inventory was +empty until `cluster resync` was run. After resync, all instances, storage +pools, networks, profiles, projects, and images appeared correctly. + +### Inventory sync latency + +- Creating workloads via `incus` CLI: not visible in OC until resync +- After resync: ~5 seconds for inventory to populate +- Live migration: location change not reflected until resync +- Evacuation: instance relocation not reflected until resync +- OC `inventory instance show ` shows: name, project, cluster, server, timestamp + +### Server management (tested) + +```bash +operations-center provisioning server list # Lists all servers + OC itself +operations-center provisioning server show +operations-center provisioning server resync +operations-center provisioning server system reboot # WARNING: see below +operations-center provisioning server system poweroff +``` + +**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot` +command sends a guest-level (ACPI) reboot. On Proxmox VMs with TPM2-based +disk encryption, this corrupts the TPM PCR measurements. The IncusOS node +becomes pingable but all services fail to start (port 8443 never opens). +Even a Proxmox-level stop/start does not recover the node. The only fix is +to destroy and redeploy the VM. + +**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot. +The proper lifecycle for maintenance is: +1. `incus cluster evacuate : --force` +2. Proxmox stop the VM +3. Proxmox start the VM +4. Wait for IncusOS to boot (~60-90s) +5. `incus cluster restore : --force` + +**OC does not track Incus cluster member state**: the server list always +shows `ready` status even when the Incus cluster reports `EVACUATED`. + +### Server edit commands + +Network and storage have `edit` subcommands (no `show`): + +```bash +operations-center provisioning server system network edit +operations-center provisioning server system storage edit +``` + +### Update management (tested) + +```bash +operations-center provisioning update list # Shows all available IncusOS versions +operations-center provisioning update show # Full file list with sizes +operations-center provisioning update refresh # Check for new updates +``` + +Updates show comprehensive file lists including sysexts: incus, operations-center, +gpu-support, incus-ceph, incus-linstor, migration-manager, debug. + +### System configuration (tested) + +```bash +operations-center system settings show # log_level +operations-center system security show # OIDC, OpenFGA, ACME, trusted certs +operations-center system updates show # Update source, filter expressions, signature CA +``` + +Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`. + +### Admin / IncusOS management (tested) + +```bash +operations-center admin os show # hostname, os_version, uptime +operations-center admin os application list +operations-center admin os application show +operations-center admin os service list # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip +operations-center admin os debug log +``` + +### Cluster operations (tested) + +```bash +operations-center provisioning cluster resync +operations-center provisioning cluster rename +operations-center provisioning cluster update --connection-url +operations-center provisioning cluster update-certificate +operations-center provisioning cluster artifact list +operations-center provisioning cluster factory-reset +``` + +### Cluster templates + +```bash +operations-center provisioning cluster-template add --description "..." +operations-center provisioning cluster-template list +operations-center provisioning cluster-template show +operations-center provisioning cluster-template remove +``` + +--- + +## Part 6: Cluster Lifecycle (Tested) + +### Live migration visibility through OC (tested) + +```bash +# Prepare VM for live migration +incus stop oc-node-01:test-vm-01 +incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1 +incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB +incus start oc-node-01:test-vm-01 + +# Migrate +incus move oc-node-01:test-vm-01 --target oc-node-03 + +# OC does NOT auto-update -- resync required +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list # now shows new location +``` + +**Tested**: ~141 MB/s migration speed. OC inventory reflected the location +change only after explicit `cluster resync`. + +### Cluster evacuation through OC (tested) + +```bash +incus cluster evacuate oc-node-01:oc-node-01 --force +# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state) +operations-center provisioning cluster resync oc-cluster +operations-center inventory instance list # shows workloads moved to other nodes + +incus cluster restore oc-node-01:oc-node-01 --force +``` + +### Node replacement lifecycle (tested) + +Full procedure tested after OC reboot broke a node: + +```bash +# 1. Force-remove dead node from Incus cluster +printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force + +# 2. Destroy broken VM via Proxmox (stop + delete) + +# 3. Redeploy via incusos-proxmox --iso (single-node config) +./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml + +# 4. New node auto-registers with OC (same hostname, new IP) +# NOTE: OC shows stale entry for old node-03. Stale entry can't be removed +# while OC thinks it's in a cluster. Resync doesn't fix stale entries. + +# 5. Manual cluster join (since node was force-removed) +incus config set oc-node-03: core.https_address :8443 +# Remove apply_defaults resources +incus config unset oc-node-03: storage.backups_volume +incus config unset oc-node-03: storage.images_volume +incus storage volume delete oc-node-03:local backups +incus storage volume delete oc-node-03:local images +incus profile device remove oc-node-03:default root +incus profile device remove oc-node-03:default eth0 +incus storage delete oc-node-03:local +incus network delete oc-node-03:incusbr0 +# Generate token and join +incus cluster add oc-node-01:oc-node-03 +printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03: +# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering) + +# 6. Fix remote after cert change +incus remote remove oc-node-03 +incus remote add oc-node-03 https://:8443 --accept-certificate +``` + +### Factory reset (not tested) + +```bash +# WARNING: destructive +operations-center provisioning cluster factory-reset oc-cluster +``` + +--- + +## OC CLI Command Reference + +### Remote management + +| Command | Description | +|---------|-------------| +| `operations-center remote add --auth-type tls` | Add OC remote | +| `operations-center remote switch ` | Set default remote | +| `operations-center remote list` | List configured remotes | +| `operations-center remote remove ` | Remove a remote | + +### Provisioning tokens + +| Command | Description | +|---------|-------------| +| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token | +| `provisioning token list` | List tokens | +| `provisioning token show ` | Token details | +| `provisioning token remove ` | Delete token | +| `provisioning token seed add --description "..."` | Attach named seed | +| `provisioning token seed list ` | List token seeds | +| `provisioning token seed show ` | Show seed details | +| `provisioning token seed get-image --type iso --architecture x86_64` | ISO from seed | +| `provisioning token get-image [preseed] --type iso --architecture x86_64` | ISO from inline preseed | + +### Provisioning servers + +| Command | Description | +|---------|-------------| +| `provisioning server list` | List registered servers (includes OC itself) | +| `provisioning server show ` | Server details + certificate | +| `provisioning server rename ` | Rename server | +| `provisioning server resync ` | Resync single server | +| `provisioning server remove ` | Remove (fails if in cluster) | +| `provisioning server edit ` | Edit server (interactive) | +| `provisioning server system reboot ` | Guest reboot (DANGEROUS on Proxmox) | +| `provisioning server system poweroff ` | Guest poweroff | +| `provisioning server system network edit ` | Edit network config | +| `provisioning server system storage edit ` | Edit storage config | + +### Provisioning clusters + +| Command | Description | +|---------|-------------| +| `provisioning cluster add --server-names ... --server-type incus -a ` | Form cluster | +| `provisioning cluster list` | List clusters | +| `provisioning cluster show ` | Cluster details + certificate | +| `provisioning cluster resync ` | Force inventory resync | +| `provisioning cluster rename ` | Rename cluster | +| `provisioning cluster update --connection-url ` | Update cluster URL | +| `provisioning cluster update-certificate ` | Update cluster cert | +| `provisioning cluster remove ` | Delete cluster | +| `provisioning cluster factory-reset ` | Factory reset | +| `provisioning cluster artifact list ` | List artifacts | +| `provisioning cluster artifact archive ` | Download artifact | + +### Cluster templates + +| Command | Description | +|---------|-------------| +| `provisioning cluster-template add --description "..."` | Create template | +| `provisioning cluster-template list` | List templates | +| `provisioning cluster-template show ` | Show template | +| `provisioning cluster-template remove ` | Delete template | + +### Inventory + +| Command | Description | +|---------|-------------| +| `inventory instance list` | List all instances | +| `inventory instance show ` | Instance details (by UUID, not name) | +| `inventory storage-pool list` | List storage pools | +| `inventory network list` | List networks | +| `inventory profile list` | List profiles | +| `inventory project list` | List projects | +| `inventory image list` | List images | + +### Updates + +| Command | Description | +|---------|-------------| +| `provisioning update list` | List available IncusOS versions | +| `provisioning update show ` | Version details + file list | +| `provisioning update refresh` | Check for new updates | + +### System + +| Command | Description | +|---------|-------------| +| `system settings show` | System settings (log_level) | +| `system security show` | OIDC, OpenFGA, ACME, trusted certs | +| `system updates show` | Update source, filter, signature CA | +| `system network show` | Network config | +| `system certificate set ` | Set server certificate | + +### Admin + +| Command | Description | +|---------|-------------| +| `admin os show` | IncusOS hostname, version, uptime | +| `admin os application list` | List applications | +| `admin os application show ` | Application state | +| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) | +| `admin os service show ` | Service config | +| `admin os debug log` | Debug logs | + +All commands prefixed with `operations-center`. + +--- + +## REST API Reference + +OC listens on port **8443** (same as Incus on IncusOS). All endpoints require +TLS client certificate authentication. + +### Common curl pattern + +```bash +curl -sk --cert ~/.config/incus/client.crt \ + --key ~/.config/incus/client.key \ + https://:8443/1.0/ +``` + +### Key endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/1.0` | Server info | +| GET | `/1.0/provisioning/updates` | List update packages | +| GET | `/1.0/provisioning/tokens` | List tokens | +| POST | `/1.0/provisioning/tokens` | Create token | +| GET | `/1.0/provisioning/servers` | List registered servers | +| GET | `/1.0/provisioning/clusters` | List clusters | +| POST | `/1.0/provisioning/clusters` | Form cluster | +| GET | `/1.0/inventory/instances` | List all instances | + +--- + +## Proxmox-Specific Considerations + +### force_reboot is required + +IncusOS does not auto-halt after installation. Without `force_reboot: true` +in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at +"please remove installation media" indefinitely. + +Options for OC-provisioned ISOs on Proxmox: +1. **Pre-seed file** passed to `get-image`: include `force_reboot: true` +2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with + `force_reboot: true` only + +### VM creation settings + +OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS +deployments (see `incusos-proxmox`): + +- `bios=ovmf`, `machine=q35` -- UEFI boot required +- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys +- `tpmstate0`: `version=v2.0` -- required for disk encryption +- `cpu=host` -- needed for x86_64_v3 instruction set requirement +- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS +- `balloon=0` -- IncusOS manages memory internally +- Minimum 50 GiB disk, minimum 4096 MiB RAM + +### Install media removal + +After installation completes, you **must** stop the VM and delete ide2 (and +ide3 if used) before booting from disk. IncusOS refuses to start if install +media is still attached. + +### IP detection + +IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to +find node IPs: + +```bash +# Get MAC from Proxmox VM config +MAC=$(curl -sk "https://:8006/api2/json/nodes/pve/qemu//config" \ + -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c " +import sys, json +d = json.load(sys.stdin)['data'] +print(d.get('net0', '').split('=')[1].split(',')[0])") + +# ARP lookup +ip neigh flush dev +ping -b -c 3 2>/dev/null +ip neigh show | grep -i "$MAC" | awk '{print $1}' +``` + +--- + +## Known Limitations (Tested Findings) + +### Confirmed limitations (v0.2.2) + +- **No brownfield adoption**: nodes must boot from OC-provisioned ISO +- **No real-time inventory sync**: requires explicit `cluster resync` +- **No cluster member state tracking**: OC always shows `ready`, even for + EVACUATED or OFFLINE nodes +- **Stale server entries**: out-of-band cluster changes (via `incus` CLI) + create stale entries that `resync` doesn't fix +- **OC reboot breaks Proxmox VMs**: guest reboot corrupts TPM encryption +- **Token seed format quirk**: requires structured YAML with section keys, + not flat format +- **Expired tokens not auto-cleaned**: remain in list after expiry +- **No cluster resize**: cannot add/remove members from existing clusters + +### Design constraints + +- OC runs on IncusOS (same immutable OS as Incus nodes) +- Uses port 8443 for both API and web UI +- Requires at least one trusted client certificate in seed +- Provisioning tokens have configurable use count and lifetime +- OC adds `meshbr0` network to clusters (not present in manual clustering) +- REST API endpoint: `/1.0` (same pattern as Incus) +- Auth methods: TLS client cert or OIDC + +--- + +## Troubleshooting (Tested) + +| Symptom | Likely cause | Fix | +|---------|-------------|-----| +| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` | +| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses | +| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` | +| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. | +| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates | +| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` | +| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors | +| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove ` then `incus remote add https://IP:8443 --accept-certificate` | +| OC reboot kills IncusOS node | TPM PCR corruption from guest reboot | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. | +| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. | +| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) | +| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove ` | +| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions | +| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt | +| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion | +| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster | +| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins | + +--- + +## Comparison: Manual vs OC Cluster Workflow (Tested) + +### Manual (incusos-proxmox + incus CLI) + +1. Write YAML config with VM specs +2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA +3. Manually set `core.https_address` on each node +4. Delete storage pool/network on joining nodes +5. `incus cluster enable` + `incus cluster join` per node +6. Re-add remotes after certificate changes +7. ~30 minutes for 3-node cluster + +### Hybrid (incusos-proxmox --iso + OC) + +1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml` +2. Create provisioning token + token seed in OC +3. Download OC-provisioned ISO via `token seed get-image` +4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml` +5. Nodes auto-register with OC (~30s each) +6. `operations-center provisioning cluster add oc-cluster ...` +7. ~20 minutes for 3-node cluster (including OC deploy) + +### What OC automates that we do manually + +| Step | Manual | OC | +|------|--------|-----| +| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add | +| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) | +| Cluster enable + join | Multi-step per node | Single `cluster add` command | +| Client cert injection | Via SEED_DATA | Via `--application-seed-config` | +| `meshbr0` network | Not created | Auto-created by OC clustering | +| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) | + +**Trade-offs**: +- Manual: full control, no extra server, works with any Incus version +- OC hybrid: auto-registration, single-command clustering, centralized + inventory/updates, but requires OC server + OC-provisioned ISOs +- OC reboot command is **dangerous** on Proxmox (breaks TPM-based encryption) +- OC inventory requires manual resync (not real-time) +- Out-of-band cluster changes (via `incus` CLI) create stale OC state + +--- + +## References + +- [Operations Center GitHub](https://github.com/FuturFusion/operations-center) +- [Operations Center releases](https://github.com/FuturFusion/operations-center/releases) +- [IncusOS GitHub](https://github.com/lxc/incus-os) +- [Incus documentation](https://linuxcontainers.org/incus/docs/main/) +- [Clustering guide](clustering-guide.md) (manual cluster formation reference)