OC-managed cluster with OVN and HA nginx: full validated deployment

Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:

- OC needs_update blocker: OC tracks update delivery through its pipeline,
  not version comparison. Nodes deployed from the latest ISO are stuck at
  needs_update=true. Fix: generate ISO from older channel (--channel
  old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
  same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
  Terraform error when cert is already injected via SEED_DATA.

Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -458,34 +458,61 @@ incu-contrib/
 - **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
 - **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
   must use structured format with section keys (`install:`, not flat).
+  **No force_reboot** in token seeds for Proxmox (same fix as standard deploys).
   ```bash
   operations-center provisioning token seed add <UUID> proxmox-preseed \
-      /tmp/preseed.yaml --description "Force reboot for Proxmox"
+      /tmp/preseed.yaml --description "No force_reboot for Proxmox"
   operations-center provisioning token seed get-image <UUID> proxmox-preseed \
-      /tmp/IncusOS-oc.iso --type iso --architecture x86_64
+      /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
   ```
 - **Hybrid deployment** (tested, recommended):
   `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
   combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
-  VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring,
+  VM creation, per-node SEED_DATA (hostname, static IP), install monitoring,
   and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
 - **Self-registration**: nodes auto-register with OC within ~30s of first
   boot. Hostname from SEED_DATA is used as the server name.
-- **Cluster formation** (tested):
+- **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires
+  all nodes to have `needs_update: false` before `provisioning cluster add`
+  succeeds. Nodes deployed from an ISO matching the latest OC update version
+  are tracked as `needs_update: true` because the OS was never delivered
+  through OC's update pipeline. The `needs_update` flag is server-side
+  computed (not a simple version comparison) and cannot be overridden via
+  REST API PUT. **Solution**: generate the ISO from an older channel
+  (`--channel old-stable`) so nodes start with an older version. OC then
+  pushes the latest update through its pipeline, clearing the flag. Use
+  `provisioning update assign-channels` to control which versions are in
+  which channels.
+- **Cluster formation** (tested, 2026-02-23):
   ```bash
+  # Use empty app config if cert already injected via SEED_DATA
+  echo '{}' > /tmp/oc-app-config.yaml
   operations-center provisioning cluster add oc-cluster \
       https://<NODE_01_IP>:8443 \
       --server-names oc-node-01,oc-node-02,oc-node-03 \
       --server-type incus \
       --application-seed-config /tmp/oc-app-config.yaml
   ```
-  OC handles: `core.https_address` → cluster enable → joins → seed
-  application → Terraform config. Adds `meshbr0` network.
+  OC handles: `core.https_address` → cluster enable → joins → storage pool
+  + network creation → Terraform config. Adds `meshbr0` network. If the
+  client cert is already in the trust store (from SEED_DATA), use an empty
+  `{}` app config to avoid "Certificate already in trust store" Terraform
+  error. The cluster forms successfully either way.
 - **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
   for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
   creation cleanly. With `apply_defaults: true`, nodes already have these
   resources and OC's Terraform fails with "already exists" errors (cluster still
   forms, but Terraform artifacts are empty).
+- **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster
+  formation via OC, OVN overlay networking works identically to manual clusters.
+  Deploy ovn-central container, enable OVN services, create UPLINK + OVN
+  network, add ovn-chassis roles. HA nginx workload with OVN load balancer
+  tested and working. See `notes/operations-center-guide.md` for full guide.
+- **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously
+  skipped uploading an ISO if one with the same filename existed on Proxmox.
+  This caused stale ISOs from previous deployments to be silently reused.
+  Fixed: when `--iso` is explicitly provided, the script now deletes the
+  existing ISO and re-uploads the new one.
 
 #### Tested limitations
 
@@ -505,6 +532,12 @@ incu-contrib/
 - **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
   node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
   if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
+- **`needs_update` tracking is pipeline-based**: OC tracks whether an update
+  was delivered through its pipeline, not just whether versions match. Nodes
+  deployed from the latest ISO are tracked as needing updates even when
+  `version == available_version`. Must deploy from older ISO to work around.
+- **OVN LB has no health checks**: connection-based hashing distributes
+  traffic to dead backends. Requests to stopped instances return empty.
 
 - See `notes/operations-center-guide.md` for full tested OC reference.
 

incusos/examples/lab-oc-deploy.yaml

@@ -33,7 +33,12 @@ defaults:
   disk: 50
   start_vmid: 920
 
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+
 vms:
   - name: oc-server
     app: operations-center
     apply_defaults: true
+    ip: 192.168.102.120/22

incusos/examples/lab-oc-nodes.yaml

@@ -20,15 +20,23 @@ defaults:
   disk: 50
   start_vmid: 400
 
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+
 vms:
   - name: oc-node-01
     app: incus
     apply_defaults: false
+    disk: 64
+    ip: 192.168.102.140/22
 
   - name: oc-node-02
     app: incus
     apply_defaults: false
+    ip: 192.168.102.141/22
 
   - name: oc-node-03
     app: incus
     apply_defaults: false
+    ip: 192.168.102.142/22

incusos/incusos-proxmox

@@ -3122,7 +3122,23 @@ phase_upload() {
     local iso_path="${WORKDIR}/${ISO_FILENAME}"
     if [[ -f "$iso_path" ]]; then
         if pve_iso_exists "$ISO_FILENAME"; then
+            if [[ -n "$LOCAL_ISO" ]]; then
+                # --iso was provided: always replace the ISO on Proxmox.
+                # A stale ISO with the same filename (e.g., IncusOS-oc.iso from
+                # a previous OC deployment) would silently use the wrong version.
+                warn "ISO '${ISO_FILENAME}' already exists on Proxmox -- replacing with provided ISO"
+                local encoded_volid
+                encoded_volid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${PVE_ISO_STORAGE}:iso/${ISO_FILENAME}', safe=''))")
+                pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
+                step "Uploading ISO: ${ISO_FILENAME}"
+                if ! pve_upload "$iso_path" "$ISO_FILENAME"; then
+                    error "Failed to upload ISO to Proxmox"
+                    exit 1
+                fi
+                success "ISO uploaded: ${ISO_FILENAME} (replaced)"
+            else
                 success "ISO already on Proxmox: ${ISO_FILENAME}"
+            fi
         else
             step "Uploading ISO: ${ISO_FILENAME}"
             if ! pve_upload "$iso_path" "$ISO_FILENAME"; then