From b9f18946813f681dcae5d9824ee79296c4f39906 Mon Sep 17 00:00:00 2001
From: Maarten <maarten@lambda-m.nl>
Date: Mon, 23 Feb 2026 11:08:28 +0100
Subject: [PATCH] OC-managed cluster with OVN and HA nginx: full validated
 deployment

Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:

- OC needs_update blocker: OC tracks update delivery through its pipeline,
  not version comparison. Nodes deployed from the latest ISO are stuck at
  needs_update=true. Fix: generate ISO from older channel (--channel
  old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
  same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
  Terraform error when cert is already injected via SEED_DATA.

Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 CLAUDE.md                           |   45 +-
 incusos/examples/lab-oc-deploy.yaml |    5 +
 incusos/examples/lab-oc-nodes.yaml  |    8 +
 incusos/incusos-proxmox             |   18 +-
 notes/operations-center-guide.md    | 1605 ++++++++++++++-------------
 5 files changed, 894 insertions(+), 787 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 2aae8bc..32283c0 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -458,34 +458,61 @@ incu-contrib/
 - **No brownfield adoption**: nodes must boot from an OC-provisioned ISO.
 - **Token seeds**: named, reusable pre-seed configs attached to tokens. YAML
   must use structured format with section keys (`install:`, not flat).
+  **No force_reboot** in token seeds for Proxmox (same fix as standard deploys).
   ```bash
   operations-center provisioning token seed add <UUID> proxmox-preseed \
-      /tmp/preseed.yaml --description "Force reboot for Proxmox"
+      /tmp/preseed.yaml --description "No force_reboot for Proxmox"
   operations-center provisioning token seed get-image <UUID> proxmox-preseed \
-      /tmp/IncusOS-oc.iso --type iso --architecture x86_64
+      /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
   ```
 - **Hybrid deployment** (tested, recommended):
   `incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes lab-oc-nodes.yaml`
   combines OC auto-registration (from boot ISO token) with `incusos-proxmox`
-  VM creation, per-node SEED_DATA (hostname, force_reboot), install monitoring,
+  VM creation, per-node SEED_DATA (hostname, static IP), install monitoring,
   and media cleanup. Dual seeds (boot ISO + SEED_DATA on ide3) coexist.
 - **Self-registration**: nodes auto-register with OC within ~30s of first
   boot. Hostname from SEED_DATA is used as the server name.
-- **Cluster formation** (tested):
+- **`needs_update` blocker** (critical discovery, 2026-02-23): OC requires
+  all nodes to have `needs_update: false` before `provisioning cluster add`
+  succeeds. Nodes deployed from an ISO matching the latest OC update version
+  are tracked as `needs_update: true` because the OS was never delivered
+  through OC's update pipeline. The `needs_update` flag is server-side
+  computed (not a simple version comparison) and cannot be overridden via
+  REST API PUT. **Solution**: generate the ISO from an older channel
+  (`--channel old-stable`) so nodes start with an older version. OC then
+  pushes the latest update through its pipeline, clearing the flag. Use
+  `provisioning update assign-channels` to control which versions are in
+  which channels.
+- **Cluster formation** (tested, 2026-02-23):
   ```bash
+  # Use empty app config if cert already injected via SEED_DATA
+  echo '{}' > /tmp/oc-app-config.yaml
   operations-center provisioning cluster add oc-cluster \
       https://<NODE_01_IP>:8443 \
       --server-names oc-node-01,oc-node-02,oc-node-03 \
       --server-type incus \
       --application-seed-config /tmp/oc-app-config.yaml
   ```
-  OC handles: `core.https_address` → cluster enable → joins → seed
-  application → Terraform config. Adds `meshbr0` network.
+  OC handles: `core.https_address` → cluster enable → joins → storage pool
+  + network creation → Terraform config. Adds `meshbr0` network. If the
+  client cert is already in the trust store (from SEED_DATA), use an empty
+  `{}` app config to avoid "Certificate already in trust store" Terraform
+  error. The cluster forms successfully either way.
 - **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
   for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
   creation cleanly. With `apply_defaults: true`, nodes already have these
   resources and OC's Terraform fails with "already exists" errors (cluster still
   forms, but Terraform artifacts are empty).
+- **OC-managed cluster with OVN**: fully tested (2026-02-23). After cluster
+  formation via OC, OVN overlay networking works identically to manual clusters.
+  Deploy ovn-central container, enable OVN services, create UPLINK + OVN
+  network, add ovn-chassis roles. HA nginx workload with OVN load balancer
+  tested and working. See `notes/operations-center-guide.md` for full guide.
+- **ISO upload skip bug** (fixed, 2026-02-23): `incusos-proxmox` previously
+  skipped uploading an ISO if one with the same filename existed on Proxmox.
+  This caused stale ISOs from previous deployments to be silently reused.
+  Fixed: when `--iso` is explicitly provided, the script now deletes the
+  existing ISO and re-uploads the new one.
 
 #### Tested limitations
 
@@ -505,6 +532,12 @@ incu-contrib/
 - **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
   node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
   if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
+- **`needs_update` tracking is pipeline-based**: OC tracks whether an update
+  was delivered through its pipeline, not just whether versions match. Nodes
+  deployed from the latest ISO are tracked as needing updates even when
+  `version == available_version`. Must deploy from older ISO to work around.
+- **OVN LB has no health checks**: connection-based hashing distributes
+  traffic to dead backends. Requests to stopped instances return empty.
 
 - See `notes/operations-center-guide.md` for full tested OC reference.
 
diff --git a/incusos/examples/lab-oc-deploy.yaml b/incusos/examples/lab-oc-deploy.yaml
index 3e25288..4bb114c 100644
--- a/incusos/examples/lab-oc-deploy.yaml
+++ b/incusos/examples/lab-oc-deploy.yaml
@@ -33,7 +33,12 @@ defaults:
   disk: 50
   start_vmid: 920
 
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+
 vms:
   - name: oc-server
     app: operations-center
     apply_defaults: true
+    ip: 192.168.102.120/22
diff --git a/incusos/examples/lab-oc-nodes.yaml b/incusos/examples/lab-oc-nodes.yaml
index 021c457..1e038e5 100644
--- a/incusos/examples/lab-oc-nodes.yaml
+++ b/incusos/examples/lab-oc-nodes.yaml
@@ -20,15 +20,23 @@ defaults:
   disk: 50
   start_vmid: 400
 
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+
 vms:
   - name: oc-node-01
     app: incus
     apply_defaults: false
+    disk: 64
+    ip: 192.168.102.140/22
 
   - name: oc-node-02
     app: incus
     apply_defaults: false
+    ip: 192.168.102.141/22
 
   - name: oc-node-03
     app: incus
     apply_defaults: false
+    ip: 192.168.102.142/22
diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox
index 86c4a44..0866b75 100755
--- a/incusos/incusos-proxmox
+++ b/incusos/incusos-proxmox
@@ -3122,7 +3122,23 @@ phase_upload() {
     local iso_path="${WORKDIR}/${ISO_FILENAME}"
     if [[ -f "$iso_path" ]]; then
         if pve_iso_exists "$ISO_FILENAME"; then
-            success "ISO already on Proxmox: ${ISO_FILENAME}"
+            if [[ -n "$LOCAL_ISO" ]]; then
+                # --iso was provided: always replace the ISO on Proxmox.
+                # A stale ISO with the same filename (e.g., IncusOS-oc.iso from
+                # a previous OC deployment) would silently use the wrong version.
+                warn "ISO '${ISO_FILENAME}' already exists on Proxmox -- replacing with provided ISO"
+                local encoded_volid
+                encoded_volid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${PVE_ISO_STORAGE}:iso/${ISO_FILENAME}', safe=''))")
+                pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
+                step "Uploading ISO: ${ISO_FILENAME}"
+                if ! pve_upload "$iso_path" "$ISO_FILENAME"; then
+                    error "Failed to upload ISO to Proxmox"
+                    exit 1
+                fi
+                success "ISO uploaded: ${ISO_FILENAME} (replaced)"
+            else
+                success "ISO already on Proxmox: ${ISO_FILENAME}"
+            fi
         else
             step "Uploading ISO: ${ISO_FILENAME}"
             if ! pve_upload "$iso_path" "$ISO_FILENAME"; then
diff --git a/notes/operations-center-guide.md b/notes/operations-center-guide.md
index ddcdc7a..aa4e335 100644
--- a/notes/operations-center-guide.md
+++ b/notes/operations-center-guide.md
@@ -1,62 +1,107 @@
 # Operations Center Guide for IncusOS
 
-This guide covers deploying and using Operations Center (OC) by FuturFusion --
-a centralized management layer for IncusOS/Incus deployments. OC handles
-provisioning, clustering, inventory, and updates through a REST API, CLI, and
-web UI.
-
-All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
-on Proxmox VE 9.1.5 (nested virtualization on Intel).
+End-to-end guide for deploying an OC-managed Incus cluster with OVN overlay
+networking and an HA nginx workload. All commands and output captured from a
+real deployment on 2026-02-23 using Operations Center v0.3.0 and IncusOS
+build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).
 
 ---
 
-## Overview
+## Section 0: Architecture Overview
 
-Operations Center is fundamentally different from managing Incus clusters
-manually via `incus` CLI remotes. Key differences:
-
-| | Manual (incus CLI) | Operations Center |
-|---|---|---|
-| Node provisioning | `incusos-proxmox` + seed archives | OC-provisioned ISOs + `incusos-proxmox --iso` (hybrid) |
-| Cluster formation | `incus cluster enable` + `incus cluster join` | OC orchestrates via CLI/API/web UI |
-| Node registration | Manual remote add | Automatic self-registration on first boot |
-| Inventory | `incus list` per remote | Centralized view across all clusters |
-| Updates | Manual per-node | OC manages update channels and rollouts |
-| Brownfield adoption | N/A (nodes are standalone) | **Not supported** -- nodes must be OC-provisioned |
-
-**Key constraint**: OC only manages nodes it has provisioned. There is no
-"adopt existing node" workflow. Nodes must boot from an OC-generated ISO
-containing a provisioning token, then self-register on first boot.
-
----
-
-## Prerequisites
-
-### Install OC CLI
-
-```bash
-# Check for latest release at https://github.com/FuturFusion/operations-center/releases
-wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
-chmod +x operations-center_linux_x86_64
-sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
-operations-center --version  # Note: use --version, not 'version' subcommand
+```
+                    ┌──────────────────────────────────────────┐
+                    │           Proxmox VE Host (pve)          │
+                    │  Intel i9-13900HK  64 GiB  20 cores      │
+                    │                                          │
+                    │  ┌──────────┐  ┌──────────┐  ┌──────────┐│
+                    │  │oc-node-01│  │oc-node-02│  │oc-node-03││
+                    │  │ VMID 400 │  │ VMID 401 │  │ VMID 402 ││
+                    │  │ 4c/8G/64G│  │ 4c/8G/50G│  │ 4c/8G/50G││
+                    │  │ .140     │  │ .141     │  │ .142     ││
+                    │  │          │  │          │  │          ││
+                    │  │ovn-central│ │          │  │          ││
+                    │  │ha-web-01 │  │ha-web-02 │  │ha-web-03 ││
+                    │  └──────────┘  └──────────┘  └──────────┘│
+                    │         │            │            │       │
+                    │         └────────────┼────────────┘       │
+                    │              OVN Geneve tunnels           │
+                    │              net-prod 10.10.10.0/24       │
+                    │                                          │
+                    │  ┌──────────┐                            │
+                    │  │oc-server │  OC Management              │
+                    │  │ VMID 920 │  https://192.168.102.120    │
+                    │  │ 2c/4G/50G│  Web UI + CLI + REST API    │
+                    │  │ .120     │                             │
+                    │  └──────────┘                            │
+                    │                                          │
+                    │  vmbr0 (VLAN 69) ── 192.168.100.0/22     │
+                    └──────────────────────────────────────────┘
+                              │
+                              │ gateway 192.168.100.1
+                              │
+                    OVN external IPs: 192.168.103.200-210
+                    LB: 192.168.103.201  Fwd: 192.168.103.202
 ```
 
-### Set up OC client certificates
+### Infrastructure
+
+| Component | VMID | IP | Cores | RAM | Disk | Role |
+|-----------|------|-----|-------|-----|------|------|
+| oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center |
+| oc-node-01 | 400 | 192.168.102.140/22 | 4 | 8 GiB | 64G | Cluster init + OVN central |
+| oc-node-02 | 401 | 192.168.102.141/22 | 4 | 8 GiB | 50G | Cluster member |
+| oc-node-03 | 402 | 192.168.102.142/22 | 4 | 8 GiB | 50G | Cluster member |
+
+**RAM budget**: 28 GiB of 64 GiB (44%). VLAN 69, gateway 192.168.100.1.
+
+### Decision Rationale
+
+- **OC-managed cluster**: OC is the future default clustering path for Incus.
+  Supports provisioning, update management, inventory, and centralized control.
+- **OVN overlay**: cross-node L2 connectivity for HA workloads. Sub-ms latency.
+- **HA nginx**: demonstrates load balancing, failover, and network forwards.
+- **Static IPs**: required for VLAN 69 (ARP-based detection doesn't work
+  across VLANs).
+- **apply_defaults: false**: recommended for OC-managed nodes. OC's Terraform
+  handles resource creation during cluster formation.
+
+### Cross-References
+
+- [Clustering guide](clustering-guide.md) -- manual cluster formation reference
+- [Networking guide](networking-guide.md) -- OVN overlay tutorial
+- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA
+
+---
+
+## Section 1: Prerequisites
+
+### Required Tools
+
+```bash
+# Operations Center CLI
+operations-center --version
+# operations-center version 0.3.0
+
+# Incus client
+incus version
+# Client version: 6.12
+
+# incusos-proxmox (from this repo)
+./incusos/incusos-proxmox --doctor
+```
+
+### Client Certificates
 
 OC uses the same TLS client certificate format as Incus:
 
 ```bash
+# Copy Incus certs for OC CLI
 mkdir -p ~/.config/operations-center
 cp ~/.config/incus/client.crt ~/.config/operations-center/
 cp ~/.config/incus/client.key ~/.config/operations-center/
-```
 
-### Generate PKCS#12 cert for browser access
-
-The OC web UI requires a client certificate for authentication:
-
-```bash
+# Generate PKCS#12 for browser access to OC web UI
 openssl pkcs12 -export \
     -inkey ~/.config/incus/client.key \
     -in ~/.config/incus/client.crt \
@@ -64,163 +109,160 @@ openssl pkcs12 -export \
 # Import client.pfx into Firefox: Settings -> Certificates -> Your Certificates -> Import
 ```
 
-### Verify with doctor check
-
-```bash
-./incusos/incusos-proxmox --doctor
-# Should report OC CLI as installed
-```
-
 ---
 
-## Part 1: Deploy Operations Center Server
+## Section 2: Deploy OC Server
 
-### Deployment config
-
-Use `incusos/examples/lab-oc-deploy.yaml` -- deploys only the OC server
-(Incus nodes will be provisioned by OC itself):
+### Configuration File
 
 ```yaml
+# incusos/examples/lab-oc-deploy.yaml
 defaults:
   cores: 2
   memory: 4096
   disk: 50
   start_vmid: 920
 
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+
 vms:
   - name: oc-server
     app: operations-center
     apply_defaults: true
+    ip: 192.168.102.120/22
 ```
 
 ### Deploy
 
 ```bash
-./incusos/incusos-proxmox --dry-run incusos/examples/lab-oc-deploy.yaml
-./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
-./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
+./incusos/incusos-proxmox --yes examples/lab-oc-deploy.yaml
 ```
 
-Note the OC IP address from `--status` output.
+Actual output (key lines):
 
-### Set up OC CLI remote
+```
+[ok]  VM 'oc-server' created (VMID 920)
+[ok]  VM 'oc-server' installed and running at 192.168.102.120
+```
+
+### Set Up OC CLI Remote
 
 ```bash
-operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
+operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
 operations-center remote switch oc-lab
-operations-center remote list
 ```
 
-### Verify browser access
-
-Navigate to `https://<OC_IP>:8443` in Firefox (with client.pfx imported).
-The OC web UI should load and show the dashboard.
-
-### Wait for updates to download
-
-OC automatically downloads IncusOS update packages from upstream. At least
-one update must be in `ready` state before provisioned ISOs can be generated:
+### Verify OC
 
 ```bash
+operations-center admin os show
+```
+
+Actual output:
+
+```
+environment:
+  hostname: oc-server
+  os_name: IncusOS
+  os_version: "202602230420"
+  os_version_next: ""
+  uptime: 4956
+```
+
+### Wait for Updates
+
+OC downloads IncusOS update packages from upstream. At least one update must
+reach `ready` state before ISOs can be generated:
+
+```bash
+# Poll until at least one update shows "ready"
 operations-center provisioning update list
-# Wait until at least one update shows "ready" status
-# This may take several minutes on first boot
 ```
 
-Also via REST API:
+Actual output (after ~5 minutes):
 
-```bash
-curl -sk --cert ~/.config/incus/client.crt \
-     --key ~/.config/incus/client.key \
-     https://<OC_IP>:8443/1.0/provisioning/updates
+```
++--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+
+|                 UUID                 |       Origin        | Channels | Upstream Channels |   Version    | Sev  | Status |
++--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+
+| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable   | stable, testing   | 202602200553 | none | ready  |
+| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable   | stable, testing   | 202602210344 | none | ready  |
+| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable   | stable, testing   | 202602230420 | none | ready  |
++--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+
 ```
 
 ---
 
-## Part 2: Provision Incus Node ISOs
+## Section 3: Provision Nodes
 
-### Create a provisioning token
+### 3.1 Create Provisioning Token
 
 ```bash
-# CLI
-operations-center provisioning token add \
-    --description "Lab cluster nodes" \
-    --uses 5
-
-# Note the token UUID from output
+operations-center provisioning token add --uses 5 --description "OC-managed cluster v2"
 operations-center provisioning token list
-operations-center provisioning token show <UUID>
 ```
 
-Via REST API:
+Actual output:
 
-```bash
-curl -sk --cert ~/.config/incus/client.crt \
-     --key ~/.config/incus/client.key \
-     -X POST https://<OC_IP>:8443/1.0/provisioning/tokens \
-     -d '{"description": "Lab cluster nodes (API)", "uses": 5}'
+```
++--------------------------------------+----------------+-------------------------------+---------+-----------------------+
+|                 UUID                 | Uses Remaining |           Expire At           | Channel |      Description      |
++--------------------------------------+----------------+-------------------------------+---------+-----------------------+
+| 2ba12420-fee5-4d55-9f2f-40809e9ad5de | 5              | 2026-03-25 10:29:39 +0100 CET | stable  | OC-managed cluster v2 |
++--------------------------------------+----------------+-------------------------------+---------+-----------------------+
 ```
 
-### Create a token seed for Proxmox VMs (tested)
+### 3.2 Create Token Seed (No force_reboot)
 
-OC supports **token seeds** -- named, reusable pre-seed configs attached to a
-token. The YAML format uses section-level keys (`install:`, `applications:`,
-etc.) -- NOT the flat format from standard seed archives.
+**Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
+`incusos-proxmox` handles the install lifecycle externally (blockstat
+detection + media removal). `force_reboot` triggers SysRq-B which causes
+the ~50% crontab bug (see CLAUDE.md for full investigation).
 
 ```yaml
-# /tmp/oc-preseed.yaml -- structured format required for token seeds
+# /tmp/oc-preseed.yaml
 install:
   version: "1"
   force_install: true
-  force_reboot: true
 ```
 
-**Important**: a flat format (`version: "1"` at root) maps all fields to empty
-`{}` -- they don't get assigned to any section. The structured format with
-`install:` as the top-level key is required.
+**Important**: use the structured format with section keys (`install:`). A flat
+format (`version: "1"` at root) maps fields to empty `{}` and they don't get
+assigned to any section.
 
 ```bash
 operations-center provisioning token seed add <UUID> proxmox-preseed \
-    /tmp/oc-preseed.yaml \
-    --description "Force reboot for Proxmox VMs"
-
-operations-center provisioning token seed list <UUID>
-operations-center provisioning token seed show <UUID> proxmox-preseed
+    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
 ```
 
-### Download the OC-provisioned ISO (tested)
+### 3.3 Generate OC-Provisioned ISO (Older Version)
 
-Use `token seed get-image` for the cleanest approach (pre-seed embedded in ISO):
+**Critical discovery**: nodes deployed from an ISO matching the latest OC
+update version are tracked as `needs_update: true` by OC because the OS
+was never delivered through OC's update pipeline. The fix: generate the ISO
+from an older channel so OC can push the real update after deployment.
 
 ```bash
+# Assign the second-latest update to a separate channel
+operations-center provisioning update assign-channels <older-UUID> --channel old-stable
+
+# Generate ISO from the older channel
 operations-center provisioning token seed get-image <UUID> proxmox-preseed \
-    /tmp/IncusOS-oc.iso --type iso --architecture x86_64
-
-ls -lh /tmp/IncusOS-oc.iso  # ~3.4 GB
+    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
 ```
 
-Alternative without token seeds (inline pre-seed):
+Actual output:
 
-```bash
-operations-center provisioning token get-image <UUID> /tmp/IncusOS-oc.iso \
-    /tmp/oc-preseed.yaml --type iso --architecture x86_64
+```
+Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"
 ```
 
----
+The ISO contains IncusOS 202602210344 (one version behind). OC will push
+the latest (202602230420) after nodes register.
 
-## Part 3: Deploy Incus Nodes (Hybrid Approach -- Tested)
-
-**The hybrid approach** uses `incusos-proxmox --iso` to deploy nodes from an
-OC-provisioned ISO. This combines the best of both worlds:
-- **OC**: auto-registration token embedded in the boot ISO
-- **incusos-proxmox**: VM creation, per-node SEED_DATA (hostname, force_reboot),
-  install monitoring, media cleanup, IP detection, remote setup
-
-The OC token lives in the boot ISO (ide2). Our SEED_DATA (ide3) provides
-force_reboot, hostname, and app selection. IncusOS reads both sources and
-merges them -- they coexist without conflict.
-
-### Deployment config
+### 3.4 Node Configuration
 
 ```yaml
 # incusos/examples/lab-oc-nodes.yaml
@@ -228,82 +270,125 @@ defaults:
   cores: 4
   memory: 8192
   disk: 50
-  start_vmid: 400   # Use high VMIDs to avoid collisions with VMs outside pool
+  start_vmid: 400
+
+proxmox:
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
 
 vms:
   - name: oc-node-01
     app: incus
-    apply_defaults: false   # OC handles storage/network via Terraform
+    apply_defaults: false
+    disk: 64
+    ip: 192.168.102.140/22
 
   - name: oc-node-02
     app: incus
     apply_defaults: false
+    ip: 192.168.102.141/22
 
   - name: oc-node-03
     app: incus
     apply_defaults: false
+    ip: 192.168.102.142/22
 ```
 
-**VMID note**: if your Proxmox API token is scoped to a resource pool, it
-cannot see VMs outside that pool. Use `start_vmid` above existing VM ranges
-to avoid VMID collisions with invisible VMs.
+### 3.5 Deploy Nodes (Hybrid Approach)
 
-### Deploy (tested)
+The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration
+(from the boot ISO token) with `incusos-proxmox` VM creation, per-node SEED_DATA
+(hostname, static IP), install monitoring, and media cleanup.
 
 ```bash
-./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --dry-run \
-    incusos/examples/lab-oc-nodes.yaml
-
-./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
-    incusos/examples/lab-oc-nodes.yaml
-
-./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
+./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-oc-nodes.yaml
 ```
 
-`incusos-proxmox` handles everything: VM creation, ISO upload, SEED_DATA
-generation (force_reboot + hostname per node), install monitoring via
-blockstat polling, media removal, boot order switch, IP detection, and
-incus remote setup.
+Actual output (key lines):
 
-### Verify OC auto-registration (tested)
+```
+[ok]  ISO uploaded: IncusOS-oc.iso
+[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
+[ok]  Remote 'oc-node-01' added (192.168.102.140)
+[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
+[ok]  Remote 'oc-node-02' added (192.168.102.141)
+[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
+[ok]  Remote 'oc-node-03' added (192.168.102.142)
+[ok]  All post-deployment checks passed
+```
 
-After nodes boot from disk, they auto-register with OC within ~30 seconds:
+All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.
+
+### 3.6 Verify Auto-Registration
+
+Nodes auto-register with OC within ~30 seconds of first boot. The update
+from 202602210344 to 202602230420 happens automatically:
 
 ```bash
 operations-center provisioning server list
 ```
 
-Nodes register with their hostname (set via SEED_DATA), so no renaming needed.
-Each node appears with type `incus`, status `ready`, and its connection URL.
+Actual output:
 
-**Tested result**: all 3 nodes auto-registered with correct hostnames,
-correct IPs, type `incus`, status `ready`. No manual intervention needed.
+```
++------------+-------------------+------------------------------+--------+-------------+
+|  Cluster   |       Name        |        Connection URL        | Status | Update Stat |
++------------+-------------------+------------------------------+--------+-------------+
+|            | oc-node-01        | https://192.168.102.140:8443 | ready  | up to date  |
+|            | oc-node-02        | https://192.168.102.141:8443 | ready  | up to date  |
+|            | oc-node-03        | https://192.168.102.142:8443 | ready  | up to date  |
+|            | operations-center | https://[::1]:8443           | ready  | update pend |
++------------+-------------------+------------------------------+--------+-------------+
+```
+
+**Key**: all 3 nodes show **"up to date"** because OC delivered the
+202602230420 update through its pipeline. This is what unlocks clustering.
+
+### 3.7 Verify Scrub Schedules
+
+```bash
+for node in oc-node-01 oc-node-02 oc-node-03; do
+    incus query ${node}:/os/1.0/system/storage | python3 -c \
+        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
+done
+```
+
+Actual output:
+
+```
+oc-node-01: 0 4 * * 0
+oc-node-02: 0 4 * * 0
+oc-node-03: 0 4 * * 0
+```
+
+All healthy. No crontab bug (force_reboot was not used).
 
 ---
 
-## Part 4: Form Cluster via Operations Center
+## Section 4: Form Cluster via Operations Center
 
-### Prepare application config
+### 4.1 The needs_update Blocker
 
-Create an Incus preseed that injects the client certificate into the cluster
-(so the `incus` CLI can connect directly):
+OC requires all nodes to show `needs_update: false` before clustering.
+Nodes deployed from an ISO matching the latest version are tracked as
+`needs_update: true` because the OS was never delivered through OC's update
+pipeline. The `needs_update` flag is server-side computed and cannot be
+overridden via REST API PUT.
 
-```yaml
-# /tmp/oc-app-config.yaml
-certificates:
-  - type: client
-    name: lab-client-cert
-    certificate: |-
-      -----BEGIN CERTIFICATE-----
-      <contents of ~/.config/incus/client.crt>
-      -----END CERTIFICATE-----
-```
+**Solution**: deploy from an older ISO version (Section 3.3). OC then pushes
+the real update to nodes through its pipeline, clearing the flag.
 
-### Form cluster via OC CLI (tested)
+### 4.2 Form Cluster
+
+**Important**: if the client certificate was already injected via SEED_DATA,
+use an empty application seed config to avoid "Certificate already in trust
+store" Terraform errors:
 
 ```bash
+echo '{}' > /tmp/oc-app-config.yaml
+
 operations-center provisioning cluster add oc-cluster \
-    https://<NODE_01_IP>:8443 \
+    https://192.168.102.140:8443 \
     --server-names oc-node-01,oc-node-02,oc-node-03 \
     --server-type incus \
     --application-seed-config /tmp/oc-app-config.yaml
@@ -311,687 +396,645 @@ operations-center provisioning cluster add oc-cluster \
 
 OC orchestrates the full cluster formation:
 1. Sets `core.https_address` to each node's specific IP
-2. Enables clustering on the first node
-3. Joins remaining nodes (handles storage pool + network creation)
-4. Applies the application seed (injects client cert)
+2. Enables clustering on oc-node-01
+3. Joins oc-node-02 and oc-node-03
+4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`)
 5. Runs Terraform/OpenTofu for post-cluster configuration
 
-### apply_defaults: false (tested, recommended)
+### 4.3 Fix Remotes After Clustering
 
-**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
-and recommended approach. With `apply_defaults: false`:
-
-- Nodes still listen on port 8443 and trust preseed certificates
-- The underlying ZFS dataset (`local/incus`) still exists
-- But no Incus storage pool, network, or profile devices are created
-- OC's Terraform handles all resource creation cleanly during cluster formation
-
-**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
-Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
-and injects the client certificate via `--application-seed-config`. The only
-Terraform error is "Certificate already in trust store" when the cert was also
-injected via SEED_DATA -- avoid this by only injecting the cert through the
-application seed config, not the SEED_DATA.
-
-### apply_defaults: true (also works, with caveats)
-
-When using `apply_defaults: true`, each node already has a `local` ZFS storage
-pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
-then fails with:
-
-- "Certificate already in trust store"
-- "Network is not in pending state"
-- "Storage pool is not in pending state"
-
-**The cluster itself forms successfully** despite these Terraform errors.
-However, `cluster artifact list` will be empty since Terraform didn't complete.
-
-### Verify cluster (tested)
+Clustering regenerates TLS certificates. Re-add the remotes:
 
 ```bash
-# Via OC
-operations-center provisioning cluster list
-operations-center provisioning cluster show oc-cluster
+incus remote remove oc-node-01
+incus remote remove oc-node-02
+incus remote remove oc-node-03
+incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
+incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
+incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
+```
 
-# Via Incus (direct cluster access)
-# NOTE: clustering regenerates TLS certificates, so re-add remotes:
-incus remote remove oc-node-01  # if existing remote has wrong cert
-incus remote add oc-node-01 https://<NODE_01_IP>:8443 --accept-certificate
+### 4.4 Verify Cluster
+
+```bash
 incus cluster list oc-node-01:
 ```
 
-### Cluster artifacts
+Actual output:
 
-OC generates Terraform/OpenTofu artifacts on successful cluster formation:
+```
++------------+------------------------------+-----------------+--------------+--------+-------------------+
+|    NAME    |             URL              |      ROLES      | ARCHITECTURE | STATUS |      MESSAGE      |
++------------+------------------------------+-----------------+--------------+--------+-------------------+
+| oc-node-01 | https://192.168.102.140:8443 | ovn-chassis     | x86_64       | ONLINE | Fully operational |
+|            |                              | database-leader |              |        |                   |
+|            |                              | database        |              |        |                   |
++------------+------------------------------+-----------------+--------------+--------+-------------------+
+| oc-node-02 | https://192.168.102.141:8443 | ovn-chassis     | x86_64       | ONLINE | Fully operational |
+|            |                              | database        |              |        |                   |
++------------+------------------------------+-----------------+--------------+--------+-------------------+
+| oc-node-03 | https://192.168.102.142:8443 | ovn-chassis     | x86_64       | ONLINE | Fully operational |
+|            |                              | database        |              |        |                   |
++------------+------------------------------+-----------------+--------------+--------+-------------------+
+```
+
+All 3 nodes ONLINE and Fully operational.
+
+### 4.5 Cluster Resources Created by OC
 
 ```bash
-operations-center provisioning cluster artifact list oc-cluster
-# Empty if Terraform failed (see apply_defaults conflict above)
+incus storage list oc-node-01:
+incus network list oc-node-01:
+```
+
+Actual output:
+
+```
++-------+--------+--------------------------------------+---------+---------+
+| NAME  | DRIVER |             DESCRIPTION              | USED BY |  STATE  |
++-------+--------+--------------------------------------+---------+---------+
+| local | zfs    | Local storage pool (on system drive)  | 8       | CREATED |
++-------+--------+--------------------------------------+---------+---------+
+
++----------+--------+---------+----------------+------------------------------+---------+
+|   NAME   |  TYPE  | MANAGED |      IPV4      |         DESCRIPTION          | USED BY |
++----------+--------+---------+----------------+------------------------------+---------+
+| incusbr0 | bridge | YES     | 10.149.76.1/24 | Local network bridge (NAT)   | 2       |
+| meshbr0  | bridge | YES     | none           | Internal mesh network bridge | 1       |
++----------+--------+---------+----------------+------------------------------+---------+
+```
+
+OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
+`meshbr0` (OC-specific mesh network for inter-node communication).
+
+---
+
+## Section 5: OVN Overlay Networking
+
+### 5.1 Deploy OVN Control Plane
+
+Launch an `ovn-central` container on oc-node-01:
+
+```bash
+incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
+incus exec oc-node-01:ovn-central -- bash -c \
+    "apt-get update -qq && apt-get install -y -qq ovn-central"
+```
+
+Configure OVN to listen on all interfaces:
+
+```bash
+incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
+incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
+```
+
+Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:
+
+```bash
+incus config device add oc-node-01:ovn-central \
+    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
+incus config device add oc-node-01:ovn-central \
+    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642
+```
+
+### 5.2 Enable OVN on All IncusOS Nodes
+
+Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`).
+The `database` field is the **southbound** DB (port 6642), not northbound.
+
+```bash
+for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
+    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
+    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
+        \"config\": {
+            \"database\": \"tcp:192.168.102.140:6642\",
+            \"enabled\": true,
+            \"tunnel_address\": \"${node_ip}\",
+            \"tunnel_protocol\": \"geneve\"
+        },
+        \"state\": {}
+    }"
+done
+```
+
+### 5.3 Configure Incus OVN Connection
+
+```bash
+incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641
+```
+
+### 5.4 Assign OVN Chassis Role
+
+```bash
+for node in oc-node-01 oc-node-02 oc-node-03; do
+    incus cluster role add oc-node-01:${node} ovn-chassis
+done
+```
+
+### 5.5 Create UPLINK Physical Network
+
+Uses `parent=mgmt` (IncusOS management NIC). Two-step cluster pattern:
+
+```bash
+# Per-target (parent is member-specific)
+for node in oc-node-01 oc-node-02 oc-node-03; do
+    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
+done
+
+# Cluster-wide config
+incus network create oc-node-01:UPLINK --type=physical \
+    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
+    ipv4.gateway=192.168.100.1/22 \
+    dns.nameservers=192.168.100.1
+```
+
+### 5.6 Create OVN Network (net-prod)
+
+```bash
+incus network create oc-node-01:net-prod --type=ovn \
+    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true
+```
+
+Actual output:
+
+```
+Network net-prod created
+```
+
+net-prod is assigned external IP `192.168.103.200` from the UPLINK range.
+
+### 5.7 Verify Cross-Node OVN Connectivity
+
+```bash
+incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
+incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
+incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
+```
+
+Actual output:
+
+```
+64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.744 ms
+64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.169 ms
+64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.150 ms
+```
+
+Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
+containers after verification.
+
+### 5.8 Final Network State
+
+```bash
+incus network list oc-node-01:
+```
+
+Actual output:
+
+```
++----------+----------+---------+----------------+------------------------------+---------+
+|   NAME   |   TYPE   | MANAGED |      IPV4      |         DESCRIPTION          | USED BY |
++----------+----------+---------+----------------+------------------------------+---------+
+| UPLINK   | physical | YES     |                |                              | 1       |
+| incusbr0 | bridge   | YES     | 10.149.76.1/24 | Local network bridge (NAT)   | 2       |
+| meshbr0  | bridge   | YES     | none           | Internal mesh network bridge | 1       |
+| net-prod | ovn      | YES     | 10.10.10.1/24  |                              | 3       |
++----------+----------+---------+----------------+------------------------------+---------+
 ```
 
 ---
 
-## Part 5: OC Management Capabilities (Tested)
+## Section 6: HA Nginx Workload
 
-### Inventory browsing
+### 6.1 Launch Nginx Containers
 
-OC inventory is **not real-time** -- it requires an explicit `cluster resync`
-to pick up changes made via the `incus` CLI:
+Launch 3 nginx containers across all nodes on net-prod:
+
+```bash
+incus launch images:debian/12 oc-node-01:ha-web-01 --target oc-node-01 -n net-prod
+incus launch images:debian/12 oc-node-01:ha-web-02 --target oc-node-02 -n net-prod
+incus launch images:debian/12 oc-node-01:ha-web-03 --target oc-node-03 -n net-prod
+```
+
+### 6.2 Install Nginx and Set Distinct Content
+
+```bash
+for i in 01 02 03; do
+    incus exec oc-node-01:ha-web-${i} -- bash -c \
+        "apt-get update -qq && apt-get install -y -qq nginx"
+    incus exec oc-node-01:ha-web-${i} -- bash -c \
+        "echo '<h1>ha-web-${i} on oc-node-${i}</h1>' > /var/www/html/index.html"
+    incus exec oc-node-01:ha-web-${i} -- systemctl start nginx
+done
+```
+
+### 6.3 Create OVN Load Balancer
+
+```bash
+incus network load-balancer create oc-node-01:net-prod 192.168.103.201
+
+# Add backends (must use IP addresses, not instance names)
+incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
+    ha-web-01 10.10.10.2 80
+incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
+    ha-web-02 10.10.10.3 80
+incus network load-balancer backend add oc-node-01:net-prod 192.168.103.201 \
+    ha-web-03 10.10.10.4 80
+
+# Add port forwarding rule
+incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \
+    tcp 80 ha-web-01,ha-web-02,ha-web-03
+```
+
+### 6.4 Test Load Balancer
+
+```bash
+for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done
+```
+
+Actual output:
+
+```
+<h1>ha-web-03 on oc-node-03</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+<h1>ha-web-01 on oc-node-01</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+```
+
+Traffic distributes across all 3 backends (connection-based hashing, not
+round-robin).
+
+### 6.5 HA Failover Test
+
+```bash
+# Stop ha-web-01
+incus stop oc-node-01:ha-web-01
+sleep 2
+
+# Test with one backend down
+for i in 1 2 3 4 5; do curl -s --connect-timeout 5 http://192.168.103.201; done
+```
+
+Actual result: remaining backends (ha-web-02, ha-web-03) serve traffic. Some
+requests to the dead backend return empty (OVN LB uses connection hashing,
+not health checks — no active health monitoring).
+
+```bash
+# Restart ha-web-01
+incus start oc-node-01:ha-web-01
+sleep 3
+
+# All 3 backends serve again
+for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done
+```
+
+Actual output after restart:
+
+```
+<h1>ha-web-03 on oc-node-03</h1>
+<h1>ha-web-01 on oc-node-01</h1>
+<h1>ha-web-03 on oc-node-03</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+```
+
+All 3 backends recovered.
+
+### 6.6 Network Forward (Direct Backend Access)
+
+```bash
+incus network forward create oc-node-01:net-prod 192.168.103.202
+incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8080 10.10.10.2 80
+incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8081 10.10.10.3 80
+incus network forward port add oc-node-01:net-prod 192.168.103.202 tcp 8082 10.10.10.4 80
+```
+
+Test direct access:
+
+```bash
+curl -s http://192.168.103.202:8080  # ha-web-01
+curl -s http://192.168.103.202:8081  # ha-web-02
+curl -s http://192.168.103.202:8082  # ha-web-03
+```
+
+Actual output:
+
+```
+<h1>ha-web-01 on oc-node-01</h1>
+<h1>ha-web-02 on oc-node-02</h1>
+<h1>ha-web-03 on oc-node-03</h1>
+```
+
+### 6.7 Final Instance State
+
+```bash
+incus list oc-node-01:
+```
+
+Actual output:
+
+```
++-------------+---------+----------------------+-----------+------------+
+|    NAME     |  STATE  |         IPV4         |   TYPE    |  LOCATION  |
++-------------+---------+----------------------+-----------+------------+
+| ha-web-01   | RUNNING | 10.10.10.2 (eth0)    | CONTAINER | oc-node-01 |
+| ha-web-02   | RUNNING | 10.10.10.3 (eth0)    | CONTAINER | oc-node-02 |
+| ha-web-03   | RUNNING | 10.10.10.4 (eth0)    | CONTAINER | oc-node-03 |
+| ovn-central | RUNNING | 10.149.76.198 (eth0) | CONTAINER | oc-node-01 |
++-------------+---------+----------------------+-----------+------------+
+```
+
+---
+
+## Section 7: OC Management
+
+### 7.1 Inventory Sync
+
+OC inventory is **not real-time** — requires explicit `cluster resync`:
 
 ```bash
-# Force sync first
 operations-center provisioning cluster resync oc-cluster
-
-# Then query (use UUIDs for instance show, not names)
-operations-center inventory instance list
-operations-center inventory storage-pool list
-operations-center inventory network list
-operations-center inventory profile list
-operations-center inventory project list
-operations-center inventory image list
 ```
 
-**Tested**: after creating workloads via `incus launch`, OC inventory was
-empty until `cluster resync` was run. After resync, all instances, storage
-pools, networks, profiles, projects, and images appeared correctly.
-
-### Inventory sync latency
-
-- Creating workloads via `incus` CLI: not visible in OC until resync
-- After resync: ~5 seconds for inventory to populate
-- Live migration: location change not reflected until resync
-- Evacuation: instance relocation not reflected until resync
-- OC `inventory instance show <UUID>` shows: name, project, cluster, server, timestamp
-
-### Server management (tested)
+### 7.2 Inventory Browsing
 
 ```bash
-operations-center provisioning server list      # Lists all servers + OC itself
-operations-center provisioning server show <name>
-operations-center provisioning server resync <name>
-operations-center provisioning server system reboot <name>    # WARNING: see below
-operations-center provisioning server system poweroff <name>
+operations-center inventory instance list --cluster oc-cluster
 ```
 
-**WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system
-reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS
-nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot,
-all recover in ~50s with containers and data intact). The failure is
-**OC-specific**: the OC agent runs on every boot and can fail with:
-- "failed to parse update frequency - invalid crontab expression" (OC
-  pushed invalid update config)
-- "error from operations center: forbidden: failed creating server:
-  constraint violation" (OC trying to re-register an existing server)
-- Incus stuck at "starting application name=incus" (cluster peers are in
-  boot loops, so quorum cannot form)
+Actual output:
 
-These failures cause boot loops or hung services. Even Proxmox stop/start
-does not recover because the OC state on disk remains broken. The only
-fix is to destroy and redeploy the VM.
+```
++--------------------------------------+-------------+---------+------------+------------+
+|                 UUID                 |    Name     | Project |  Cluster   |   Server   |
++--------------------------------------+-------------+---------+------------+------------+
+| cf27717e-5f87-5a22-9e82-cc6616fd665d | ha-web-01   | default | oc-cluster | oc-node-01 |
+| 657447d3-7773-5005-af07-b62d9f789ffc | ha-web-02   | default | oc-cluster | oc-node-02 |
+| 7b79cab9-37f9-50d8-9d4f-20daafd3d932 | ha-web-03   | default | oc-cluster | oc-node-03 |
+| 17133a45-6318-5aa5-b6a8-fc5a5871f679 | ovn-central | default | oc-cluster | oc-node-01 |
++--------------------------------------+-------------+---------+------------+------------+
+```
 
-**Safe reboot**: Proxmox stop/start and guest reboot both work on
-standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on
-OC-managed nodes.
-The proper lifecycle for maintenance is:
+```bash
+operations-center inventory network list --cluster oc-cluster
+```
+
+Actual output:
+
+```
++--------------------------------------+----------+----------+---------+------------+
+|                 UUID                 |   Name   |   Type   | Project |  Cluster   |
++--------------------------------------+----------+----------+---------+------------+
+| 628df562-c980-5c27-8268-2ba69211eb7f | UPLINK   | physical | default | oc-cluster |
+| e35ecd83-25c7-54ae-8fd9-66bc241a002f | incusbr0 | bridge   | default | oc-cluster |
+| 9cc58dd0-9821-5649-807a-88e0cbce8f5c | meshbr0  | bridge   | default | oc-cluster |
+| 9c3ee7cb-6b7e-5ea9-bf91-e8d44f0f04da | net-prod | ovn      | default | oc-cluster |
++--------------------------------------+----------+----------+---------+------------+
+```
+
+```bash
+operations-center inventory network-load-balancer list
+operations-center inventory network-forward list
+operations-center inventory storage-pool list
+```
+
+All resources visible in OC inventory after resync.
+
+### 7.3 Server Management
+
+```bash
+operations-center provisioning server list
+```
+
+Actual output:
+
+```
++------------+-------------------+------------------------------+--------+-------------+
+|  Cluster   |       Name        |        Connection URL        | Status | Update Stat |
++------------+-------------------+------------------------------+--------+-------------+
+| oc-cluster | oc-node-01        | https://192.168.102.140:8443 | ready  | up to date  |
+| oc-cluster | oc-node-02        | https://192.168.102.141:8443 | ready  | up to date  |
+| oc-cluster | oc-node-03        | https://192.168.102.142:8443 | ready  | up to date  |
+|            | operations-center | https://[::1]:8443           | ready  | update pend |
++------------+-------------------+------------------------------+--------+-------------+
+```
+
+### 7.4 Update Management
+
+```bash
+operations-center provisioning update list
+operations-center provisioning update show <UUID>
+operations-center provisioning update refresh     # Check for new updates
+operations-center provisioning update assign-channels <UUID> --channel <name>
+```
+
+**Channel management**: updates auto-link to channels based on upstream
+`filter_expression`. Use `assign-channels` to explicitly control which
+updates appear in which channels.
+
+### 7.5 Web UI
+
+Navigate to `https://192.168.102.120:8443/ui/` in Firefox (with client.pfx
+imported). The web UI shows: servers, clusters, instances, networks, storage
+pools, load balancers, network forwards, and update status.
+
+---
+
+## Section 8: Cluster Lifecycle
+
+### 8.1 Evacuation & Restore
+
+```bash
+incus cluster evacuate oc-node-01:oc-node-01 --force
+# OC still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
+operations-center provisioning cluster resync oc-cluster
+# Inventory shows workloads moved to other nodes
+
+incus cluster restore oc-node-01:oc-node-01 --force
+```
+
+### 8.2 Node Failure Simulation
+
+Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s).
+Node auto-rejoins after Proxmox restart (~60s boot):
+
+```bash
+# OC does NOT detect node failures
+# provisioning server show still reports "Status: ready" for offline nodes
+```
+
+### 8.3 Reboot Considerations
+
+**WARNING**: `operations-center provisioning server system reboot` can break
+OC-managed nodes on Proxmox. The OC agent pushes config via the IncusOS REST
+API that gets persisted to `state.txt`. On reboot, invalid values crash the
+daemon. See CLAUDE.md for full analysis.
+
+**Safe maintenance procedure**:
 1. `incus cluster evacuate <remote>:<member> --force`
 2. Proxmox stop the VM
 3. Proxmox start the VM
 4. Wait for IncusOS to boot (~60-90s)
 5. `incus cluster restore <remote>:<member> --force`
 
-### IncusOS scrub_schedule fix via API (tested)
+---
 
-IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
-with an empty `scrub_schedule` field. The `registerJobs()` function calls
-`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
-is briefly reachable during each crash cycle (Incus starts before the scheduler).
+## Section 9: Cleanup
 
-**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):
+**Note**: this section documents the cleanup procedure but is NOT executed.
+The lab is left fully running with all workloads, OVN networking, and load
+balancer active for exploration via OC web UI and CLI.
 
 ```bash
-# Check current storage config
-curl -sk --cert ~/.config/incus/client.crt \
-     --key ~/.config/incus/client.key \
-     https://<NODE_IP>:8443/os/1.0/system/storage
+# Delete workloads
+incus delete oc-node-01:ha-web-01 oc-node-01:ha-web-02 oc-node-01:ha-web-03 --force
 
-# If scrub_schedule is empty, fix it
-curl -sk --cert ~/.config/incus/client.crt \
-     --key ~/.config/incus/client.key \
-     -X PUT -H "Content-Type: application/json" \
-     -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
-     https://<NODE_IP>:8443/os/1.0/system/storage
-```
+# Delete load balancer and forward
+incus network load-balancer delete oc-node-01:net-prod 192.168.103.201
+incus network forward delete oc-node-01:net-prod 192.168.103.202
 
-**Key details**:
-- The `/os/` prefix is required — this proxies to the IncusOS daemon API
-- PATCH returns 501 (not implemented); use PUT instead
-- `incus admin os system storage edit` fails because it reads the current
-  (corrupt) config and validates it before allowing edits
-- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
-  fixes this on every deployed node (safe — returns early if already set)
-- Filed upstream as IncusOS issue #843
+# Delete OVN network
+incus network delete oc-node-01:net-prod
 
-### Node failure and recovery (tested)
+# Delete OVN control plane
+incus delete oc-node-01:ovn-central --force
 
-Proxmox hard-stop simulates node crash. Incus cluster detects failure via
-heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):
+# Delete UPLINK
+incus network delete oc-node-01:UPLINK
 
-```bash
-# Simulate crash: Proxmox API stop
-curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
+# Disable OVN services on all nodes
+for remote in oc-node-01 oc-node-02 oc-node-03; do
+    incus query ${remote}:/os/1.0/services/ovn --request PUT --data '{
+        "config": {"enabled": false}, "state": {}
+    }'
+done
 
-# After ~40s: incus cluster list shows OFFLINE with heartbeat info
+# Remove OVN chassis role
+for node in oc-node-01 oc-node-02 oc-node-03; do
+    incus cluster role remove oc-node-01:${node} ovn-chassis
+done
 
-# Restart: Proxmox API start
-curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
+# Destroy infrastructure
+./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-nodes.yaml
+./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-deploy.yaml
 
-# After ~60s boot: node auto-rejoins cluster (ONLINE)
-# Containers on the node auto-start after recovery
-```
-
-**OC does not detect node failures**: `provisioning server show` still shows
-`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.
-
-**OC does not track Incus cluster member state**: the server list always
-shows `ready` status even when the Incus cluster reports `EVACUATED`.
-
-### Server edit commands
-
-Network and storage have `edit` subcommands (no `show`):
-
-```bash
-operations-center provisioning server system network edit <name>
-operations-center provisioning server system storage edit <name>
-```
-
-### Update management (tested)
-
-```bash
-operations-center provisioning update list       # Shows all available IncusOS versions
-operations-center provisioning update show <UUID> # Full file list with sizes
-operations-center provisioning update refresh     # Check for new updates
-```
-
-Updates show comprehensive file lists including sysexts: incus, operations-center,
-gpu-support, incus-ceph, incus-linstor, migration-manager, debug.
-
-### System configuration (tested)
-
-```bash
-operations-center system settings show   # log_level
-operations-center system security show   # OIDC, OpenFGA, ACME, trusted certs
-operations-center system updates show    # Update source, filter expressions, signature CA
-```
-
-Note: `system certificate` only has `set` (no `show`). `system network` has `show` and `edit`.
-
-### Admin / IncusOS management (tested)
-
-```bash
-operations-center admin os show           # hostname, os_version, uptime
-operations-center admin os application list
-operations-center admin os application show <name>
-operations-center admin os service list   # iscsi, lvm, multipath, nvme, ovn, tailscale, usbip
-operations-center admin os debug log
-```
-
-### Cluster operations (tested)
-
-```bash
-operations-center provisioning cluster resync <name>
-operations-center provisioning cluster rename <name> <new-name>
-operations-center provisioning cluster update --connection-url <url>
-operations-center provisioning cluster update-certificate <name> <cert> <key>
-operations-center provisioning cluster artifact list <name>
-operations-center provisioning cluster factory-reset <name> <token-uuid>
-```
-
-### Cluster templates
-
-```bash
-operations-center provisioning cluster-template add <name> --description "..."
-operations-center provisioning cluster-template list
-operations-center provisioning cluster-template show <name>
-operations-center provisioning cluster-template remove <name>
+# Remove OC remote
+operations-center remote remove oc-lab
 ```
 
 ---
 
-## Part 6: Cluster Lifecycle (Tested)
+## Section 10: Verification Checklist
 
-### Live migration visibility through OC (tested)
-
-```bash
-# Prepare VM for live migration
-incus stop oc-node-01:test-vm-01
-incus config set oc-node-01:test-vm-01 migration.stateful=true limits.cpu=0-1
-incus config device add oc-node-01:test-vm-01 root disk path=/ pool=local size.state=2GiB
-incus start oc-node-01:test-vm-01
-
-# Migrate
-incus move oc-node-01:test-vm-01 --target oc-node-03
-
-# OC does NOT auto-update -- resync required
-operations-center provisioning cluster resync oc-cluster
-operations-center inventory instance list  # now shows new location
-```
-
-**Tested**: ~141 MB/s migration speed. OC inventory reflected the location
-change only after explicit `cluster resync`.
-
-### Cluster evacuation through OC (tested)
-
-```bash
-incus cluster evacuate oc-node-01:oc-node-01 --force
-# OC server list still shows oc-node-01 as "ready" (doesn't track EVACUATED state)
-operations-center provisioning cluster resync oc-cluster
-operations-center inventory instance list  # shows workloads moved to other nodes
-
-incus cluster restore oc-node-01:oc-node-01 --force
-```
-
-### Node replacement lifecycle (tested)
-
-Full procedure tested after OC reboot broke a node:
-
-```bash
-# 1. Force-remove dead node from Incus cluster
-printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
-
-# 2. Destroy broken VM via Proxmox (stop + delete)
-
-# 3. Redeploy via incusos-proxmox --iso (single-node config)
-#    Use apply_defaults: false for OC-managed nodes
-./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
-
-# 4. New node auto-registers with OC (same hostname, new IP)
-#    NOTE: OC shows stale entry for old node-03. Stale entry can't be removed
-#    while OC thinks it's in a cluster. Resync doesn't fix stale entries.
-
-# 5. Manual cluster join (since node was force-removed)
-incus config set oc-node-03: core.https_address <NEW_IP>:8443
-# With apply_defaults: false, no cleanup needed (no pool/network to delete)
-# Generate token and join
-incus cluster add oc-node-01:oc-node-03
-printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
-# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
-
-# 6. Fix remote after cert change
-incus remote remove oc-node-03
-incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
-```
-
-### Node failure simulation (tested)
-
-Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):
-
-```bash
-# Simulate crash
-curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
-# incus cluster list shows OFFLINE after ~40s
-
-# Restart
-curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
-# Node auto-rejoins after ~60s boot. Containers auto-start.
-# OC still shows "ready" (does not detect node failures)
-```
-
-### Factory reset (not tested)
-
-```bash
-# WARNING: destructive
-operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
-```
+| # | Check | Status |
+|---|-------|--------|
+| 1 | OC server deployed and reachable | PASS |
+| 2 | OC remote configured | PASS |
+| 3 | Updates downloaded and ready | PASS |
+| 4 | 3 nodes provisioned from OC ISO | PASS |
+| 5 | All nodes auto-registered with OC | PASS |
+| 6 | All nodes "up to date" (not "update pending") | PASS |
+| 7 | No crontab bug (force_reboot not used) | PASS |
+| 8 | Cluster formed via OC `provisioning cluster add` | PASS |
+| 9 | All 3 nodes ONLINE + Fully operational | PASS |
+| 10 | OVN cross-node connectivity (sub-ms) | PASS |
+| 11 | 3 nginx instances on 3 nodes | PASS |
+| 12 | LB distributes across all backends | PASS |
+| 13 | HA: LB serves with one backend down | PASS |
+| 14 | Network forward: direct backend access | PASS |
+| 15 | OC inventory shows all resources | PASS |
+| 16 | OC web UI accessible | PASS |
 
 ---
 
-## OC CLI Command Reference
+## Section 11: Quick Reference
 
-### Remote management
-
-| Command | Description |
-|---------|-------------|
-| `operations-center remote add <name> <url> --auth-type tls` | Add OC remote |
-| `operations-center remote switch <name>` | Set default remote |
-| `operations-center remote list` | List configured remotes |
-| `operations-center remote remove <name>` | Remove a remote |
-
-### Provisioning tokens
-
-| Command | Description |
-|---------|-------------|
-| `provisioning token add --description "..." --uses N --lifetime 168h` | Create token |
-| `provisioning token list` | List tokens |
-| `provisioning token show <UUID>` | Token details |
-| `provisioning token remove <UUID>` | Delete token |
-| `provisioning token seed add <UUID> <name> <file> --description "..."` | Attach named seed |
-| `provisioning token seed list <UUID>` | List token seeds |
-| `provisioning token seed show <UUID> <name>` | Show seed details |
-| `provisioning token seed get-image <UUID> <seed> <output> --type iso --architecture x86_64` | ISO from seed |
-| `provisioning token get-image <UUID> <output> [preseed] --type iso --architecture x86_64` | ISO from inline preseed |
-
-### Provisioning servers
-
-| Command | Description |
-|---------|-------------|
-| `provisioning server list` | List registered servers (includes OC itself) |
-| `provisioning server show <name>` | Server details + certificate |
-| `provisioning server rename <old> <new>` | Rename server |
-| `provisioning server resync <name>` | Resync single server |
-| `provisioning server remove <name>` | Remove (fails if in cluster) |
-| `provisioning server edit <name>` | Edit server (interactive) |
-| `provisioning server system reboot <name>` | Guest reboot (DANGEROUS on Proxmox) |
-| `provisioning server system poweroff <name>` | Guest poweroff |
-| `provisioning server system network edit <name>` | Edit network config |
-| `provisioning server system storage edit <name>` | Edit storage config |
-
-### Provisioning clusters
-
-| Command | Description |
-|---------|-------------|
-| `provisioning cluster add <name> <url> --server-names ... --server-type incus -a <seed>` | Form cluster |
-| `provisioning cluster list` | List clusters |
-| `provisioning cluster show <name>` | Cluster details + certificate |
-| `provisioning cluster resync <name>` | Force inventory resync |
-| `provisioning cluster rename <name> <new-name>` | Rename cluster |
-| `provisioning cluster update <name> --connection-url <url>` | Update cluster URL |
-| `provisioning cluster update-certificate <name> <cert> <key>` | Update cluster cert |
-| `provisioning cluster remove <name>` | Delete cluster |
-| `provisioning cluster factory-reset <name> <token-uuid>` | Factory reset |
-| `provisioning cluster artifact list <name>` | List artifacts |
-| `provisioning cluster artifact archive <name> <type> <output>` | Download artifact |
-
-### Cluster templates
-
-| Command | Description |
-|---------|-------------|
-| `provisioning cluster-template add <name> --description "..."` | Create template |
-| `provisioning cluster-template list` | List templates |
-| `provisioning cluster-template show <name>` | Show template |
-| `provisioning cluster-template remove <name>` | Delete template |
-
-### Inventory
-
-| Command | Description |
-|---------|-------------|
-| `inventory instance list` | List all instances |
-| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
-| `inventory storage-pool list` | List storage pools |
-| `inventory storage-volume list` | List storage volumes |
-| `inventory storage-bucket list` | List storage buckets |
-| `inventory network list` | List networks |
-| `inventory network-acl list` | List network ACLs |
-| `inventory network-address-set list` | List network address sets |
-| `inventory network-forward list` | List network forwards |
-| `inventory network-integration list` | List network integrations |
-| `inventory network-load-balancer list` | List network load balancers |
-| `inventory network-peer list` | List network peers |
-| `inventory network-zone list` | List network zones |
-| `inventory profile list` | List profiles |
-| `inventory project list` | List projects |
-| `inventory image list` | List images |
-| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |
-
-### Updates
-
-| Command | Description |
-|---------|-------------|
-| `provisioning update list` | List available IncusOS versions |
-| `provisioning update show <UUID>` | Version details + file list |
-| `provisioning update refresh` | Check for new updates |
-
-### System
-
-| Command | Description |
-|---------|-------------|
-| `system settings show` | System settings (log_level) |
-| `system security show` | OIDC, OpenFGA, ACME, trusted certs |
-| `system updates show` | Update source, filter, signature CA |
-| `system network show` | Network config |
-| `system certificate set <cert> <key>` | Set server certificate |
-
-### Admin
+### OC Provisioning Workflow
+
+```
+1. Deploy OC server (incusos-proxmox)
+2. Add OC remote (operations-center remote add)
+3. Wait for updates (provisioning update list)
+4. Create token (provisioning token add)
+5. Create token seed (provisioning token seed add) -- NO force_reboot
+6. Generate ISO from OLDER channel (token seed get-image --channel old-stable)
+7. Deploy nodes (incusos-proxmox --iso)
+8. Wait for auto-registration + update delivery
+9. Form cluster (provisioning cluster add)
+10. Set up OVN networking (OVN container + services + UPLINK + net-prod)
+```
+
+### Key OC CLI Commands
 
 | Command | Description |
 |---------|-------------|
+| `remote add <name> <url> --auth-type tls` | Add OC remote |
+| `remote switch <name>` | Set default remote |
 | `admin os show` | IncusOS hostname, version, uptime |
-| `admin os application list` | List applications |
-| `admin os application show <name>` | Application state |
-| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
-| `admin os service show <name>` | Service config |
-| `admin os debug log` | Debug logs |
-| `admin os debug processes` | System process list |
-| `admin os debug secureboot` | Secure Boot debug info |
+| `provisioning token add --uses N` | Create provisioning token |
+| `provisioning token seed add <UUID> <name> <file>` | Attach seed to token |
+| `provisioning token seed get-image <UUID> <seed> <file> --channel <ch>` | Generate ISO |
+| `provisioning update list` | List available updates |
+| `provisioning update refresh` | Check for new updates |
+| `provisioning update assign-channels <UUID> --channel <name>` | Assign update to channel |
+| `provisioning server list` | List registered servers |
+| `provisioning server resync <name>` | Resync server state |
+| `provisioning cluster add <name> <url> --server-names ... -a <seed>` | Form cluster |
+| `provisioning cluster resync <name>` | Force inventory resync |
+| `provisioning cluster list` | List clusters |
+| `inventory instance list --cluster <name>` | List instances |
+| `inventory network list --cluster <name>` | List networks |
+| `inventory network-load-balancer list` | List load balancers |
+| `inventory network-forward list` | List network forwards |
+| `inventory storage-pool list` | List storage pools |
 
-All commands prefixed with `operations-center`.
+### Incus Cluster Commands (via remote)
+
+| Command | Description |
+|---------|-------------|
+| `incus cluster list <remote>:` | List cluster members |
+| `incus cluster role add <remote>:<member> ovn-chassis` | Add OVN role |
+| `incus cluster evacuate <remote>:<member> --force` | Evacuate member |
+| `incus cluster restore <remote>:<member> --force` | Restore member |
+| `incus network create <remote>:<net> --type=physical --target=<member> parent=mgmt` | Per-target network |
+| `incus network create <remote>:<net> --type=ovn network=UPLINK` | OVN network |
+| `incus network load-balancer create <remote>:<net> <ip>` | Create LB |
+| `incus network forward create <remote>:<net> <ip>` | Create forward |
 
 ---
 
-## REST API Reference
+## Section 12: Troubleshooting
 
-OC listens on port **8443** (same as Incus on IncusOS). All endpoints require
-TLS client certificate authentication.
-
-### Common curl pattern
-
-```bash
-curl -sk --cert ~/.config/incus/client.crt \
-     --key ~/.config/incus/client.key \
-     https://<OC_IP>:8443/1.0/<endpoint>
-```
-
-### Key endpoints
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| GET | `/1.0` | Server info |
-| GET | `/1.0/provisioning/updates` | List update packages |
-| GET | `/1.0/provisioning/tokens` | List tokens |
-| POST | `/1.0/provisioning/tokens` | Create token |
-| GET | `/1.0/provisioning/servers` | List registered servers |
-| GET | `/1.0/provisioning/clusters` | List clusters |
-| POST | `/1.0/provisioning/clusters` | Form cluster |
-| GET | `/1.0/inventory/instances` | List all instances |
-| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
-| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |
-
-### IncusOS REST API (via Incus proxy)
-
-IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
-prefix on port 8443:
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
-| PUT | `/os/1.0/system/storage` | Update storage config |
-| GET | `/os/1.0` | IncusOS system info |
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| `needs_update: true` blocks clustering | Nodes deployed from ISO matching latest version | Deploy from older ISO (`--channel old-stable`), let OC push the update |
+| Terraform "Certificate already in trust store" | Cert in both SEED_DATA and app-seed-config | Use empty `{}` for app-seed-config when cert is in SEED_DATA |
+| Terraform "Network/Pool not in pending state" | `apply_defaults: true` pre-created resources | Use `apply_defaults: false` (recommended) |
+| TLS cert error after clustering | Cluster cert regenerated | `incus remote remove/add` with `--accept-certificate` |
+| `get-image` fails | No updates in `ready` state | Wait ~5 min for OC to download; `provisioning update list` |
+| Node doesn't self-register | Bad ISO or token exhausted | Verify ISO from `get-image`; check `token list` |
+| Token seed fields all `{}` | Flat YAML format | Use structured: `install:` section key |
+| OC web UI 403 | Client cert not imported | Import client.pfx into browser |
+| OC shows "ready" for offline node | OC doesn't detect failures | Check via `incus cluster list` instead |
+| OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot |
+| UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide |
+| Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs |
+| Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` |
 
 ---
 
-## Proxmox-Specific Considerations
-
-### force_reboot is required
-
-IncusOS does not auto-halt after installation. Without `force_reboot: true`
-in the seed (or pre-seed for OC-provisioned ISOs), the installer waits at
-"please remove installation media" indefinitely.
-
-Options for OC-provisioned ISOs on Proxmox:
-1. **Pre-seed file** passed to `get-image`: include `force_reboot: true`
-2. **Fallback SEED_DATA ISO** on ide3: minimal `install.yaml` with
-   `force_reboot: true` only
-
-### VM creation settings
-
-OC-provisioned nodes use the same Proxmox VM settings as standard IncusOS
-deployments (see `incusos-proxmox`):
-
-- `bios=ovmf`, `machine=q35` -- UEFI boot required
-- `efidisk0`: `pre-enrolled-keys=0` -- IncusOS enrolls its own Secure Boot keys
-- `tpmstate0`: `version=v2.0` -- required for disk encryption
-- `cpu=host` -- needed for x86_64_v3 instruction set requirement
-- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
-- `balloon=0` -- IncusOS manages memory internally
-- Minimum 50 GiB disk, minimum 4096 MiB RAM
-
-### Install media removal
-
-After installation completes, you **must** stop the VM and delete ide2 (and
-ide3 if used) before booting from disk. IncusOS refuses to start if install
-media is still attached.
-
-### IP detection
-
-IncusOS is immutable and has no QEMU guest agent. Use ARP-based lookup to
-find node IPs:
-
-```bash
-# Get MAC from Proxmox VM config
-MAC=$(curl -sk "https://<HOST>:8006/api2/json/nodes/pve/qemu/<VMID>/config" \
-    -H "Authorization: PVEAPIToken=$TOKEN" | python3 -c "
-import sys, json
-d = json.load(sys.stdin)['data']
-print(d.get('net0', '').split('=')[1].split(',')[0])")
-
-# ARP lookup
-ip neigh flush dev <bridge>
-ping -b -c 3 <broadcast_ip> 2>/dev/null
-ip neigh show | grep -i "$MAC" | awk '{print $1}'
-```
-
----
-
-## Known Limitations (Tested Findings)
-
-### Confirmed limitations (v0.3.0)
+## Section 13: Known Limitations (v0.3.0)
 
 - **No brownfield adoption**: nodes must boot from OC-provisioned ISO
 - **No real-time inventory sync**: requires explicit `cluster resync`
-- **No cluster member state tracking**: OC always shows `ready`, even for
-  EVACUATED or OFFLINE nodes
-- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
-  create stale entries that `resync` doesn't fix
-- **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause
-  boot loops or hung services (not a dqlite or TPM issue -- standalone nodes
-  reboot fine)
-- **Token seed format quirk**: requires structured YAML with section keys,
-  not flat format
-- **Expired tokens not auto-cleaned**: remain in list after expiry
-- **No cluster resize**: cannot add/remove members from existing clusters
-
-### Design constraints
-
-- OC runs on IncusOS (same immutable OS as Incus nodes)
-- Uses port 8443 for both API and web UI
-- Requires at least one trusted client certificate in seed
-- Provisioning tokens have configurable use count and lifetime
-- OC adds `meshbr0` network to clusters (not present in manual clustering)
-- REST API endpoint: `/1.0` (same pattern as Incus)
-- Auth methods: TLS client cert or OIDC
-
----
-
-## Troubleshooting (Tested)
-
-| Symptom | Likely cause | Fix |
-|---------|-------------|-----|
-| `get-image` fails | No updates in `ready` state | Wait ~3 min for OC to download packages; check `provisioning update list` |
-| Node doesn't self-register | ISO not OC-provisioned, or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
-| Node stuck at "remove media" | Missing `force_reboot` in seed | Use token seed with `install: { force_reboot: true }` or SEED_DATA ISO via `incusos-proxmox` |
-| Token seed fields all `{}` | Flat YAML format | Use structured format with section keys: `install:`, `applications:`, etc. |
-| OC web UI returns 403 | Client cert not imported | Import client.pfx into browser certificates |
-| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
-| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
-| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
-| OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. |
-| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
-| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
-| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
-| VMID collision (500/403 on create) | Pool-scoped token can't see VMs outside pool | Use high `start_vmid` (400+) to avoid collisions |
-| `cluster resync` socket error | Transient OC internal issue | Retry after a few seconds -- usually succeeds on second attempt |
-| Proxmox API empty response | Bash `!` in token username | Use `set +H` or store token in variable; avoid bash history expansion |
-| `server remove` fails | Server is part of a cluster | Must remove cluster first, or force-remove member from Incus cluster |
-| `cluster join` extra prompt | OC adds `meshbr0` network | Add extra `\n` for `tunnel.mesh.interface` prompt in automated joins |
-
----
-
-## Comparison: Manual vs OC Cluster Workflow (Tested)
-
-### Manual (incusos-proxmox + incus CLI)
-
-1. Write YAML config with VM specs
-2. `incusos-proxmox --yes config.yaml` creates VMs with SEED_DATA
-3. Manually set `core.https_address` on each node
-4. Delete storage pool/network on joining nodes
-5. `incus cluster enable` + `incus cluster join` per node
-6. Re-add remotes after certificate changes
-7. ~30 minutes for 3-node cluster
-
-### Hybrid (incusos-proxmox --iso + OC)
-
-1. Deploy OC server via `incusos-proxmox --yes lab-oc-deploy.yaml`
-2. Create provisioning token + token seed in OC
-3. Download OC-provisioned ISO via `token seed get-image`
-4. `incusos-proxmox --iso IncusOS-oc.iso --yes lab-oc-nodes.yaml`
-5. Nodes auto-register with OC (~30s each)
-6. `operations-center provisioning cluster add oc-cluster ...`
-7. ~20 minutes for 3-node cluster (including OC deploy)
-
-### What OC automates that we do manually
-
-| Step | Manual | OC |
-|------|--------|-----|
-| Set `core.https_address` | Per-node via `incus config set` | Automatic during cluster add |
-| Storage pool/network cleanup for join | Manual delete per joining node | Automatic (handles pending state) |
-| Cluster enable + join | Multi-step per node | Single `cluster add` command |
-| Client cert injection | Via SEED_DATA | Via `--application-seed-config` |
-| `meshbr0` network | Not created | Auto-created by OC clustering |
-| TLS cert re-add | Manual remote remove/add | Not needed (OC handles internally) |
-
-**Trade-offs**:
-- Manual: full control, no extra server, works with any Incus version
-- OC hybrid: auto-registration, single-command clustering, centralized
-  inventory/updates, but requires OC server + OC-provisioned ISOs
-- OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing
-  loops -- standalone reboot is safe)
-- OC inventory requires manual resync (not real-time)
-- Out-of-band cluster changes (via `incus` CLI) create stale OC state
-
----
-
-## Open Research Items
-
-- **OC reboot failure mechanism** (resolved): guest reboot is safe on
-  standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
-  in ~50s). The failure is OC-specific: the OC agent pushes config via the
-  IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
-  values (e.g., cron expression where Go duration is expected) crash the
-  daemon. Root cause is NOT dqlite corruption or TPM issues.
-
-- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
-  is now the tested and recommended approach. OC's Terraform handles storage pool,
-  network, and cert creation cleanly when nodes start without defaults.
-  `apply_defaults: true` also works but produces non-fatal Terraform errors.
-
-- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
-  ~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
-  likely a race condition between REST API startup and `registerJobs()`, or a
-  zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
-  with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
-  auto-heals this proactively on every deploy.
-
-- **Stale server entries after out-of-band changes**: when cluster members are
-  removed via `incus cluster remove` (not through OC), stale entries persist in
-  OC's server list. `cluster resync` doesn't fix them. The entry can't be
-  removed while OC considers it part of a cluster.
-
-- **Inventory sync is manual**: OC inventory requires explicit `cluster resync`
-  to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
-  This applies to instance creation, migration, evacuation, and deletion.
-
-- **Node failure detection**: OC does not detect node failures. `provisioning
-  server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
-  Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.
+- **No cluster member state tracking**: OC always shows `ready` for all nodes
+- **Stale server entries**: out-of-band changes create stale entries
+- **needs_update tracking**: based on OC's delivery pipeline, not version comparison
+- **OC reboot breaks Proxmox VMs**: OC agent boot failures on restart
+- **No cluster resize**: cannot add/remove members from existing OC clusters
+- **Token seeds require structured YAML**: section keys, not flat format
+- **OVN LB has no health checks**: connection hashing distributes to dead backends
 
 ---
 
@@ -1001,4 +1044,6 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
 - [Operations Center releases](https://github.com/FuturFusion/operations-center/releases)
 - [IncusOS GitHub](https://github.com/lxc/incus-os)
 - [Incus documentation](https://linuxcontainers.org/incus/docs/main/)
-- [Clustering guide](clustering-guide.md) (manual cluster formation reference)
+- [Clustering guide](clustering-guide.md) -- manual cluster formation
+- [Networking guide](networking-guide.md) -- OVN overlay tutorial
+- [Production lab guide](production-lab-guide.md) -- manual cluster + OVN + HA