maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Validate OC guide: teardown + rebuild with fresh output 

Full teardown and rebuild of the OC-managed cluster lab to validate
the guide works end-to-end with zero manual interventions. Key fixes:

- Add missing `provisioning channel add old-stable` step (was causing
  "Not found" error on `assign-channels`)
- Fix command paths: `examples/` → `incusos/examples/` (repo-relative)
- Fix version numbers: Incus 6.12 → 6.21, OC output format
- Fix update list table format (Severity not Sev, add Published At note)
- Fix cluster list: remove premature ovn-chassis roles at Section 4.4
- Add WARNING line to `admin os show` output
- Add TLS cert acceptance note for `remote add`
- Add cleanup --yes flag to teardown commands
- Add test container cleanup command in Section 5.7
- Note variable-per-deployment values (incusbr0 subnet, token UUIDs)
- Add `provisioning channel add` to Quick Reference and CLI table
- Add channel creation to troubleshooting table
- Use consistent <token-UUID> placeholder throughout

All 16 verification checklist items PASS. Lab left running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-23 12:23:49 +01:00
parent b9f1894681
commit 896731a7ed
1 changed files with 128 additions and 88 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

216
notes/operations-center-guide.md
View File

@ -81,11 +81,11 @@ build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).
```bash ```bash
# Operations Center CLI # Operations Center CLI
operations-center --version operations-center --version
# operations-center version 0.3.0 # 0.3.0
# Incus client # Incus client
incus version incus version
# Client version: 6.12 # Client version: 6.21
# incusos-proxmox (from this repo) # incusos-proxmox (from this repo)
./incusos/incusos-proxmox --doctor ./incusos/incusos-proxmox --doctor
@ -137,7 +137,7 @@ vms:
### Deploy ### Deploy
```bash ```bash
./incusos/incusos-proxmox --yes examples/lab-oc-deploy.yaml ./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
``` ```
Actual output (key lines): Actual output (key lines):
@ -150,6 +150,7 @@ Actual output (key lines):
### Set Up OC CLI Remote ### Set Up OC CLI Remote
```bash ```bash
# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab operations-center remote switch oc-lab
``` ```
@ -160,15 +161,17 @@ operations-center remote switch oc-lab
operations-center admin os show operations-center admin os show
``` ```
Actual output: Actual output (uptime will vary):
``` ```
WARNING: The IncusOS API and configuration is subject to change
environment: environment:
hostname: oc-server hostname: oc-server
os_name: IncusOS os_name: IncusOS
os_version: "202602230420" os_version: "202602230420"
os_version_next: "" os_version_next: ""
uptime: 4956 uptime: 63
``` ```
### Wait for Updates ### Wait for Updates
@ -181,18 +184,23 @@ reach `ready` state before ISOs can be generated:
operations-center provisioning update list operations-center provisioning update list
``` ```
Actual output (after ~5 minutes): Actual output (after ~8 minutes; UUIDs are stable across deployments):
``` ```
+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ +--------------------------------------+---------------------+----------+--------------+----------+--------+
| UUID | Origin | Channels | Upstream Channels | Version | Sev | Status | | UUID | Origin | Channels | Version | Severity | Status |
+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ +--------------------------------------+---------------------+----------+--------------+----------+--------+
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | stable, testing | 202602200553 | none | ready | | 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready |
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | stable, testing | 202602210344 | none | ready | | 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready |
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | stable, testing | 202602230420 | none | ready | | c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready |
+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ +--------------------------------------+---------------------+----------+--------------+----------+--------+
``` ```
The table also includes `Upstream Channels` and `Published At` columns (omitted
for width). All 3 updates may not be ready at the same time -- at least one
`ready` is sufficient to proceed. The latest version (202602230420) typically
reaches `ready` first.
--- ---
## Section 3: Provision Nodes ## Section 3: Provision Nodes
@ -204,16 +212,18 @@ operations-center provisioning token add --uses 5 --description "OC-managed clus
operations-center provisioning token list operations-center provisioning token list
``` ```
Actual output: Actual output (UUID changes every run):
``` ```
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+ +--------------------------------------+----------------+-------------------------------+---------+-----------------------+
| UUID | Uses Remaining | Expire At | Channel | Description | | UUID | Uses Remaining | Expire At | Channel | Description |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+ +--------------------------------------+----------------+-------------------------------+---------+-----------------------+
| 2ba12420-fee5-4d55-9f2f-40809e9ad5de | 5 | 2026-03-25 10:29:39 +0100 CET | stable | OC-managed cluster v2 | | <token-UUID> | 5 | <30 days from now> | stable | OC-managed cluster v2 |
+--------------------------------------+----------------+-------------------------------+---------+-----------------------+ +--------------------------------------+----------------+-------------------------------+---------+-----------------------+
``` ```
Save the `<token-UUID>` -- you'll need it for the next steps.
### 3.2 Create Token Seed (No force_reboot) ### 3.2 Create Token Seed (No force_reboot)
**Critical**: the token seed must NOT include `force_reboot`. On Proxmox, **Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
@ -233,7 +243,7 @@ format (`version: "1"` at root) maps fields to empty `{}` and they don't get
assigned to any section. assigned to any section.
```bash ```bash
operations-center provisioning token seed add <UUID> proxmox-preseed \ operations-center provisioning token seed add <token-UUID> proxmox-preseed \
/tmp/oc-preseed.yaml --description "No force_reboot for Proxmox" /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
``` ```
@ -245,11 +255,16 @@ was never delivered through OC's update pipeline. The fix: generate the ISO
from an older channel so OC can push the real update after deployment. from an older channel so OC can push the real update after deployment.
```bash ```bash
# Assign the second-latest update to a separate channel # Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
--description "Older stable versions for initial provisioning"
# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable operations-center provisioning update assign-channels <older-UUID> --channel old-stable
# Generate ISO from the older channel # Generate ISO from the older channel
operations-center provisioning token seed get-image <UUID> proxmox-preseed \ operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
/tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
``` ```
@ -301,7 +316,7 @@ The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration
(hostname, static IP), install monitoring, and media cleanup. (hostname, static IP), install monitoring, and media cleanup.
```bash ```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-oc-nodes.yaml ./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml
``` ```
Actual output (key lines): Actual output (key lines):
@ -328,21 +343,25 @@ from 202602210344 to 202602230420 happens automatically:
operations-center provisioning server list operations-center provisioning server list
``` ```
Actual output: Actual output (key columns; full table includes Type, Channel, Certificate
Fingerprint, Public Connection URL, Last Updated, Last Seen):
``` ```
+------------+-------------------+------------------------------+--------+-------------+ +---------+-------------------+------------------------------+--------+----------------+
| Cluster | Name | Connection URL | Status | Update Stat | | Cluster | Name | Connection URL | Status | Update Status |
+------------+-------------------+------------------------------+--------+-------------+ +---------+-------------------+------------------------------+--------+----------------+
| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | | | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | | | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | | | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pend | | | operations-center | https://[::1]:8443 | ready | update pending |
+------------+-------------------+------------------------------+--------+-------------+ +---------+-------------------+------------------------------+--------+----------------+
``` ```
**Key**: all 3 nodes show **"up to date"** because OC delivered the **Key**: all 3 nodes show **"up to date"** because OC delivered the
202602230420 update through its pipeline. This is what unlocks clustering. 202602230420 update through its pipeline. This is what unlocks clustering.
Nodes may already be up to date by the time the last node finishes
deploying -- the update gets pushed while `incusos-proxmox` deploys
subsequent nodes sequentially.
### 3.7 Verify Scrub Schedules ### 3.7 Verify Scrub Schedules
@ -420,25 +439,23 @@ incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
incus cluster list oc-node-01: incus cluster list oc-node-01:
``` ```
Actual output: Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):
``` ```
+------------+------------------------------+-----------------+--------------+--------+-------------------+ +------------+------------------------------+-----------------+--------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE | | NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE |
+------------+------------------------------+-----------------+--------------+--------+-------------------+ +------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-01 | https://192.168.102.140:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | oc-node-01 | https://192.168.102.140:8443 | database-leader | x86_64 | ONLINE | Fully operational |
| | | database-leader | | | |
| | | database | | | | | | | database | | | |
+------------+------------------------------+-----------------+--------------+--------+-------------------+ +------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-02 | https://192.168.102.141:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational |
| | | database | | | |
+------------+------------------------------+-----------------+--------------+--------+-------------------+ +------------+------------------------------+-----------------+--------------+--------+-------------------+
| oc-node-03 | https://192.168.102.142:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | | oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational |
| | | database | | | |
+------------+------------------------------+-----------------+--------------+--------+-------------------+ +------------+------------------------------+-----------------+--------------+--------+-------------------+
``` ```
All 3 nodes ONLINE and Fully operational. All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added
later in Section 5.4.
### 4.5 Cluster Resources Created by OC ### 4.5 Cluster Resources Created by OC
@ -447,25 +464,26 @@ incus storage list oc-node-01:
incus network list oc-node-01: incus network list oc-node-01:
``` ```
Actual output: Actual output (incusbr0 subnet varies per deployment):
``` ```
+-------+--------+--------------------------------------+---------+---------+ +-------+--------+--------------------------------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE | | NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+-------+--------+--------------------------------------+---------+---------+ +-------+--------+--------------------------------------+---------+---------+
| local | zfs | Local storage pool (on system drive) | 8 | CREATED | | local | zfs | Local storage pool (on system drive) | 8 | CREATED |
+-------+--------+--------------------------------------+---------+---------+ +-------+--------+--------------------------------------+---------+---------+
+----------+--------+---------+----------------+------------------------------+---------+ +----------+--------+---------+-----------------+------------------------------+---------+
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
+----------+--------+---------+----------------+------------------------------+---------+ +----------+--------+---------+-----------------+------------------------------+---------+
| incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | | incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | | meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
+----------+--------+---------+----------------+------------------------------+---------+ +----------+--------+---------+-----------------+------------------------------+---------+
``` ```
OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
`meshbr0` (OC-specific mesh network for inter-node communication). `meshbr0` (OC-specific mesh network for inter-node communication). The
table also includes IPv6 and STATE columns.
--- ---
@ -574,13 +592,17 @@ incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
Actual output: Actual output:
``` ```
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.744 ms 64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.169 ms 64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.150 ms 64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms
``` ```
Sub-millisecond cross-node latency via Geneve tunnels. Clean up test Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
containers after verification. containers after verification:
```bash
incus delete oc-node-01:test-1 oc-node-01:test-2 --force
```
### 5.8 Final Network State ### 5.8 Final Network State
@ -595,12 +617,16 @@ Actual output:
| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
+----------+----------+---------+----------------+------------------------------+---------+ +----------+----------+---------+----------------+------------------------------+---------+
| UPLINK | physical | YES | | | 1 | | UPLINK | physical | YES | | | 1 |
| incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | | incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | | meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
| net-prod | ovn | YES | 10.10.10.1/24 | | 3 | | net-prod | ovn | YES | 10.10.10.1/24 | | 0 |
+----------+----------+---------+----------------+------------------------------+---------+ +----------+----------+---------+----------------+------------------------------+---------+
``` ```
The incusbr0 subnet is randomly assigned per deployment. The USED BY count
for net-prod is 0 at this point (test containers deleted); it increases as
workloads are added in Section 6.
--- ---
## Section 6: HA Nginx Workload ## Section 6: HA Nginx Workload
@ -651,18 +677,18 @@ incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \
for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done
``` ```
Actual output: Actual output (distribution varies per run due to connection-based hashing):
``` ```
<h1>ha-web-03 on oc-node-03</h1> <h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-02 on oc-node-02</h1> <h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-01 on oc-node-01</h1> <h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-02 on oc-node-02</h1> <h1>ha-web-02 on oc-node-02</h1>
``` ```
Traffic distributes across all 3 backends (connection-based hashing, not Traffic distributes across backends (connection-based hashing, not
round-robin). round-robin). Not all backends may appear in every 5-request sample.
### 6.5 HA Failover Test ### 6.5 HA Failover Test
@ -688,15 +714,15 @@ sleep 3
for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done
``` ```
Actual output after restart: Actual output after restart (all 3 backends serving again):
``` ```
<h1>ha-web-03 on oc-node-03</h1>
<h1>ha-web-01 on oc-node-01</h1> <h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-03 on oc-node-03</h1>
<h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-02 on oc-node-02</h1> <h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-02 on oc-node-02</h1> <h1>ha-web-02 on oc-node-02</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>
<h1>ha-web-01 on oc-node-01</h1>
``` ```
All 3 backends recovered. All 3 backends recovered.
@ -732,19 +758,22 @@ Actual output:
incus list oc-node-01: incus list oc-node-01:
``` ```
Actual output: Actual output (key columns; full table includes IPv6, SNAPSHOTS):
``` ```
+-------------+---------+----------------------+-----------+------------+ +-------------+---------+-----------------------+-----------+------------+
| NAME | STATE | IPV4 | TYPE | LOCATION | | NAME | STATE | IPV4 | TYPE | LOCATION |
+-------------+---------+----------------------+-----------+------------+ +-------------+---------+-----------------------+-----------+------------+
| ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 | | ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 |
| ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 | | ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 |
| ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 | | ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 |
| ovn-central | RUNNING | 10.149.76.198 (eth0) | CONTAINER | oc-node-01 | | ovn-central | RUNNING | 10.x.x.x (eth0) | CONTAINER | oc-node-01 |
+-------------+---------+----------------------+-----------+------------+ +-------------+---------+-----------------------+-----------+------------+
``` ```
The ovn-central container is on incusbr0 (random subnet per deployment).
The ha-web containers are on net-prod (10.10.10.0/24, fixed).
--- ---
## Section 7: OC Management ## Section 7: OC Management
@ -763,7 +792,7 @@ operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list --cluster oc-cluster operations-center inventory instance list --cluster oc-cluster
``` ```
Actual output: Actual output (UUIDs are deterministic based on name/project/cluster):
``` ```
+--------------------------------------+-------------+---------+------------+------------+ +--------------------------------------+-------------+---------+------------+------------+
@ -776,6 +805,8 @@ Actual output:
+--------------------------------------+-------------+---------+------------+------------+ +--------------------------------------+-------------+---------+------------+------------+
``` ```
The full table also includes a `Last Updated` column.
```bash ```bash
operations-center inventory network list --cluster oc-cluster operations-center inventory network list --cluster oc-cluster
``` ```
@ -793,6 +824,8 @@ Actual output:
+--------------------------------------+----------+----------+---------+------------+ +--------------------------------------+----------+----------+---------+------------+
``` ```
The full table also includes a `Last Updated` column.
```bash ```bash
operations-center inventory network-load-balancer list operations-center inventory network-load-balancer list
operations-center inventory network-forward list operations-center inventory network-forward list
@ -807,19 +840,22 @@ All resources visible in OC inventory after resync.
operations-center provisioning server list operations-center provisioning server list
``` ```
Actual output: Actual output (key columns):
``` ```
+------------+-------------------+------------------------------+--------+-------------+ +------------+-------------------+------------------------------+--------+----------------+
| Cluster | Name | Connection URL | Status | Update Stat | | Cluster | Name | Connection URL | Status | Update Status |
+------------+-------------------+------------------------------+--------+-------------+ +------------+-------------------+------------------------------+--------+----------------+
| oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | | oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | | oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | | oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pend | | | operations-center | https://[::1]:8443 | ready | update pending |
+------------+-------------------+------------------------------+--------+-------------+ +------------+-------------------+------------------------------+--------+----------------+
``` ```
Nodes now show their cluster membership. The OC server itself always shows
`update pending` (it cannot update itself via its own pipeline).
### 7.4 Update Management ### 7.4 Update Management
```bash ```bash
@ -915,9 +951,9 @@ for node in oc-node-01 oc-node-02 oc-node-03; do
incus cluster role remove oc-node-01:${node} ovn-chassis incus cluster role remove oc-node-01:${node} ovn-chassis
done done
# Destroy infrastructure # Destroy infrastructure (--yes to skip confirmation prompts)
./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-nodes.yaml ./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-deploy.yaml ./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml
# Remove OC remote # Remove OC remote
operations-center remote remove oc-lab operations-center remote remove oc-lab
@ -955,14 +991,16 @@ operations-center remote remove oc-lab
``` ```
1. Deploy OC server (incusos-proxmox) 1. Deploy OC server (incusos-proxmox)
2. Add OC remote (operations-center remote add) 2. Add OC remote (operations-center remote add)
3. Wait for updates (provisioning update list) 3. Wait for updates (provisioning update list -- at least one "ready")
4. Create token (provisioning token add) 4. Create token (provisioning token add)
5. Create token seed (provisioning token seed add) -- NO force_reboot 5. Create token seed (provisioning token seed add) -- NO force_reboot
6. Generate ISO from OLDER channel (token seed get-image --channel old-stable) 6. Create old-stable channel (provisioning channel add old-stable)
7. Deploy nodes (incusos-proxmox --iso) 7. Assign older update to old-stable (provisioning update assign-channels)
8. Wait for auto-registration + update delivery 8. Generate ISO from OLDER channel (token seed get-image --channel old-stable)
9. Form cluster (provisioning cluster add) 9. Deploy nodes (incusos-proxmox --iso)
10. Set up OVN networking (OVN container + services + UPLINK + net-prod) 10. Wait for auto-registration + update delivery
11. Form cluster (provisioning cluster add)
12. Set up OVN networking (OVN container + services + UPLINK + net-prod)
``` ```
### Key OC CLI Commands ### Key OC CLI Commands
@ -975,6 +1013,7 @@ operations-center remote remove oc-lab
| `provisioning token add --uses N` | Create provisioning token | | `provisioning token add --uses N` | Create provisioning token |
| `provisioning token seed add <UUID> <name> <file>` | Attach seed to token | | `provisioning token seed add <UUID> <name> <file>` | Attach seed to token |
| `provisioning token seed get-image <UUID> <seed> <file> --channel <ch>` | Generate ISO | | `provisioning token seed get-image <UUID> <seed> <file> --channel <ch>` | Generate ISO |
| `provisioning channel add <name> --description <desc>` | Create provisioning channel |
| `provisioning update list` | List available updates | | `provisioning update list` | List available updates |
| `provisioning update refresh` | Check for new updates | | `provisioning update refresh` | Check for new updates |
| `provisioning update assign-channels <UUID> --channel <name>` | Assign update to channel | | `provisioning update assign-channels <UUID> --channel <name>` | Assign update to channel |
@ -1020,6 +1059,7 @@ operations-center remote remove oc-lab
| OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot | | OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot |
| UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide | | UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide |
| Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs | | Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs |
| `assign-channels` "Not found" | Channel doesn't exist yet | Run `provisioning channel add old-stable` first |
| Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` | | Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` |
--- ---