diff --git a/notes/operations-center-guide.md b/notes/operations-center-guide.md index aa4e335..d3a7cdd 100644 --- a/notes/operations-center-guide.md +++ b/notes/operations-center-guide.md @@ -81,11 +81,11 @@ build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel). ```bash # Operations Center CLI operations-center --version -# operations-center version 0.3.0 +# 0.3.0 # Incus client incus version -# Client version: 6.12 +# Client version: 6.21 # incusos-proxmox (from this repo) ./incusos/incusos-proxmox --doctor @@ -137,7 +137,7 @@ vms: ### Deploy ```bash -./incusos/incusos-proxmox --yes examples/lab-oc-deploy.yaml +./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml ``` Actual output (key lines): @@ -150,6 +150,7 @@ Actual output (key lines): ### Set Up OC CLI Remote ```bash +# Accept the TLS certificate when prompted operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls operations-center remote switch oc-lab ``` @@ -160,15 +161,17 @@ operations-center remote switch oc-lab operations-center admin os show ``` -Actual output: +Actual output (uptime will vary): ``` +WARNING: The IncusOS API and configuration is subject to change + environment: hostname: oc-server os_name: IncusOS os_version: "202602230420" os_version_next: "" - uptime: 4956 + uptime: 63 ``` ### Wait for Updates @@ -181,18 +184,23 @@ reach `ready` state before ISOs can be generated: operations-center provisioning update list ``` -Actual output (after ~5 minutes): +Actual output (after ~8 minutes; UUIDs are stable across deployments): ``` -+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ -| UUID | Origin | Channels | Upstream Channels | Version | Sev | Status | -+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ -| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | stable, testing | 202602200553 | none | ready | -| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | stable, testing | 202602210344 | none | ready | -| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | stable, testing | 202602230420 | none | ready | -+--------------------------------------+---------------------+----------+-------------------+--------------+------+--------+ ++--------------------------------------+---------------------+----------+--------------+----------+--------+ +| UUID | Origin | Channels | Version | Severity | Status | ++--------------------------------------+---------------------+----------+--------------+----------+--------+ +| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready | +| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready | +| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready | ++--------------------------------------+---------------------+----------+--------------+----------+--------+ ``` +The table also includes `Upstream Channels` and `Published At` columns (omitted +for width). All 3 updates may not be ready at the same time -- at least one +`ready` is sufficient to proceed. The latest version (202602230420) typically +reaches `ready` first. + --- ## Section 3: Provision Nodes @@ -204,16 +212,18 @@ operations-center provisioning token add --uses 5 --description "OC-managed clus operations-center provisioning token list ``` -Actual output: +Actual output (UUID changes every run): ``` +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ | UUID | Uses Remaining | Expire At | Channel | Description | +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ -| 2ba12420-fee5-4d55-9f2f-40809e9ad5de | 5 | 2026-03-25 10:29:39 +0100 CET | stable | OC-managed cluster v2 | +| | 5 | <30 days from now> | stable | OC-managed cluster v2 | +--------------------------------------+----------------+-------------------------------+---------+-----------------------+ ``` +Save the `` -- you'll need it for the next steps. + ### 3.2 Create Token Seed (No force_reboot) **Critical**: the token seed must NOT include `force_reboot`. On Proxmox, @@ -233,7 +243,7 @@ format (`version: "1"` at root) maps fields to empty `{}` and they don't get assigned to any section. ```bash -operations-center provisioning token seed add proxmox-preseed \ +operations-center provisioning token seed add proxmox-preseed \ /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox" ``` @@ -245,11 +255,16 @@ was never delivered through OC's update pipeline. The fix: generate the ISO from an older channel so OC can push the real update after deployment. ```bash -# Assign the second-latest update to a separate channel +# Create the old-stable channel (must exist before assigning updates to it) +operations-center provisioning channel add old-stable \ + --description "Older stable versions for initial provisioning" + +# Assign the second-latest update to the old-stable channel +# (use the UUID for 202602210344 from `provisioning update list`) operations-center provisioning update assign-channels --channel old-stable # Generate ISO from the older channel -operations-center provisioning token seed get-image proxmox-preseed \ +operations-center provisioning token seed get-image proxmox-preseed \ /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable ``` @@ -301,7 +316,7 @@ The hybrid approach uses `incusos-proxmox --iso` to combine OC auto-registration (hostname, static IP), install monitoring, and media cleanup. ```bash -./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes examples/lab-oc-nodes.yaml +./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml ``` Actual output (key lines): @@ -328,21 +343,25 @@ from 202602210344 to 202602230420 happens automatically: operations-center provisioning server list ``` -Actual output: +Actual output (key columns; full table includes Type, Channel, Certificate +Fingerprint, Public Connection URL, Last Updated, Last Seen): ``` -+------------+-------------------+------------------------------+--------+-------------+ -| Cluster | Name | Connection URL | Status | Update Stat | -+------------+-------------------+------------------------------+--------+-------------+ -| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | -| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | -| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | -| | operations-center | https://[::1]:8443 | ready | update pend | -+------------+-------------------+------------------------------+--------+-------------+ ++---------+-------------------+------------------------------+--------+----------------+ +| Cluster | Name | Connection URL | Status | Update Status | ++---------+-------------------+------------------------------+--------+----------------+ +| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | +| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | +| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | +| | operations-center | https://[::1]:8443 | ready | update pending | ++---------+-------------------+------------------------------+--------+----------------+ ``` **Key**: all 3 nodes show **"up to date"** because OC delivered the 202602230420 update through its pipeline. This is what unlocks clustering. +Nodes may already be up to date by the time the last node finishes +deploying -- the update gets pushed while `incusos-proxmox` deploys +subsequent nodes sequentially. ### 3.7 Verify Scrub Schedules @@ -420,25 +439,23 @@ incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate incus cluster list oc-node-01: ``` -Actual output: +Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION): ``` +------------+------------------------------+-----------------+--------------+--------+-------------------+ | NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE | +------------+------------------------------+-----------------+--------------+--------+-------------------+ -| oc-node-01 | https://192.168.102.140:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | -| | | database-leader | | | | +| oc-node-01 | https://192.168.102.140:8443 | database-leader | x86_64 | ONLINE | Fully operational | | | | database | | | | +------------+------------------------------+-----------------+--------------+--------+-------------------+ -| oc-node-02 | https://192.168.102.141:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | -| | | database | | | | +| oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational | +------------+------------------------------+-----------------+--------------+--------+-------------------+ -| oc-node-03 | https://192.168.102.142:8443 | ovn-chassis | x86_64 | ONLINE | Fully operational | -| | | database | | | | +| oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational | +------------+------------------------------+-----------------+--------------+--------+-------------------+ ``` -All 3 nodes ONLINE and Fully operational. +All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added +later in Section 5.4. ### 4.5 Cluster Resources Created by OC @@ -447,25 +464,26 @@ incus storage list oc-node-01: incus network list oc-node-01: ``` -Actual output: +Actual output (incusbr0 subnet varies per deployment): ``` +-------+--------+--------------------------------------+---------+---------+ | NAME | DRIVER | DESCRIPTION | USED BY | STATE | +-------+--------+--------------------------------------+---------+---------+ -| local | zfs | Local storage pool (on system drive) | 8 | CREATED | +| local | zfs | Local storage pool (on system drive) | 8 | CREATED | +-------+--------+--------------------------------------+---------+---------+ -+----------+--------+---------+----------------+------------------------------+---------+ -| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | -+----------+--------+---------+----------------+------------------------------+---------+ -| incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | -| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | -+----------+--------+---------+----------------+------------------------------+---------+ ++----------+--------+---------+-----------------+------------------------------+---------+ +| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | ++----------+--------+---------+-----------------+------------------------------+---------+ +| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 | +| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | ++----------+--------+---------+-----------------+------------------------------+---------+ ``` OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and -`meshbr0` (OC-specific mesh network for inter-node communication). +`meshbr0` (OC-specific mesh network for inter-node communication). The +table also includes IPv6 and STATE columns. --- @@ -574,13 +592,17 @@ incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3 Actual output: ``` -64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.744 ms -64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.169 ms -64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.150 ms +64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms +64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms +64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms ``` Sub-millisecond cross-node latency via Geneve tunnels. Clean up test -containers after verification. +containers after verification: + +```bash +incus delete oc-node-01:test-1 oc-node-01:test-2 --force +``` ### 5.8 Final Network State @@ -595,12 +617,16 @@ Actual output: | NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY | +----------+----------+---------+----------------+------------------------------+---------+ | UPLINK | physical | YES | | | 1 | -| incusbr0 | bridge | YES | 10.149.76.1/24 | Local network bridge (NAT) | 2 | +| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 | | meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 | -| net-prod | ovn | YES | 10.10.10.1/24 | | 3 | +| net-prod | ovn | YES | 10.10.10.1/24 | | 0 | +----------+----------+---------+----------------+------------------------------+---------+ ``` +The incusbr0 subnet is randomly assigned per deployment. The USED BY count +for net-prod is 0 at this point (test containers deleted); it increases as +workloads are added in Section 6. + --- ## Section 6: HA Nginx Workload @@ -651,18 +677,18 @@ incus network load-balancer port add oc-node-01:net-prod 192.168.103.201 \ for i in 1 2 3 4 5; do curl -s http://192.168.103.201; done ``` -Actual output: +Actual output (distribution varies per run due to connection-based hashing): ``` -

ha-web-03 on oc-node-03

-

ha-web-02 on oc-node-02

+

ha-web-01 on oc-node-01

ha-web-02 on oc-node-02

ha-web-01 on oc-node-01

+

ha-web-01 on oc-node-01

ha-web-02 on oc-node-02

``` -Traffic distributes across all 3 backends (connection-based hashing, not -round-robin). +Traffic distributes across backends (connection-based hashing, not +round-robin). Not all backends may appear in every 5-request sample. ### 6.5 HA Failover Test @@ -688,15 +714,15 @@ sleep 3 for i in 1 2 3 4 5 6; do curl -s http://192.168.103.201; done ``` -Actual output after restart: +Actual output after restart (all 3 backends serving again): ``` -

ha-web-03 on oc-node-03

ha-web-01 on oc-node-01

-

ha-web-03 on oc-node-03

-

ha-web-02 on oc-node-02

ha-web-02 on oc-node-02

ha-web-02 on oc-node-02

+

ha-web-01 on oc-node-01

+

ha-web-01 on oc-node-01

+

ha-web-01 on oc-node-01

``` All 3 backends recovered. @@ -732,19 +758,22 @@ Actual output: incus list oc-node-01: ``` -Actual output: +Actual output (key columns; full table includes IPv6, SNAPSHOTS): ``` -+-------------+---------+----------------------+-----------+------------+ -| NAME | STATE | IPV4 | TYPE | LOCATION | -+-------------+---------+----------------------+-----------+------------+ -| ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 | -| ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 | -| ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 | -| ovn-central | RUNNING | 10.149.76.198 (eth0) | CONTAINER | oc-node-01 | -+-------------+---------+----------------------+-----------+------------+ ++-------------+---------+-----------------------+-----------+------------+ +| NAME | STATE | IPV4 | TYPE | LOCATION | ++-------------+---------+-----------------------+-----------+------------+ +| ha-web-01 | RUNNING | 10.10.10.2 (eth0) | CONTAINER | oc-node-01 | +| ha-web-02 | RUNNING | 10.10.10.3 (eth0) | CONTAINER | oc-node-02 | +| ha-web-03 | RUNNING | 10.10.10.4 (eth0) | CONTAINER | oc-node-03 | +| ovn-central | RUNNING | 10.x.x.x (eth0) | CONTAINER | oc-node-01 | ++-------------+---------+-----------------------+-----------+------------+ ``` +The ovn-central container is on incusbr0 (random subnet per deployment). +The ha-web containers are on net-prod (10.10.10.0/24, fixed). + --- ## Section 7: OC Management @@ -763,7 +792,7 @@ operations-center provisioning cluster resync oc-cluster operations-center inventory instance list --cluster oc-cluster ``` -Actual output: +Actual output (UUIDs are deterministic based on name/project/cluster): ``` +--------------------------------------+-------------+---------+------------+------------+ @@ -776,6 +805,8 @@ Actual output: +--------------------------------------+-------------+---------+------------+------------+ ``` +The full table also includes a `Last Updated` column. + ```bash operations-center inventory network list --cluster oc-cluster ``` @@ -793,6 +824,8 @@ Actual output: +--------------------------------------+----------+----------+---------+------------+ ``` +The full table also includes a `Last Updated` column. + ```bash operations-center inventory network-load-balancer list operations-center inventory network-forward list @@ -807,19 +840,22 @@ All resources visible in OC inventory after resync. operations-center provisioning server list ``` -Actual output: +Actual output (key columns): ``` -+------------+-------------------+------------------------------+--------+-------------+ -| Cluster | Name | Connection URL | Status | Update Stat | -+------------+-------------------+------------------------------+--------+-------------+ -| oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | -| oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | -| oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | -| | operations-center | https://[::1]:8443 | ready | update pend | -+------------+-------------------+------------------------------+--------+-------------+ ++------------+-------------------+------------------------------+--------+----------------+ +| Cluster | Name | Connection URL | Status | Update Status | ++------------+-------------------+------------------------------+--------+----------------+ +| oc-cluster | oc-node-01 | https://192.168.102.140:8443 | ready | up to date | +| oc-cluster | oc-node-02 | https://192.168.102.141:8443 | ready | up to date | +| oc-cluster | oc-node-03 | https://192.168.102.142:8443 | ready | up to date | +| | operations-center | https://[::1]:8443 | ready | update pending | ++------------+-------------------+------------------------------+--------+----------------+ ``` +Nodes now show their cluster membership. The OC server itself always shows +`update pending` (it cannot update itself via its own pipeline). + ### 7.4 Update Management ```bash @@ -915,9 +951,9 @@ for node in oc-node-01 oc-node-02 oc-node-03; do incus cluster role remove oc-node-01:${node} ovn-chassis done -# Destroy infrastructure -./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-nodes.yaml -./incusos/incusos-proxmox --cleanup --deep examples/lab-oc-deploy.yaml +# Destroy infrastructure (--yes to skip confirmation prompts) +./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml +./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml # Remove OC remote operations-center remote remove oc-lab @@ -955,14 +991,16 @@ operations-center remote remove oc-lab ``` 1. Deploy OC server (incusos-proxmox) 2. Add OC remote (operations-center remote add) -3. Wait for updates (provisioning update list) +3. Wait for updates (provisioning update list -- at least one "ready") 4. Create token (provisioning token add) 5. Create token seed (provisioning token seed add) -- NO force_reboot -6. Generate ISO from OLDER channel (token seed get-image --channel old-stable) -7. Deploy nodes (incusos-proxmox --iso) -8. Wait for auto-registration + update delivery -9. Form cluster (provisioning cluster add) -10. Set up OVN networking (OVN container + services + UPLINK + net-prod) +6. Create old-stable channel (provisioning channel add old-stable) +7. Assign older update to old-stable (provisioning update assign-channels) +8. Generate ISO from OLDER channel (token seed get-image --channel old-stable) +9. Deploy nodes (incusos-proxmox --iso) +10. Wait for auto-registration + update delivery +11. Form cluster (provisioning cluster add) +12. Set up OVN networking (OVN container + services + UPLINK + net-prod) ``` ### Key OC CLI Commands @@ -975,6 +1013,7 @@ operations-center remote remove oc-lab | `provisioning token add --uses N` | Create provisioning token | | `provisioning token seed add ` | Attach seed to token | | `provisioning token seed get-image --channel ` | Generate ISO | +| `provisioning channel add --description ` | Create provisioning channel | | `provisioning update list` | List available updates | | `provisioning update refresh` | Check for new updates | | `provisioning update assign-channels --channel ` | Assign update to channel | @@ -1020,6 +1059,7 @@ operations-center remote remove oc-lab | OC reboot breaks node | OC agent pushes invalid config | Use Proxmox stop/start, not OC reboot | | UPLINK type mismatch error | Missing `--type=physical` on cluster-wide create | Include `--type=physical` on both per-target and cluster-wide | | Stale ISO on Proxmox | `incusos-proxmox` skipped upload | Fixed in script: `--iso` now replaces existing same-named ISOs | +| `assign-channels` "Not found" | Channel doesn't exist yet | Run `provisioning channel add old-stable` first | | Bash `!` in API token | History expansion corrupts auth header | Use Python `urllib.request` or `set +H` | ---