From fd30ef27d4b592c694b43e5ae6f97e862258ecdb Mon Sep 17 00:00:00 2001 From: Maarten Date: Mon, 23 Feb 2026 22:59:30 +0100 Subject: [PATCH] Expand Aether guide: deploy forms, blueprints, instance mgmt, OVN recovery Document the full Aether platform after hands-on testing: - Deploy form fields, validation, IP reservation, AWX integration - Blueprint design (cluster-agnostic, component structure, JSON API) - Blueprint deployment lifecycle (3-phase deploy, decommission workflow) - Instance management (state control, resources, snapshots, console, ACLs) - OVN recovery procedure after ovn-central container move (5-phase) - Node resize procedure (Proxmox disk grow + ZFS pool expansion) - Update lab-oc-nodes defaults to match resized nodes (20G RAM, 100G disk) Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 21 + README.md | 3 + incusos/examples/lab-oc-nodes.yaml | 4 +- notes/aether-guide.md | 605 ++++++++++++++++++++++++++++- 4 files changed, 618 insertions(+), 15 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5530611..6ee7b85 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,6 +32,7 @@ incu-contrib/ ├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested) ├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end) ├── migration-guide.md # Migration paths into Incus from other hypervisors + ├── aether-guide.md # Aether management platform (deploy, blueprints, API) ├── incus-version-compatibility.md # Incus versions across platforms ├── iso-download-methods.md # ISO download/customization research └── utm-support.md # UTM support design document (future) @@ -564,6 +565,26 @@ incu-contrib/ downloads application sysext (~30-120s depending on CDN speed). Do not reduce — premature retries can corrupt the TPM encryption key permanently. +### IncusOS disk resize (Proxmox VMs) + +- **Disk grow is non-destructive**: Proxmox `qm resize` only grows disks. +- **Partition 11 auto-expands**: `systemd-repart` runs in initrd on every + boot. The `local-data` partition has no `SizeMaxBytes`, so it fills all + remaining space automatically. +- **ZFS pool does NOT auto-expand**: IncusOS creates the pool without + `autoexpand=on`. After partition grow, `zpool list` shows the correct + `EXPANDSZ` but the pool stays at its original size. +- **Manual expansion via privileged container**: create a container with + `security.privileged=true`, pass `/dev/zfs` (unix-char) and `/dev/sda11` + (unix-block), install `zfsutils-linux`, create a symlink at + `/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11` → + `/dev/sda11`, then run `zpool online -e local + scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11`. Clean up the container + after expansion. Full procedure in `notes/aether-guide.md`. +- **Warning**: running `zpool online -e` with a non-existent device path + inside the container causes pool SUSPENSION. Recovery: Proxmox stop/start + (ZFS re-imports cleanly, no data loss). + ### IncusOS first-boot sequence Complete lifecycle from Proxmox VM start to port 8443 ready: diff --git a/README.md b/README.md index af6a6db..22100e7 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,9 @@ Guides and reference material (roughly in learning order): Manual cluster + OVN + HA workloads (validated end-to-end). - [`migration-guide.md`](notes/migration-guide.md) -- Migration paths into Incus from Proxmox, UTM, VMware Fusion, and Docker. +- [`aether-guide.md`](notes/aether-guide.md) -- + Aether management platform: deployment, instance/blueprint lifecycle, API + reference, OVN recovery, and node resizing. - [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) -- Incus package versions across platforms and install instructions. - [`utm-support.md`](notes/utm-support.md) -- diff --git a/incusos/examples/lab-oc-nodes.yaml b/incusos/examples/lab-oc-nodes.yaml index 1e038e5..6d07611 100644 --- a/incusos/examples/lab-oc-nodes.yaml +++ b/incusos/examples/lab-oc-nodes.yaml @@ -16,8 +16,8 @@ defaults: cores: 4 - memory: 8192 - disk: 50 + memory: 20480 + disk: 100 start_vmid: 400 proxmox: diff --git a/notes/aether-guide.md b/notes/aether-guide.md index b5f421a..7d66428 100644 --- a/notes/aether-guide.md +++ b/notes/aether-guide.md @@ -198,26 +198,605 @@ connection. ### Cluster members (as seen in Aether) -| Node | URL | Status | Memory | Load | Roles | -|------|-----|--------|--------|------|-------| -| oc-node-01 | https://192.168.102.140:8443 | Online | 35% (5.5/16.0 GB) | 19% | ovn-chassis, database | -| oc-node-02 | https://192.168.102.141:8443 | Online | 92% (14.7/16.0 GB) | 28% | ovn-chassis, database-leader | -| oc-node-03 | https://192.168.102.142:8443 | Online | 54% (8.7/16.0 GB) | 25% | ovn-chassis, database | +| Node | URL | Status | RAM | Disk | Roles | +|------|-----|--------|-----|------|-------| +| oc-node-01 | https://192.168.102.140:8443 | Online | 20 GiB | 64G (29.5G pool) | ovn-chassis, database | +| oc-node-02 | https://192.168.102.141:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database | +| oc-node-03 | https://192.168.102.142:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database-leader | ### Instances visible in Aether -| Name | Type | Status | IPv4 | Image | Location | -|------|------|--------|------|-------|----------| -| aether | VM | Running | 192.168.102.160 | ubuntu noble | oc-node-01 | -| ovn-central | Container | Running | 10.207.217.23 | Debian bookworm | oc-node-02 | -| ha-web-02 | Container | Stopped | 10.10.10.3 | Debian bookworm | oc-node-02 | -| ha-web-03 | Container | Stopped | 10.10.10.4 | Debian bookworm | oc-node-03 | +| Name | Type | Status | IPv4 | Network | Location | Tags | +|------|------|--------|------|---------|----------|------| +| aether | VM | Running | 192.168.102.160 | macvlan (mgmt) | oc-node-01 | - | +| aether-test-ct | Container | Running | 10.207.217.22 | incusbr0 | oc-node-02 | owner: admin | +| aether-test-vm | VM | Running | 10.207.217.64 | incusbr0 | oc-node-03 | owner: admin | +| demo-web-tier-web-1 | Container | Running | 10.207.217.2 | incusbr0 | oc-node-02 | blueprint: web-tier, ha-group: demo-web-tier-web | +| demo-web-tier-web-2 | Container | Running | 10.207.217.3 | incusbr0 | oc-node-01 | blueprint: web-tier, ha-group: demo-web-tier-web | +| demo-web-tier-app-1 | Container | Running | 10.207.217.4 | incusbr0 | oc-node-03 | blueprint: web-tier | +| ovn-central | Container | Running | 10.207.217.23 | incusbr0 | oc-node-03 | - | + +### Storage usage + +| Node | Pool Size | Used | Available | +|------|-----------|------|-----------| +| oc-node-01 | 29.5 GiB | 8.9 GiB | 20.6 GiB | +| oc-node-02 | 64.4 GiB | 7.4 GiB | 57.1 GiB | +| oc-node-03 | 64.4 GiB | 7.9 GiB | 56.5 GiB | ### Networks visible in Aether | Name | Type | IPv4 | Uplink | NAT | Used By | |------|------|------|--------|-----|---------| | UPLINK | physical | 192.168.100.1/22 | - | - | 1 | -| incusbr0 | bridge | 10.207.217.1/24 | - | IPv4, IPv6 | 2 | +| incusbr0 | bridge | 10.207.217.1/24 | - | IPv4, IPv6 | 7 | | meshbr0 | bridge | none | - | IPv6 | 1 | -| net-prod | ovn | 10.10.10.1/24 | UPLINK | IPv4, IPv6 | 2 | +| net-prod | ovn | 10.10.10.1/24 | UPLINK | IPv4, IPv6 | 0 | + +## Deploying Instances + +Navigate to **Deploy VM/Container/Blueprint** (`/deploy`). The form uses +progressive disclosure — each section appears after the preceding selection. + +### Deploy flow (single instance) + +1. **Select Cluster** — dropdown populated from `/deploy/clusters` +2. **Select Type** — radio: Virtual Machine, Container, or Blueprint +3. **Select Image** — dropdown filtered by type, fetched from + `/deploy/images/{clusterID}?type={type}` +4. **Name** — validated in real-time against the cluster + (`/deploy/check-name/{clusterID}/{name}`) +5. **Network** — dropdown shows `name (TYPE) - ipv4_network` +6. **IP Address** — available IPs from the chosen network, ping-verified + before assignment (`/deploy/verify-ip/{clusterID}`) +7. **Resources** — CPU (1-64, default 2), Memory (default 4 GiB), Root Disk + (default 50 GiB). Memory validated against cluster available memory with + warning at 90-95% and block at >95% +8. **Storage Pool** — auto-selects pool with most free space +9. **Additional Disks** — optional, VM only. Each disk has size + pool +10. **Instance Tags** — optional key-value pairs (max 10). Reserved keys: + `owner`, `deployed_by`, `deployed_at` (set automatically) +11. **Deploy** — reserves IP → creates instance → optionally triggers AWX + post-deploy playbook → releases IP reservation + +### Deploy form fields + +| Field | Type | Default | Constraints | +|-------|------|---------|-------------| +| Cluster | select | - | Required | +| Instance type | radio | - | virtual-machine, container, blueprint | +| Image | select | - | Filtered by type | +| Name | text | - | `[a-zA-Z][a-zA-Z0-9-]{0,62}`, unique on cluster | +| Network | select | - | Lists all cluster networks with CIDR | +| IP Address | select | - | Available IPs in network, ping-verified | +| CPU | select | 2 | 1, 2, 4, 8, 16, 32, 64 | +| Memory | number + unit | 4 GiB | 1-1024 GiB or 64-1048576 MiB | +| Root Disk | number | 50 GiB | 1-10000 GiB | +| Storage Pool | select | (most free) | Shows driver + available GiB | +| Additional Disks | number + pool | 100 GiB | 1-10000 GiB each, VM only | +| Tags | key-value | - | Max 10, key: `[a-zA-Z0-9._-]+` | + +### Instance creation payload + +```json +{ + "cluster_id": 52, + "name": "my-instance", + "type": "virtual-machine", + "image_fingerprint": "abc123...", + "image_alias": "debian-12", + "network_name": "incusbr0", + "ip_address": "10.207.217.50", + "cpu": 2, + "memory_value": 4, + "memory_unit": "GiB", + "root_disk_gib": 50, + "storage_pool": "local", + "additional_disks": [{"size_gib": 100, "pool": "local"}], + "custom_tags": {"department": "engineering"} +} +``` + +### Tested: deploy container + +Deployed `aether-test-ct` (container, 1 CPU, 1 GiB RAM, 10 GiB disk) on +incusbr0 via the deploy form. Instance appeared immediately in the Instances +tab and via `incus list`. Aether auto-tags: `owner`, `deployed_by`, +`deployed_at`. + +### Tested: deploy VM + +Deployed `aether-test-vm` (VM, 2 CPU, 2 GiB RAM, 20 GiB disk) on incusbr0. +Took ~30s to start (QEMU boot). Same auto-tagging. Both test instances are +on the bridge network (node-local only). + +### AWX integration + +Aether can trigger Ansible (AWX) playbooks after instance creation and +before decommission. The deploy page shows a "Your Recent Ansible Automation +Jobs" table that auto-refreshes every 10s. AWX template IDs are configured +per blueprint or globally. If AWX is not configured, the step is skipped. + +## Blueprint Design + +Navigate to **Blueprint Design** (`/blueprintdesign`). Blueprints are +reusable deployment templates that define multiple instance components. + +### Blueprint structure + +A blueprint contains: +- **Name** — unique, `[a-zA-Z][a-zA-Z0-9-]{0,99}` +- **Description** — free text, max 500 chars +- **Components** — 1+ component definitions (max 20 total instances) +- **AWX Post-Deploy Template ID** — optional, runs after all instances deploy +- **AWX Decommission Template ID** — optional, runs before teardown + +Each component defines: + +| Field | Type | Default | Constraints | +|-------|------|---------|-------------| +| Name | text | - | Max 50 chars, required | +| Count | number | 1 | 1-10 per component, max 20 total | +| Type | select | virtual-machine | virtual-machine or container | +| Image | select | - | From reference cluster, filtered by type | +| CPU | number | 2 | 1-128 | +| Memory | number + unit | 4 GiB | 1-1024 GiB or 1-1048576 MiB | +| Root Disk | number | 50 GiB | 1-10000 GiB | +| Additional Disks | number[] | - | VM only, 1-10000 GiB each | + +Blueprints are **cluster-agnostic** — they store image aliases, not +fingerprints. The "Reference Cluster" is only used to populate the image +dropdown during design. The same blueprint can deploy to any cluster that +has matching image aliases. + +### Blueprint JSON + +```json +{ + "name": "web-tier", + "description": "2 web containers + 1 app container", + "components": [ + { + "name": "web", + "count": 2, + "type": "container", + "image_alias": "alpine-3.21", + "cpu": 1, + "memory_value": 256, + "memory_unit": "MiB", + "root_disk_gib": 2 + }, + { + "name": "app", + "count": 1, + "type": "container", + "image_alias": "alpine-3.21", + "cpu": 1, + "memory_value": 512, + "memory_unit": "MiB", + "root_disk_gib": 4 + } + ] +} +``` + +### Blueprint API + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| GET | `/api/blueprints` | List all blueprints | +| POST | `/api/blueprints` | Create blueprint | +| PUT | `/api/blueprints/{id}` | Update blueprint | +| DELETE | `/api/blueprints/{id}` | Delete (blocked if deployments exist) | +| GET | `/api/blueprints/images/{clusterID}` | List images for design | + +### Tested: web-tier blueprint + +Created blueprint `web-tier` with 2 components: +- `web`: 2x Alpine 3.21 containers, 1 CPU, 256 MiB, 2 GiB disk +- `app`: 1x Alpine 3.21 container, 1 CPU, 512 MiB, 4 GiB disk + +Total: 3 instances, 3 vCPUs, 1 GiB memory, 8 GiB storage. + +## Deploying Blueprints + +When "Blueprint" is selected as instance type on the deploy page, the form +changes: + +1. **Select Blueprint** — eligible blueprints for the cluster (checks image + availability). Shows component preview cards with resource totals +2. **Base Name** — prefix for all instances. Names generated as + `{base}-{blueprint}-{component}-{N}` (e.g., `demo-web-tier-web-1`). + Validated against cluster in real-time +3. **Network** — same as single instance, but no individual IP selection. + Verifies enough available IPs for all instances +4. **Storage Pool** — validates total storage fits +5. **Deploy** — three-phase process: + - `POST /deploy/blueprint/start` — resolves images, assigns IPs + - Sequential `POST /deploy/blueprint/{id}/instance` per instance + - `POST /deploy/blueprint/{id}/complete` — finalizes, triggers AWX + +### Blueprint deploy progress + +A modal shows real-time step-by-step progress with per-instance status. +Users can: +- **Cancel** — stops creating remaining instances, server rolls back +- **Run in Background** — hides modal, shows indicator in bottom-right + corner (click to re-open) + +### Instance naming convention + +``` +{base_name}-{blueprint_name}-{component_name}-{N} +``` + +Example with base `demo`, blueprint `web-tier`: +- `demo-web-tier-web-1` (web component, instance 1) +- `demo-web-tier-web-2` (web component, instance 2) +- `demo-web-tier-app-1` (app component, instance 1) + +### Blueprint instance tags + +Blueprint-deployed instances receive automatic tags: + +| Tag | Example | +|-----|---------| +| `blueprint` | web-tier | +| `blueprint_base` | demo | +| `blueprint_component` | web | +| `ha-group` | demo-web-tier-web | +| `deployed_by` | admin | +| `deployed_at` | 2026-02-23T20:52:19Z | +| `owner` | admin | + +The `ha-group` tag groups instances of the same component for HA tracking. + +### Tested: deploy web-tier blueprint + +Deployed with base name `demo` on `incusbr0` network, `local` storage pool. +All 3 instances created successfully on different nodes (scheduler spread them +across oc-node-01, oc-node-02, oc-node-03). Each instance received correct +tags. IPs auto-assigned from incusbr0 DHCP range. + +## Deployed Blueprints (Lifecycle) + +Navigate to **Deployed Blueprints** (`/deployedblueprints`). Tracks all +blueprint deployments with lifecycle management. + +### Table columns + +| Column | Description | +|--------|-------------| +| Blueprint | Blueprint template name | +| Base Name | Instance name prefix | +| Cluster | Target cluster | +| Network | Deployment network | +| Instances | Count (clickable — shows detail modal) | +| Status | deploying, deployed, decommissioning, decommissioned, failed | +| Deployed By | Username | +| Deployed At | Timestamp | +| Actions | Change Owner (admin), Decommission | + +### Decommission workflow + +1. Click **Decommission** → confirmation dialog +2. `POST /api/deployedblueprints/{id}/decommission/start` — triggers AWX + decommission playbook (if configured) +3. Sequential `DELETE /api/deployedblueprints/{id}/decommission/instance/{name}` + per instance — deletes from cluster one by one +4. `POST /api/deployedblueprints/{id}/decommission/complete` — finalizes record + +Progress modal shows per-instance deletion status. Blueprints with active +deployments cannot be deleted from the Blueprint Design page. + +### Deployed blueprint API + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| GET | `/api/deployedblueprints` | List all deployments | +| POST | `/api/deployedblueprints/{id}/change-owner` | Transfer ownership | +| POST | `/api/deployedblueprints/{id}/decommission/start` | Start teardown | +| DELETE | `/api/deployedblueprints/{id}/decommission/instance/{name}` | Delete instance | +| POST | `/api/deployedblueprints/{id}/decommission/complete` | Finalize | + +## Managing Instances + +Navigate to **Manage VMs/Containers** (`/infra`). Shows all instances across +connected clusters with full lifecycle management. + +### Instance table columns + +| Column | Width | Content | +|--------|-------|---------| +| Manage | 8% | Opens management popup | +| Name | 15% | Instance name | +| Type | 8% | virtual-machine or container | +| Status | 8% | Running, Stopped, Frozen | +| Location | 12% | Cluster member node | +| IPs | 22% | NIC name: IP address | +| Instance Tags | 27% | Key-value metadata (`user.*` namespace) | + +### Management actions + +**State control:** +- Running → Restart, Stop, Force Stop, Freeze +- Stopped → Start +- Frozen → Unfreeze + +**Resource editing:** +- Edit CPU — number or range (e.g., `0-3`). Running VMs offer "Stop & Apply" +- Edit Memory — value + unit. Warns about OOM (containers) or hotplug limits (VMs) +- Resize Disk — grow only, warns about manual filesystem expansion for VMs +- Add/Remove Disk — non-root disks only + +**Snapshots:** +- Take Snapshot — named, optional expiration (default 7 days) +- Restore Snapshot — reverts instance to snapshot state +- Delete Snapshot + +**Console access:** +- Text Console — WebSocket terminal (xterm.js) +- Exec Shell — shell exec via WebSocket +- VGA Console — SPICE graphical console (VMs only, with Ctrl+Alt+Del) + +**Instance Tags** — add, edit, delete key-value metadata. Reserved keys +excluded. Changes committed in batch. + +**ACLs** — view applied firewall rules per NIC, with ingress/egress filtering. + +### Instance management API + +| Method | Endpoint | Purpose | +|--------|----------|---------| +| GET | `/infra/details/{clusterID}/{name}` | Full instance details | +| POST | `/infra/state/{clusterID}/{name}` | Change state | +| DELETE | `/infra/instance/{clusterID}/{name}` | Delete instance | +| POST | `/infra/snapshot/create/{clusterID}/{name}` | Create snapshot | +| POST | `/infra/snapshot/action/{clusterID}/{name}/{snap}` | Restore/delete | +| POST | `/infra/resources/{clusterID}/{name}` | Edit CPU/memory/disk | +| POST | `/infra/disk/{clusterID}/{name}` | Add disk | +| DELETE | `/infra/disk/{clusterID}/{name}/{disk}` | Remove disk | +| POST | `/infra/metadata/{clusterID}/{name}` | Update tags | +| WS | `/infra/console/{clusterID}/{name}` | Text console | +| WS | `/infra/exec/{clusterID}/{name}` | Shell exec | +| WS | `/infra/vga-console/{clusterID}/{name}` | SPICE VGA | + +## Recovering OVN After ovn-central Container Move + +Moving the `ovn-central` container between nodes causes the OVN northbound +(NB) database to be reset/empty. The southbound (SB) database retains chassis +entries but becomes stale. The `net-prod` OVN network exists in Incus but has +no backing OVN objects. `incus network delete` hangs because Incus tries to +clean non-existent OVN resources. + +### Root causes discovered + +1. **OVN NB database loss**: moving the ovn-central container creates a fresh + NB database. All logical switches, routers, DNS records, NAT rules, DHCP + options, and HA chassis groups are lost. +2. **Stale OVS bridges**: when Incus creates a physical uplink (UPLINK), it + creates an OVS bridge named `incusovnN` (where N is the network DB ID). + Deleting and recreating the network assigns a new ID, creating a new bridge + while the old one persists in OVS. The old bridge still holds the physical + parent interface mapping. +3. **OVN chassis UUID mismatch**: toggling the OVN service on IncusOS nodes + can change the `system-id` in the OVN SB. But the `system-id` stored in + the local OVS `external_ids` persists. This creates a mismatch where Incus + writes `requested-chassis=` but the SB chassis has a + different name. Ports can't be bound. + +### Recovery procedure (tested) + +**Phase 1: Remove stale network from Incus DB** + +```bash +# Get the stale network ID +incus admin sql :global \ + "SELECT id, name FROM networks WHERE name = 'net-prod'" + +# Delete all related DB entries (children first) +incus admin sql :global "DELETE FROM networks_load_balancers WHERE network_id = " +incus admin sql :global "DELETE FROM networks_forwards WHERE network_id = " +incus admin sql :global "DELETE FROM networks_nodes WHERE network_id = " +incus admin sql :global "DELETE FROM networks_config WHERE network_id = " +incus admin sql :global "DELETE FROM networks WHERE id = " +``` + +**Phase 2: Proxmox stop/start all nodes** + +Required to clear the Incus daemon's in-memory cache of stale OVN state. +Guest reboot is NOT safe on OC-managed nodes (see CLAUDE.md). Proxmox +stop/start IS safe. + +```bash +# Stop each VM via Proxmox API, wait 10s, start +# One node at a time, wait for rejoin before next +``` + +**Phase 3: Clean stale OVS bridges on each node** + +Use a privileged container with OVS tools to access the host's OVS database. +**Critical**: mount the host's OVS socket at a different path to avoid the +container's own OVS service masking it. + +```bash +incus launch images:debian/12 :ovs-fix --target \ + -c security.privileged=true +incus config device add :ovs-fix ovs-run disk \ + source=/run/openvswitch path=/host-ovs + +# Install OVS with auto-start prevention +incus exec :ovs-fix -- bash -c " + echo -e '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d + chmod +x /usr/sbin/policy-rc.d + apt-get update -qq && apt-get install -y -qq openvswitch-switch + rm /usr/sbin/policy-rc.d + systemctl stop openvswitch-switch" + +# Delete old stale bridge (e.g., incusovn3 from the old UPLINK) +incus exec :ovs-fix -- \ + ovs-vsctl --db=unix:/host-ovs/db.sock del-br incusovn3 + +# Fix bridge mapping to point to new bridge only +incus exec :ovs-fix -- \ + ovs-vsctl --db=unix:/host-ovs/db.sock set Open_vSwitch . \ + external-ids:ovn-bridge-mappings="UPLINK:incusovn" + +# Move container to next node and repeat +``` + +**Phase 4: Fix chassis UUID mismatch (node-01 specific)** + +If a node's OVS `system-id` doesn't match the SB chassis name, delete the +stale SB chassis and let ovn-controller re-register: + +```bash +# Check OVS system-id +incus exec :ovs-fix -- \ + ovs-vsctl --db=unix:/host-ovs/db.sock get Open_vSwitch . external-ids:system-id + +# Delete stale chassis from SB +incus exec :ovn-central -- ovn-sbctl chassis-del "" +# ovn-controller re-registers with the correct system-id within seconds +``` + +**Phase 5: Recreate networks** + +```bash +# Recreate UPLINK (must use --target per node, talk to each node directly) +incus network create oc-node-01:UPLINK --type physical parent=mgmt --target oc-node-01 +incus network create oc-node-02:UPLINK --type physical parent=mgmt --target oc-node-02 +incus network create oc-node-03:UPLINK --type physical parent=mgmt --target oc-node-03 +# Finalize +incus network create :UPLINK --type physical \ + ipv4.gateway=192.168.100.1/22 \ + ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \ + dns.nameservers=192.168.100.1 + +# Create OVN network +incus network create :net-prod --type=ovn network=UPLINK \ + ipv4.address=10.10.10.1/24 ipv4.nat=true \ + ipv6.address=auto ipv6.nat=true +``` + +### Key lessons + +- **Never toggle OVN service on IncusOS nodes unnecessarily** — it can change + the `system-id`, creating chassis UUID mismatches. +- **Mount host OVS at `/host-ovs`** (not `/run/openvswitch`) — the container's + own OVS service masks the host's socket at the default path. +- **`incus network create --target` must be sent directly to each node** — + cluster-forwarded `--target` calls can time out. +- **Proxmox stop/start is safe** for OC-managed nodes; guest reboot is NOT. + +## Resizing IncusOS Nodes (Proxmox) + +Growing IncusOS VMs requires three steps: Proxmox resize, boot (partition +auto-expand), and manual ZFS pool expansion. + +### Step 1: Proxmox resize (VM must be stopped) + +```bash +# Stop VM via Proxmox API +curl -fsSk -H "Authorization: PVEAPIToken=..." \ + -X POST "https://:8006/api2/json/nodes//qemu//status/stop" + +# Resize disk (non-destructive, grow only) +curl -fsSk -H "Authorization: PVEAPIToken=..." \ + -X PUT -d "disk=scsi0&size=100G" \ + "https://:8006/api2/json/nodes//qemu//resize" + +# Set memory (in MiB) +curl -fsSk -H "Authorization: PVEAPIToken=..." \ + -X PUT -d "memory=20480" \ + "https://:8006/api2/json/nodes//qemu//config" + +# Start VM +curl -fsSk -H "Authorization: PVEAPIToken=..." \ + -X POST "https://:8006/api2/json/nodes//qemu//status/start" +``` + +### Step 2: Partition auto-expand (automatic on boot) + +IncusOS uses `systemd-repart` which runs during initrd on every boot. The +`local-data` partition (partition 11) has no `SizeMaxBytes`, so it +automatically grows to fill all remaining disk space. This is automatic — +no action needed. + +Verify via the IncusOS API: +```bash +incus query :/os/1.0/system/resources | python3 -c " +import json,sys; d=json.load(sys.stdin)['storage'] +for disk in d['disks']: + for part in disk['partitions']: + if part['partition'] == 11: + print(f'Part 11: {part[\"size\"]/1073741824:.1f} GiB')" +``` + +### Step 3: ZFS pool expansion (manual — not automatic) + +**Critical discovery**: IncusOS creates the ZFS pool without `autoexpand=on`. +After the partition grows, the ZFS pool does NOT automatically use the new +space. The pool's `EXPANDSZ` column shows the available expansion, but it +must be triggered manually. + +Since IncusOS has no shell access and no API for `zpool online -e`, use a +**privileged container** to run the ZFS expansion: + +```bash +# Create privileged container on the target node +incus launch images:debian/12 :zfs-expand --target \ + -c security.privileged=true + +# Add devices for ZFS control and the partition +incus config device add :zfs-expand zfs unix-char \ + source=/dev/zfs path=/dev/zfs +incus config device add :zfs-expand sda11 unix-block \ + source=/dev/sda11 path=/dev/sda11 + +# Install ZFS tools and create device symlink +incus exec :zfs-expand -- bash -c " + apt-get update -qq && apt-get install -y -qq zfsutils-linux + mkdir -p /dev/disk/by-id + ln -sf /dev/sda11 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11" + +# Verify pool sees expansion available +incus exec :zfs-expand -- zpool list +# Output shows EXPANDSZ column with available expansion (e.g., 50G) + +# Expand the pool +incus exec :zfs-expand -- zpool online -e local \ + scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11 + +# Verify expansion +incus exec :zfs-expand -- zpool list +# EXPANDSZ should now show "-" and SIZE should reflect the new total + +# Clean up +incus delete :zfs-expand --force +``` + +**Why this works**: privileged containers share the host kernel. The `/dev/zfs` +character device is the ZFS kernel module's control interface. The container's +`zpool` commands talk directly to the host kernel's ZFS, which manages the +`local` pool. The `/dev/disk/by-id/` symlink must be created inside the +container because the container doesn't have udev populating `/dev/disk/`. + +**Warning**: do NOT run `zpool online -e` with a device path that doesn't +exist inside the container. This causes the ZFS pool to go SUSPENDED (I/O +errors). Recovery requires a Proxmox stop/start of the VM (ZFS re-imports +cleanly on boot, no data loss). + +### Rolling resize procedure (cluster) + +Resize one node at a time to maintain quorum (2 of 3 nodes): + +1. **Node-03 first** (lowest risk — fewest workloads) +2. **Node-02** — if it hosts OVN central, move it to another node first + and update OVN config on all nodes +3. **Node-01 last** — if it hosts Aether, stop it, resize, restart, then + start Aether + +After each node: verify cluster status (`incus cluster list`), storage +size (`incus storage info :local --target `), and port 8443 +reachability.