maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Expand Aether guide: deploy forms, blueprints, instance mgmt, OVN recovery 

Document the full Aether platform after hands-on testing:
- Deploy form fields, validation, IP reservation, AWX integration
- Blueprint design (cluster-agnostic, component structure, JSON API)
- Blueprint deployment lifecycle (3-phase deploy, decommission workflow)
- Instance management (state control, resources, snapshots, console, ACLs)
- OVN recovery procedure after ovn-central container move (5-phase)
- Node resize procedure (Proxmox disk grow + ZFS pool expansion)
- Update lab-oc-nodes defaults to match resized nodes (20G RAM, 100G disk)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-23 22:59:30 +01:00
parent d2b49f5ef0
commit fd30ef27d4
4 changed files with 618 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
CLAUDE.md
View File

@ -32,6 +32,7 @@ incu-contrib/
├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested)
├── production-lab-guide.md # Manual cluster + OVN + HA (validated end-to-end)
├── migration-guide.md # Migration paths into Incus from other hypervisors
├── aether-guide.md # Aether management platform (deploy, blueprints, API)
├── incus-version-compatibility.md # Incus versions across platforms
├── iso-download-methods.md # ISO download/customization research
└── utm-support.md # UTM support design document (future)
@ -564,6 +565,26 @@ incu-contrib/
downloads application sysext (~30-120s depending on CDN speed). Do not
reduce — premature retries can corrupt the TPM encryption key permanently.
### IncusOS disk resize (Proxmox VMs)
- **Disk grow is non-destructive**: Proxmox `qm resize` only grows disks.
- **Partition 11 auto-expands**: `systemd-repart` runs in initrd on every
boot. The `local-data` partition has no `SizeMaxBytes`, so it fills all
remaining space automatically.
- **ZFS pool does NOT auto-expand**: IncusOS creates the pool without
`autoexpand=on`. After partition grow, `zpool list` shows the correct
`EXPANDSZ` but the pool stays at its original size.
- **Manual expansion via privileged container**: create a container with
`security.privileged=true`, pass `/dev/zfs` (unix-char) and `/dev/sda11`
(unix-block), install `zfsutils-linux`, create a symlink at
`/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11` →
`/dev/sda11`, then run `zpool online -e local
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11`. Clean up the container
after expansion. Full procedure in `notes/aether-guide.md`.
- **Warning**: running `zpool online -e` with a non-existent device path
inside the container causes pool SUSPENSION. Recovery: Proxmox stop/start
(ZFS re-imports cleanly, no data loss).
### IncusOS first-boot sequence
Complete lifecycle from Proxmox VM start to port 8443 ready:

3
README.md
View File

@ -41,6 +41,9 @@ Guides and reference material (roughly in learning order):
Manual cluster + OVN + HA workloads (validated end-to-end).
- [`migration-guide.md`](notes/migration-guide.md) --
Migration paths into Incus from Proxmox, UTM, VMware Fusion, and Docker.
- [`aether-guide.md`](notes/aether-guide.md) --
Aether management platform: deployment, instance/blueprint lifecycle, API
reference, OVN recovery, and node resizing.
- [`incus-version-compatibility.md`](notes/incus-version-compatibility.md) --
Incus package versions across platforms and install instructions.
- [`utm-support.md`](notes/utm-support.md) --

4
incusos/examples/lab-oc-nodes.yaml
View File

@ -16,8 +16,8 @@
defaults:
cores: 4
memory: 8192
disk: 50
memory: 20480
disk: 100
start_vmid: 400
proxmox:

605
notes/aether-guide.md
View File

@ -198,26 +198,605 @@ connection.
### Cluster members (as seen in Aether)
| Node | URL | Status | Memory | Load | Roles |
|------|-----|--------|--------|------|-------|
| oc-node-01 | https://192.168.102.140:8443 | Online | 35% (5.5/16.0 GB) | 19% | ovn-chassis, database |
| oc-node-02 | https://192.168.102.141:8443 | Online | 92% (14.7/16.0 GB) | 28% | ovn-chassis, database-leader |
| oc-node-03 | https://192.168.102.142:8443 | Online | 54% (8.7/16.0 GB) | 25% | ovn-chassis, database |
| Node | URL | Status | RAM | Disk | Roles |
|------|-----|--------|-----|------|-------|
| oc-node-01 | https://192.168.102.140:8443 | Online | 20 GiB | 64G (29.5G pool) | ovn-chassis, database |
| oc-node-02 | https://192.168.102.141:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database |
| oc-node-03 | https://192.168.102.142:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database-leader |
### Instances visible in Aether
| Name | Type | Status | IPv4 | Image | Location |
|------|------|--------|------|-------|----------|
| aether | VM | Running | 192.168.102.160 | ubuntu noble | oc-node-01 |
| ovn-central | Container | Running | 10.207.217.23 | Debian bookworm | oc-node-02 |
| ha-web-02 | Container | Stopped | 10.10.10.3 | Debian bookworm | oc-node-02 |
| ha-web-03 | Container | Stopped | 10.10.10.4 | Debian bookworm | oc-node-03 |
| Name | Type | Status | IPv4 | Network | Location | Tags |
|------|------|--------|------|---------|----------|------|
| aether | VM | Running | 192.168.102.160 | macvlan (mgmt) | oc-node-01 | - |
| aether-test-ct | Container | Running | 10.207.217.22 | incusbr0 | oc-node-02 | owner: admin |
| aether-test-vm | VM | Running | 10.207.217.64 | incusbr0 | oc-node-03 | owner: admin |
| demo-web-tier-web-1 | Container | Running | 10.207.217.2 | incusbr0 | oc-node-02 | blueprint: web-tier, ha-group: demo-web-tier-web |
| demo-web-tier-web-2 | Container | Running | 10.207.217.3 | incusbr0 | oc-node-01 | blueprint: web-tier, ha-group: demo-web-tier-web |
| demo-web-tier-app-1 | Container | Running | 10.207.217.4 | incusbr0 | oc-node-03 | blueprint: web-tier |
| ovn-central | Container | Running | 10.207.217.23 | incusbr0 | oc-node-03 | - |
### Storage usage
| Node | Pool Size | Used | Available |
|------|-----------|------|-----------|
| oc-node-01 | 29.5 GiB | 8.9 GiB | 20.6 GiB |
| oc-node-02 | 64.4 GiB | 7.4 GiB | 57.1 GiB |
| oc-node-03 | 64.4 GiB | 7.9 GiB | 56.5 GiB |
### Networks visible in Aether
| Name | Type | IPv4 | Uplink | NAT | Used By |
|------|------|------|--------|-----|---------|
| UPLINK | physical | 192.168.100.1/22 | - | - | 1 |
| incusbr0 | bridge | 10.207.217.1/24 | - | IPv4, IPv6 | 2 |
| incusbr0 | bridge | 10.207.217.1/24 | - | IPv4, IPv6 | 7 |
| meshbr0 | bridge | none | - | IPv6 | 1 |
| net-prod | ovn | 10.10.10.1/24 | UPLINK | IPv4, IPv6 | 2 |
| net-prod | ovn | 10.10.10.1/24 | UPLINK | IPv4, IPv6 | 0 |
## Deploying Instances
Navigate to **Deploy VM/Container/Blueprint** (`/deploy`). The form uses
progressive disclosure — each section appears after the preceding selection.
### Deploy flow (single instance)
1. **Select Cluster** — dropdown populated from `/deploy/clusters`
2. **Select Type** — radio: Virtual Machine, Container, or Blueprint
3. **Select Image** — dropdown filtered by type, fetched from
`/deploy/images/{clusterID}?type={type}`
4. **Name** — validated in real-time against the cluster
(`/deploy/check-name/{clusterID}/{name}`)
5. **Network** — dropdown shows `name (TYPE) - ipv4_network`
6. **IP Address** — available IPs from the chosen network, ping-verified
before assignment (`/deploy/verify-ip/{clusterID}`)
7. **Resources** — CPU (1-64, default 2), Memory (default 4 GiB), Root Disk
(default 50 GiB). Memory validated against cluster available memory with
warning at 90-95% and block at >95%
8. **Storage Pool** — auto-selects pool with most free space
9. **Additional Disks** — optional, VM only. Each disk has size + pool
10. **Instance Tags** — optional key-value pairs (max 10). Reserved keys:
`owner`, `deployed_by`, `deployed_at` (set automatically)
11. **Deploy** — reserves IP → creates instance → optionally triggers AWX
post-deploy playbook → releases IP reservation
### Deploy form fields
| Field | Type | Default | Constraints |
|-------|------|---------|-------------|
| Cluster | select | - | Required |
| Instance type | radio | - | virtual-machine, container, blueprint |
| Image | select | - | Filtered by type |
| Name | text | - | `[a-zA-Z][a-zA-Z0-9-]{0,62}`, unique on cluster |
| Network | select | - | Lists all cluster networks with CIDR |
| IP Address | select | - | Available IPs in network, ping-verified |
| CPU | select | 2 | 1, 2, 4, 8, 16, 32, 64 |
| Memory | number + unit | 4 GiB | 1-1024 GiB or 64-1048576 MiB |
| Root Disk | number | 50 GiB | 1-10000 GiB |
| Storage Pool | select | (most free) | Shows driver + available GiB |
| Additional Disks | number + pool | 100 GiB | 1-10000 GiB each, VM only |
| Tags | key-value | - | Max 10, key: `[a-zA-Z0-9._-]+` |
### Instance creation payload
```json
{
"cluster_id": 52,
"name": "my-instance",
"type": "virtual-machine",
"image_fingerprint": "abc123...",
"image_alias": "debian-12",
"network_name": "incusbr0",
"ip_address": "10.207.217.50",
"cpu": 2,
"memory_value": 4,
"memory_unit": "GiB",
"root_disk_gib": 50,
"storage_pool": "local",
"additional_disks": [{"size_gib": 100, "pool": "local"}],
"custom_tags": {"department": "engineering"}
}
```
### Tested: deploy container
Deployed `aether-test-ct` (container, 1 CPU, 1 GiB RAM, 10 GiB disk) on
incusbr0 via the deploy form. Instance appeared immediately in the Instances
tab and via `incus list`. Aether auto-tags: `owner`, `deployed_by`,
`deployed_at`.
### Tested: deploy VM
Deployed `aether-test-vm` (VM, 2 CPU, 2 GiB RAM, 20 GiB disk) on incusbr0.
Took ~30s to start (QEMU boot). Same auto-tagging. Both test instances are
on the bridge network (node-local only).
### AWX integration
Aether can trigger Ansible (AWX) playbooks after instance creation and
before decommission. The deploy page shows a "Your Recent Ansible Automation
Jobs" table that auto-refreshes every 10s. AWX template IDs are configured
per blueprint or globally. If AWX is not configured, the step is skipped.
## Blueprint Design
Navigate to **Blueprint Design** (`/blueprintdesign`). Blueprints are
reusable deployment templates that define multiple instance components.
### Blueprint structure
A blueprint contains:
- **Name** — unique, `[a-zA-Z][a-zA-Z0-9-]{0,99}`
- **Description** — free text, max 500 chars
- **Components** — 1+ component definitions (max 20 total instances)
- **AWX Post-Deploy Template ID** — optional, runs after all instances deploy
- **AWX Decommission Template ID** — optional, runs before teardown
Each component defines:
| Field | Type | Default | Constraints |
|-------|------|---------|-------------|
| Name | text | - | Max 50 chars, required |
| Count | number | 1 | 1-10 per component, max 20 total |
| Type | select | virtual-machine | virtual-machine or container |
| Image | select | - | From reference cluster, filtered by type |
| CPU | number | 2 | 1-128 |
| Memory | number + unit | 4 GiB | 1-1024 GiB or 1-1048576 MiB |
| Root Disk | number | 50 GiB | 1-10000 GiB |
| Additional Disks | number[] | - | VM only, 1-10000 GiB each |
Blueprints are **cluster-agnostic** — they store image aliases, not
fingerprints. The "Reference Cluster" is only used to populate the image
dropdown during design. The same blueprint can deploy to any cluster that
has matching image aliases.
### Blueprint JSON
```json
{
"name": "web-tier",
"description": "2 web containers + 1 app container",
"components": [
{
"name": "web",
"count": 2,
"type": "container",
"image_alias": "alpine-3.21",
"cpu": 1,
"memory_value": 256,
"memory_unit": "MiB",
"root_disk_gib": 2
},
{
"name": "app",
"count": 1,
"type": "container",
"image_alias": "alpine-3.21",
"cpu": 1,
"memory_value": 512,
"memory_unit": "MiB",
"root_disk_gib": 4
}
]
}
```
### Blueprint API
| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/blueprints` | List all blueprints |
| POST | `/api/blueprints` | Create blueprint |
| PUT | `/api/blueprints/{id}` | Update blueprint |
| DELETE | `/api/blueprints/{id}` | Delete (blocked if deployments exist) |
| GET | `/api/blueprints/images/{clusterID}` | List images for design |
### Tested: web-tier blueprint
Created blueprint `web-tier` with 2 components:
- `web`: 2x Alpine 3.21 containers, 1 CPU, 256 MiB, 2 GiB disk
- `app`: 1x Alpine 3.21 container, 1 CPU, 512 MiB, 4 GiB disk
Total: 3 instances, 3 vCPUs, 1 GiB memory, 8 GiB storage.
## Deploying Blueprints
When "Blueprint" is selected as instance type on the deploy page, the form
changes:
1. **Select Blueprint** — eligible blueprints for the cluster (checks image
availability). Shows component preview cards with resource totals
2. **Base Name** — prefix for all instances. Names generated as
`{base}-{blueprint}-{component}-{N}` (e.g., `demo-web-tier-web-1`).
Validated against cluster in real-time
3. **Network** — same as single instance, but no individual IP selection.
Verifies enough available IPs for all instances
4. **Storage Pool** — validates total storage fits
5. **Deploy** — three-phase process:
- `POST /deploy/blueprint/start` — resolves images, assigns IPs
- Sequential `POST /deploy/blueprint/{id}/instance` per instance
- `POST /deploy/blueprint/{id}/complete` — finalizes, triggers AWX
### Blueprint deploy progress
A modal shows real-time step-by-step progress with per-instance status.
Users can:
- **Cancel** — stops creating remaining instances, server rolls back
- **Run in Background** — hides modal, shows indicator in bottom-right
corner (click to re-open)
### Instance naming convention
```
{base_name}-{blueprint_name}-{component_name}-{N}
```
Example with base `demo`, blueprint `web-tier`:
- `demo-web-tier-web-1` (web component, instance 1)
- `demo-web-tier-web-2` (web component, instance 2)
- `demo-web-tier-app-1` (app component, instance 1)
### Blueprint instance tags
Blueprint-deployed instances receive automatic tags:
| Tag | Example |
|-----|---------|
| `blueprint` | web-tier |
| `blueprint_base` | demo |
| `blueprint_component` | web |
| `ha-group` | demo-web-tier-web |
| `deployed_by` | admin |
| `deployed_at` | 2026-02-23T20:52:19Z |
| `owner` | admin |
The `ha-group` tag groups instances of the same component for HA tracking.
### Tested: deploy web-tier blueprint
Deployed with base name `demo` on `incusbr0` network, `local` storage pool.
All 3 instances created successfully on different nodes (scheduler spread them
across oc-node-01, oc-node-02, oc-node-03). Each instance received correct
tags. IPs auto-assigned from incusbr0 DHCP range.
## Deployed Blueprints (Lifecycle)
Navigate to **Deployed Blueprints** (`/deployedblueprints`). Tracks all
blueprint deployments with lifecycle management.
### Table columns
| Column | Description |
|--------|-------------|
| Blueprint | Blueprint template name |
| Base Name | Instance name prefix |
| Cluster | Target cluster |
| Network | Deployment network |
| Instances | Count (clickable — shows detail modal) |
| Status | deploying, deployed, decommissioning, decommissioned, failed |
| Deployed By | Username |
| Deployed At | Timestamp |
| Actions | Change Owner (admin), Decommission |
### Decommission workflow
1. Click **Decommission** → confirmation dialog
2. `POST /api/deployedblueprints/{id}/decommission/start` — triggers AWX
decommission playbook (if configured)
3. Sequential `DELETE /api/deployedblueprints/{id}/decommission/instance/{name}`
per instance — deletes from cluster one by one
4. `POST /api/deployedblueprints/{id}/decommission/complete` — finalizes record
Progress modal shows per-instance deletion status. Blueprints with active
deployments cannot be deleted from the Blueprint Design page.
### Deployed blueprint API
| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/deployedblueprints` | List all deployments |
| POST | `/api/deployedblueprints/{id}/change-owner` | Transfer ownership |
| POST | `/api/deployedblueprints/{id}/decommission/start` | Start teardown |
| DELETE | `/api/deployedblueprints/{id}/decommission/instance/{name}` | Delete instance |
| POST | `/api/deployedblueprints/{id}/decommission/complete` | Finalize |
## Managing Instances
Navigate to **Manage VMs/Containers** (`/infra`). Shows all instances across
connected clusters with full lifecycle management.
### Instance table columns
| Column | Width | Content |
|--------|-------|---------|
| Manage | 8% | Opens management popup |
| Name | 15% | Instance name |
| Type | 8% | virtual-machine or container |
| Status | 8% | Running, Stopped, Frozen |
| Location | 12% | Cluster member node |
| IPs | 22% | NIC name: IP address |
| Instance Tags | 27% | Key-value metadata (`user.*` namespace) |
### Management actions
**State control:**
- Running → Restart, Stop, Force Stop, Freeze
- Stopped → Start
- Frozen → Unfreeze
**Resource editing:**
- Edit CPU — number or range (e.g., `0-3`). Running VMs offer "Stop & Apply"
- Edit Memory — value + unit. Warns about OOM (containers) or hotplug limits (VMs)
- Resize Disk — grow only, warns about manual filesystem expansion for VMs
- Add/Remove Disk — non-root disks only
**Snapshots:**
- Take Snapshot — named, optional expiration (default 7 days)
- Restore Snapshot — reverts instance to snapshot state
- Delete Snapshot
**Console access:**
- Text Console — WebSocket terminal (xterm.js)
- Exec Shell — shell exec via WebSocket
- VGA Console — SPICE graphical console (VMs only, with Ctrl+Alt+Del)
**Instance Tags** — add, edit, delete key-value metadata. Reserved keys
excluded. Changes committed in batch.
**ACLs** — view applied firewall rules per NIC, with ingress/egress filtering.
### Instance management API
| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/infra/details/{clusterID}/{name}` | Full instance details |
| POST | `/infra/state/{clusterID}/{name}` | Change state |
| DELETE | `/infra/instance/{clusterID}/{name}` | Delete instance |
| POST | `/infra/snapshot/create/{clusterID}/{name}` | Create snapshot |
| POST | `/infra/snapshot/action/{clusterID}/{name}/{snap}` | Restore/delete |
| POST | `/infra/resources/{clusterID}/{name}` | Edit CPU/memory/disk |
| POST | `/infra/disk/{clusterID}/{name}` | Add disk |
| DELETE | `/infra/disk/{clusterID}/{name}/{disk}` | Remove disk |
| POST | `/infra/metadata/{clusterID}/{name}` | Update tags |
| WS | `/infra/console/{clusterID}/{name}` | Text console |
| WS | `/infra/exec/{clusterID}/{name}` | Shell exec |
| WS | `/infra/vga-console/{clusterID}/{name}` | SPICE VGA |
## Recovering OVN After ovn-central Container Move
Moving the `ovn-central` container between nodes causes the OVN northbound
(NB) database to be reset/empty. The southbound (SB) database retains chassis
entries but becomes stale. The `net-prod` OVN network exists in Incus but has
no backing OVN objects. `incus network delete` hangs because Incus tries to
clean non-existent OVN resources.
### Root causes discovered
1. **OVN NB database loss**: moving the ovn-central container creates a fresh
NB database. All logical switches, routers, DNS records, NAT rules, DHCP
options, and HA chassis groups are lost.
2. **Stale OVS bridges**: when Incus creates a physical uplink (UPLINK), it
creates an OVS bridge named `incusovnN` (where N is the network DB ID).
Deleting and recreating the network assigns a new ID, creating a new bridge
while the old one persists in OVS. The old bridge still holds the physical
parent interface mapping.
3. **OVN chassis UUID mismatch**: toggling the OVN service on IncusOS nodes
can change the `system-id` in the OVN SB. But the `system-id` stored in
the local OVS `external_ids` persists. This creates a mismatch where Incus
writes `requested-chassis=<OVS-system-id>` but the SB chassis has a
different name. Ports can't be bound.
### Recovery procedure (tested)
**Phase 1: Remove stale network from Incus DB**
```bash
# Get the stale network ID
incus admin sql <remote>:global \
"SELECT id, name FROM networks WHERE name = 'net-prod'"
# Delete all related DB entries (children first)
incus admin sql <remote>:global "DELETE FROM networks_load_balancers WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_forwards WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_nodes WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_config WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks WHERE id = <ID>"
```
**Phase 2: Proxmox stop/start all nodes**
Required to clear the Incus daemon's in-memory cache of stale OVN state.
Guest reboot is NOT safe on OC-managed nodes (see CLAUDE.md). Proxmox
stop/start IS safe.
```bash
# Stop each VM via Proxmox API, wait 10s, start
# One node at a time, wait for rejoin before next
```
**Phase 3: Clean stale OVS bridges on each node**
Use a privileged container with OVS tools to access the host's OVS database.
**Critical**: mount the host's OVS socket at a different path to avoid the
container's own OVS service masking it.
```bash
incus launch images:debian/12 <remote>:ovs-fix --target <node> \
-c security.privileged=true
incus config device add <remote>:ovs-fix ovs-run disk \
source=/run/openvswitch path=/host-ovs
# Install OVS with auto-start prevention
incus exec <remote>:ovs-fix -- bash -c "
echo -e '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d
chmod +x /usr/sbin/policy-rc.d
apt-get update -qq && apt-get install -y -qq openvswitch-switch
rm /usr/sbin/policy-rc.d
systemctl stop openvswitch-switch"
# Delete old stale bridge (e.g., incusovn3 from the old UPLINK)
incus exec <remote>:ovs-fix -- \
ovs-vsctl --db=unix:/host-ovs/db.sock del-br incusovn3
# Fix bridge mapping to point to new bridge only
incus exec <remote>:ovs-fix -- \
ovs-vsctl --db=unix:/host-ovs/db.sock set Open_vSwitch . \
external-ids:ovn-bridge-mappings="UPLINK:incusovn<NEW_ID>"
# Move container to next node and repeat
```
**Phase 4: Fix chassis UUID mismatch (node-01 specific)**
If a node's OVS `system-id` doesn't match the SB chassis name, delete the
stale SB chassis and let ovn-controller re-register:
```bash
# Check OVS system-id
incus exec <remote>:ovs-fix -- \
ovs-vsctl --db=unix:/host-ovs/db.sock get Open_vSwitch . external-ids:system-id
# Delete stale chassis from SB
incus exec <remote>:ovn-central -- ovn-sbctl chassis-del "<stale-name>"
# ovn-controller re-registers with the correct system-id within seconds
```
**Phase 5: Recreate networks**
```bash
# Recreate UPLINK (must use --target per node, talk to each node directly)
incus network create oc-node-01:UPLINK --type physical parent=mgmt --target oc-node-01
incus network create oc-node-02:UPLINK --type physical parent=mgmt --target oc-node-02
incus network create oc-node-03:UPLINK --type physical parent=mgmt --target oc-node-03
# Finalize
incus network create <remote>:UPLINK --type physical \
ipv4.gateway=192.168.100.1/22 \
ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
dns.nameservers=192.168.100.1
# Create OVN network
incus network create <remote>:net-prod --type=ovn network=UPLINK \
ipv4.address=10.10.10.1/24 ipv4.nat=true \
ipv6.address=auto ipv6.nat=true
```
### Key lessons
- **Never toggle OVN service on IncusOS nodes unnecessarily** — it can change
the `system-id`, creating chassis UUID mismatches.
- **Mount host OVS at `/host-ovs`** (not `/run/openvswitch`) — the container's
own OVS service masks the host's socket at the default path.
- **`incus network create --target` must be sent directly to each node** —
cluster-forwarded `--target` calls can time out.
- **Proxmox stop/start is safe** for OC-managed nodes; guest reboot is NOT.
## Resizing IncusOS Nodes (Proxmox)
Growing IncusOS VMs requires three steps: Proxmox resize, boot (partition
auto-expand), and manual ZFS pool expansion.
### Step 1: Proxmox resize (VM must be stopped)
```bash
# Stop VM via Proxmox API
curl -fsSk -H "Authorization: PVEAPIToken=..." \
-X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/stop"
# Resize disk (non-destructive, grow only)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
-X PUT -d "disk=scsi0&size=100G" \
"https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/resize"
# Set memory (in MiB)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
-X PUT -d "memory=20480" \
"https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/config"
# Start VM
curl -fsSk -H "Authorization: PVEAPIToken=..." \
-X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/start"
```
### Step 2: Partition auto-expand (automatic on boot)
IncusOS uses `systemd-repart` which runs during initrd on every boot. The
`local-data` partition (partition 11) has no `SizeMaxBytes`, so it
automatically grows to fill all remaining disk space. This is automatic —
no action needed.
Verify via the IncusOS API:
```bash
incus query <remote>:/os/1.0/system/resources | python3 -c "
import json,sys; d=json.load(sys.stdin)['storage']
for disk in d['disks']:
for part in disk['partitions']:
if part['partition'] == 11:
print(f'Part 11: {part[\"size\"]/1073741824:.1f} GiB')"
```
### Step 3: ZFS pool expansion (manual — not automatic)
**Critical discovery**: IncusOS creates the ZFS pool without `autoexpand=on`.
After the partition grows, the ZFS pool does NOT automatically use the new
space. The pool's `EXPANDSZ` column shows the available expansion, but it
must be triggered manually.
Since IncusOS has no shell access and no API for `zpool online -e`, use a
**privileged container** to run the ZFS expansion:
```bash
# Create privileged container on the target node
incus launch images:debian/12 <remote>:zfs-expand --target <node> \
-c security.privileged=true
# Add devices for ZFS control and the partition
incus config device add <remote>:zfs-expand zfs unix-char \
source=/dev/zfs path=/dev/zfs
incus config device add <remote>:zfs-expand sda11 unix-block \
source=/dev/sda11 path=/dev/sda11
# Install ZFS tools and create device symlink
incus exec <remote>:zfs-expand -- bash -c "
apt-get update -qq && apt-get install -y -qq zfsutils-linux
mkdir -p /dev/disk/by-id
ln -sf /dev/sda11 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11"
# Verify pool sees expansion available
incus exec <remote>:zfs-expand -- zpool list
# Output shows EXPANDSZ column with available expansion (e.g., 50G)
# Expand the pool
incus exec <remote>:zfs-expand -- zpool online -e local \
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11
# Verify expansion
incus exec <remote>:zfs-expand -- zpool list
# EXPANDSZ should now show "-" and SIZE should reflect the new total
# Clean up
incus delete <remote>:zfs-expand --force
```
**Why this works**: privileged containers share the host kernel. The `/dev/zfs`
character device is the ZFS kernel module's control interface. The container's
`zpool` commands talk directly to the host kernel's ZFS, which manages the
`local` pool. The `/dev/disk/by-id/` symlink must be created inside the
container because the container doesn't have udev populating `/dev/disk/`.
**Warning**: do NOT run `zpool online -e` with a device path that doesn't
exist inside the container. This causes the ZFS pool to go SUSPENDED (I/O
errors). Recovery requires a Proxmox stop/start of the VM (ZFS re-imports
cleanly on boot, no data loss).
### Rolling resize procedure (cluster)
Resize one node at a time to maintain quorum (2 of 3 nodes):
1. **Node-03 first** (lowest risk — fewest workloads)
2. **Node-02** — if it hosts OVN central, move it to another node first
and update OVN config on all nodes
3. **Node-01 last** — if it hosts Aether, stop it, resize, restart, then
start Aether
After each node: verify cluster status (`incus cluster list`), storage
size (`incus storage info <remote>:local --target <node>`), and port 8443
reachability.