maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/aether-guide.md

33 KiB
Raw Blame History

Aether — Management Platform for Incus + OVN

Aether is a web-based management application that connects to Incus clusters and Operations Center instances. It provides VM/container deployment via blueprints, OVN network management, NSX firewall rule translation to OVN ACLs, HAProxy load balancer management, Ansible automation, and RBAC.

Deployment

Requirements

  • Image: Ubuntu Noble (24.04) golden image (~6.6 GiB compressed, 200 GiB virtual disk)
  • Resources: 8 GiB RAM minimum
  • Networking: needs direct LAN/VLAN access (macvlan recommended)
  • Services: PostgreSQL (bundled), FFSDN Go application on port 8443

Deployment on Incus cluster

# Import image
incus image import sources/aether-golden-image-v6.tar.gz \
    --alias aether-golden-image-v6 <remote>:

# Create VM (200 GiB disk required — qcow2 virtual size)
incus init <remote>:aether-golden-image-v6 <remote>:aether --vm \
    --target <node> --config limits.memory=8GiB -d root,size=200GiB

# Configure macvlan networking for direct VLAN access
incus config device remove <remote>:aether eth0 2>/dev/null
incus config device add <remote>:aether eth0 nic nictype=macvlan parent=mgmt

# Start and wait ~30s for boot
incus start <remote>:aether

# Run post-deploy (configures static IP, regenerates SSH keys, sets up DB)
incus exec <remote>:aether -- /home/ffsdn/post_deploy.sh <IP/PREFIX> <GATEWAY> <DNS>

Disk size considerations

The golden image is a qcow2 with 200 GiB virtual size but only ~11 GiB actual data. The target node must have at least 200 GiB of allocatable storage (even if thin-provisioned). On a fresh IncusOS node with a 64 GiB disk, only ~29 GiB is free after the OS. Nodes with 50 GiB disks (~8.5 GiB free) cannot host Aether. Target the node with the most available storage.

Lab deployment details

Setting Value
VM name aether
Location oc-node-01
IP 192.168.102.160/22 (VLAN 69)
Gateway 192.168.100.1
DNS 192.168.100.1
Port 8443 (HTTPS)
RAM 8 GiB
Disk 200 GiB (virtual), ~11 GiB actual

Web UI

Access at https://<IP>:8443. Default credentials: admin / (set during setup).

REST API

Aether provides a documented REST API with OpenAPI 3.0 spec.

Resource URL
Swagger UI https://<IP>:8443/api/docs
OpenAPI spec https://<IP>:8443/api/swagger.yaml
Auth endpoint POST /api/auth/token

Authentication: JWT Bearer token. Obtain via username/password:

# Get a JWT (valid 24 hours)
AETHER_TOKEN=$(curl -sk -X POST https://192.168.102.160:8443/api/auth/token \
    -H "Content-Type: application/json" \
    -d '{"username":"admin","password":"YOUR_PASSWORD"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

# Use the token
curl -sk https://192.168.102.160:8443/api/clusters \
    -H "Authorization: Bearer $AETHER_TOKEN"

JWT payload includes: sub (username), id (user ID), is_ffsnd_full_admin (bool), is_ffsnd_acl_admin (bool), exp (expiry).

Rate limits: 3 req/s on auth endpoint, 10 req/s on all others. Exceeding returns HTTP 429.

API coverage (v6.4.317, API v1.0.0):

Tag Endpoints Description
Authentication POST /api/auth/token JWT token issuance
Clusters GET /api/clusters List clusters (admin only)
Local ACLs GET/POST /api/clusters/{id}/rules Per-cluster firewall rules
Local ACLs PUT/DELETE /api/clusters/{id}/rules/{rule_id} Update/delete rules
Global ACLs GET/POST /api/global/rules Global firewall rules
Global ACLs PUT/DELETE /api/global/rules/{rule_id} Update/delete global rules

Features not yet in the API (UI-only as of v6.4.317):

  • AWX endpoint registration and cluster AWX binding
  • Instance deployment and management
  • Blueprint design and deployment
  • Inventory, networking, OVN management
  • HAProxy load balancers
  • RBAC user management
  • Health checks

The API is actively developed. Check /api/swagger.yaml in newer Aether versions for additional endpoints — features may move from UI-only to API.

ACL rule schema (source/destination support):

  • IP addresses: 192.168.1.1
  • CIDR: 10.0.0.0/8
  • Metadata expressions: user.key=value, user.key=^prefix, user.key=*substring
  • Boolean logic: (user.env=prod AND user.tier=web) OR user.region=^eu
  • Comma-separated: 192.168.1.1,user.app=web
  • Protocols: tcp, udp, icmp4
  • Actions: allow, drop, reject
  • Direction: ingress, egress, in_out
  • Apply to: instances, dfw_ovn

Post-deploy script

/home/ffsdn/post_deploy.sh performs 7 steps:

  1. Configure network (netplan)
  2. Apply network configuration
  3. Regenerate SSH host keys
  4. Wait for FFSDN to create PostgreSQL user
  5. Transfer database ownership
  6. Transfer table and sequence ownership
  7. Restart FFSDN service

The script deletes itself after successful execution.

Navigation

The sidebar provides access to all features:

Menu Item URL Path Description
Home /home Dashboard, welcome page
Manage Global FW rules /global-acls Global firewall / ACL management
Manage Cluster FW rules (submenu) Per-cluster firewall rules
Manage RBAC (submenu) Role-based access control
Manage VMs/Containers /infra View and manage instances
Deploy VM/Container/Blueprint /deploy Launch new workloads
Deployed Blueprints /deployedblueprints Track blueprint deployments
Blueprint Design /blueprintdesign Create deployment templates
Ansible Automation /awx-endpoints Ansible playbook management
Manage INCUS Clusters /incus-infra Connect and manage Incus clusters
Operations Center /operationcenter Connect to OC instances
HAProxy Load Balancers /haproxy Load balancer configuration
Trace Network Flow /traceflow Network flow analysis (NSX → OVN ACL translation?)
View Live GO Log /logs/live Real-time application logs
AETHER Ledger /logs Audit/activity ledger
View Sync Logs /synclogs Synchronization logs
AETHER Health /health System health checks
Change Password /change-password Change admin password
Licensing /licensing License management
Settings /settings Application settings

Initial Setup

Step 1: Connect Incus Cluster

Navigate to Manage INCUS Clusters (/incus-infra) → click Add/Edit/Delete INCUS clusters from/to AETHER.

The form requires:

  • Cluster Name: label for this cluster (e.g., oc-lab-cluster)
  • URL: Incus API endpoint (e.g., https://192.168.102.141:8443)
  • Trust Token: generated on the cluster

Generate a trust token on the Incus cluster:

incus config trust add <remote>:AETHER

Paste the token into the form and click Add Cluster. The cluster appears in the "Current INCUS Clusters" table with its TLS certificate fingerprint and expiry date.

After adding, select the cluster from the Select Cluster dropdown on the INCUS Infrastructure Management page. This loads the cluster dashboard with tabs:

Tab Description
Clustering: Members Cluster nodes with status, memory, load, roles
Clustering: Cluster Groups Logical groupings of nodes
Storage: Pools Storage pool configuration
Storage: Volumes Storage volume management
Profiles Instance profiles with devices and config
Operations Running and completed operations
Warnings Cluster warnings
Settings Cluster configuration
Images Cached images across the cluster
Configuration Cluster-level config keys
Instances All instances with actions (Start/Stop/Migrate/Console/etc.)
Networking Networks (bridge, OVN, physical) with View/Edit
ACLs Network ACL management
Address Sets Address set management for ACLs
OS IncusOS node management

The Instances tab provides per-instance action buttons: Start, Restart, Freeze, Unfreeze, Stop, Migrate, Snapshot, Console, Logs, Delete. Instances can be filtered by type (VM/Container) and status. Each row shows name, type, status, IPv4, memory usage, image, snapshots, and location (cluster member).

The Cluster Members view shows real-time memory and load bars per node, OVN roles, and an Evacuate action button.

Step 2: Connect Operations Center

Navigate to Operations Center (/operationcenter) → click + Add Operations Center.

The form requires:

  • Name: label for this OC (e.g., oc-lab)
  • URL: OC API endpoint (e.g., https://192.168.102.120:8443)
  • Certificate (PEM): client TLS certificate (PEM format)
  • Private Key (PEM): matching private key (PEM format)

Unlike Incus (which uses trust tokens), OC uses mutual TLS — Aether connects with a client certificate that must already be in the OC trust store. Use the same client.crt and client.key that were injected into the OC seed during deployment (typically at ~/.config/incus/client.crt and ~/.config/incus/client.key).

Click Test Connection to verify — a successful test shows: Connected! API: 1.0 (devel).

Click Save to add the OC. The OC appears in the "Configured Operations Centers" table with status, fingerprint, last connected timestamp, and Edit/Test/Delete actions. A green Manage badge indicates an active connection.

Connection comparison

Incus Cluster Operations Center
Auth method Trust token Mutual TLS (client cert + key)
Token/cert source incus config trust add Existing client cert from ~/.config/incus/
API version Incus REST API OC API 1.0
Port 8443 8443

Lab State After Setup

Connected infrastructure

Connection Name URL Status
Incus Cluster oc-lab-cluster https://192.168.102.141:8443 Connected
Operations Center oc-lab https://192.168.102.120:8443 Connected

Cluster members (as seen in Aether)

Node URL Status RAM Disk Roles
oc-node-01 https://192.168.102.140:8443 Online 20 GiB 64G (29.5G pool) ovn-chassis, database
oc-node-02 https://192.168.102.141:8443 Online 20 GiB 100G (64.4G pool) ovn-chassis, database
oc-node-03 https://192.168.102.142:8443 Online 20 GiB 100G (64.4G pool) ovn-chassis, database-leader

Instances visible in Aether

Name Type Status IPv4 Network Location Tags
aether VM Running 192.168.102.160 macvlan (mgmt) oc-node-01 -
aether-test-ct Container Running 10.207.217.22 incusbr0 oc-node-02 owner: admin
aether-test-vm VM Running 10.207.217.64 incusbr0 oc-node-03 owner: admin
demo-web-tier-web-1 Container Running 10.207.217.2 incusbr0 oc-node-02 blueprint: web-tier, ha-group: demo-web-tier-web
demo-web-tier-web-2 Container Running 10.207.217.3 incusbr0 oc-node-01 blueprint: web-tier, ha-group: demo-web-tier-web
demo-web-tier-app-1 Container Running 10.207.217.4 incusbr0 oc-node-03 blueprint: web-tier
ovn-central Container Running 10.207.217.23 incusbr0 oc-node-03 -

Storage usage

Node Pool Size Used Available
oc-node-01 29.5 GiB 8.9 GiB 20.6 GiB
oc-node-02 64.4 GiB 7.4 GiB 57.1 GiB
oc-node-03 64.4 GiB 7.9 GiB 56.5 GiB

Networks visible in Aether

Name Type IPv4 Uplink NAT Used By
UPLINK physical 192.168.100.1/22 - - 1
incusbr0 bridge 10.207.217.1/24 - IPv4, IPv6 7
meshbr0 bridge none - IPv6 1
net-prod ovn 10.10.10.1/24 UPLINK IPv4, IPv6 0

Deploying Instances

Navigate to Deploy VM/Container/Blueprint (/deploy). The form uses progressive disclosure — each section appears after the preceding selection.

Deploy flow (single instance)

  1. Select Cluster — dropdown populated from /deploy/clusters
  2. Select Type — radio: Virtual Machine, Container, or Blueprint
  3. Select Image — dropdown filtered by type, fetched from /deploy/images/{clusterID}?type={type}
  4. Name — validated in real-time against the cluster (/deploy/check-name/{clusterID}/{name})
  5. Network — dropdown shows name (TYPE) - ipv4_network
  6. IP Address — available IPs from the chosen network, ping-verified before assignment (/deploy/verify-ip/{clusterID})
  7. Resources — CPU (1-64, default 2), Memory (default 4 GiB), Root Disk (default 50 GiB). Memory validated against cluster available memory with warning at 90-95% and block at >95%
  8. Storage Pool — auto-selects pool with most free space
  9. Additional Disks — optional, VM only. Each disk has size + pool
  10. Instance Tags — optional key-value pairs (max 10). Reserved keys: owner, deployed_by, deployed_at (set automatically)
  11. Deploy — reserves IP → creates instance → optionally triggers AWX post-deploy playbook → releases IP reservation

Deploy form fields

Field Type Default Constraints
Cluster select - Required
Instance type radio - virtual-machine, container, blueprint
Image select - Filtered by type
Name text - [a-zA-Z][a-zA-Z0-9-]{0,62}, unique on cluster
Network select - Lists all cluster networks with CIDR
IP Address select - Available IPs in network, ping-verified
CPU select 2 1, 2, 4, 8, 16, 32, 64
Memory number + unit 4 GiB 1-1024 GiB or 64-1048576 MiB
Root Disk number 50 GiB 1-10000 GiB
Storage Pool select (most free) Shows driver + available GiB
Additional Disks number + pool 100 GiB 1-10000 GiB each, VM only
Tags key-value - Max 10, key: [a-zA-Z0-9._-]+

Instance creation payload

{
    "cluster_id": 52,
    "name": "my-instance",
    "type": "virtual-machine",
    "image_fingerprint": "abc123...",
    "image_alias": "debian-12",
    "network_name": "incusbr0",
    "ip_address": "10.207.217.50",
    "cpu": 2,
    "memory_value": 4,
    "memory_unit": "GiB",
    "root_disk_gib": 50,
    "storage_pool": "local",
    "additional_disks": [{"size_gib": 100, "pool": "local"}],
    "custom_tags": {"department": "engineering"}
}

Tested: deploy container

Deployed aether-test-ct (container, 1 CPU, 1 GiB RAM, 10 GiB disk) on incusbr0 via the deploy form. Instance appeared immediately in the Instances tab and via incus list. Aether auto-tags: owner, deployed_by, deployed_at.

Tested: deploy VM

Deployed aether-test-vm (VM, 2 CPU, 2 GiB RAM, 20 GiB disk) on incusbr0. Took ~30s to start (QEMU boot). Same auto-tagging. Both test instances are on the bridge network (node-local only).

AWX integration

Aether can trigger Ansible (AWX) playbooks after instance creation and before decommission. The deploy page shows a "Your Recent Ansible Automation Jobs" table that auto-refreshes every 10s. AWX template IDs are configured per blueprint or globally. If AWX is not configured, the step is skipped.

Blueprint Design

Navigate to Blueprint Design (/blueprintdesign). Blueprints are reusable deployment templates that define multiple instance components.

Blueprint structure

A blueprint contains:

  • Name — unique, [a-zA-Z][a-zA-Z0-9-]{0,99}
  • Description — free text, max 500 chars
  • Components — 1+ component definitions (max 20 total instances)
  • AWX Post-Deploy Template ID — optional, runs after all instances deploy
  • AWX Decommission Template ID — optional, runs before teardown

Each component defines:

Field Type Default Constraints
Name text - Max 50 chars, required
Count number 1 1-10 per component, max 20 total
Type select virtual-machine virtual-machine or container
Image select - From reference cluster, filtered by type
CPU number 2 1-128
Memory number + unit 4 GiB 1-1024 GiB or 1-1048576 MiB
Root Disk number 50 GiB 1-10000 GiB
Additional Disks number[] - VM only, 1-10000 GiB each

Blueprints are cluster-agnostic — they store image aliases, not fingerprints. The "Reference Cluster" is only used to populate the image dropdown during design. The same blueprint can deploy to any cluster that has matching image aliases.

Blueprint JSON

{
    "name": "web-tier",
    "description": "2 web containers + 1 app container",
    "components": [
        {
            "name": "web",
            "count": 2,
            "type": "container",
            "image_alias": "alpine-3.21",
            "cpu": 1,
            "memory_value": 256,
            "memory_unit": "MiB",
            "root_disk_gib": 2
        },
        {
            "name": "app",
            "count": 1,
            "type": "container",
            "image_alias": "alpine-3.21",
            "cpu": 1,
            "memory_value": 512,
            "memory_unit": "MiB",
            "root_disk_gib": 4
        }
    ]
}

Blueprint API

Method Endpoint Purpose
GET /api/blueprints List all blueprints
POST /api/blueprints Create blueprint
PUT /api/blueprints/{id} Update blueprint
DELETE /api/blueprints/{id} Delete (blocked if deployments exist)
GET /api/blueprints/images/{clusterID} List images for design

Tested: web-tier blueprint

Created blueprint web-tier with 2 components:

  • web: 2x Alpine 3.21 containers, 1 CPU, 256 MiB, 2 GiB disk
  • app: 1x Alpine 3.21 container, 1 CPU, 512 MiB, 4 GiB disk

Total: 3 instances, 3 vCPUs, 1 GiB memory, 8 GiB storage.

Deploying Blueprints

When "Blueprint" is selected as instance type on the deploy page, the form changes:

  1. Select Blueprint — eligible blueprints for the cluster (checks image availability). Shows component preview cards with resource totals
  2. Base Name — prefix for all instances. Names generated as {base}-{blueprint}-{component}-{N} (e.g., demo-web-tier-web-1). Validated against cluster in real-time
  3. Network — same as single instance, but no individual IP selection. Verifies enough available IPs for all instances
  4. Storage Pool — validates total storage fits
  5. Deploy — three-phase process:
    • POST /deploy/blueprint/start — resolves images, assigns IPs
    • Sequential POST /deploy/blueprint/{id}/instance per instance
    • POST /deploy/blueprint/{id}/complete — finalizes, triggers AWX

Blueprint deploy progress

A modal shows real-time step-by-step progress with per-instance status. Users can:

  • Cancel — stops creating remaining instances, server rolls back
  • Run in Background — hides modal, shows indicator in bottom-right corner (click to re-open)

Instance naming convention

{base_name}-{blueprint_name}-{component_name}-{N}

Example with base demo, blueprint web-tier:

  • demo-web-tier-web-1 (web component, instance 1)
  • demo-web-tier-web-2 (web component, instance 2)
  • demo-web-tier-app-1 (app component, instance 1)

Blueprint instance tags

Blueprint-deployed instances receive automatic tags:

Tag Example
blueprint web-tier
blueprint_base demo
blueprint_component web
ha-group demo-web-tier-web
deployed_by admin
deployed_at 2026-02-23T20:52:19Z
owner admin

The ha-group tag groups instances of the same component for HA tracking.

Tested: deploy web-tier blueprint

Deployed with base name demo on incusbr0 network, local storage pool. All 3 instances created successfully on different nodes (scheduler spread them across oc-node-01, oc-node-02, oc-node-03). Each instance received correct tags. IPs auto-assigned from incusbr0 DHCP range.

Deployed Blueprints (Lifecycle)

Navigate to Deployed Blueprints (/deployedblueprints). Tracks all blueprint deployments with lifecycle management.

Table columns

Column Description
Blueprint Blueprint template name
Base Name Instance name prefix
Cluster Target cluster
Network Deployment network
Instances Count (clickable — shows detail modal)
Status deploying, deployed, decommissioning, decommissioned, failed
Deployed By Username
Deployed At Timestamp
Actions Change Owner (admin), Decommission

Decommission workflow

  1. Click Decommission → confirmation dialog
  2. POST /api/deployedblueprints/{id}/decommission/start — triggers AWX decommission playbook (if configured)
  3. Sequential DELETE /api/deployedblueprints/{id}/decommission/instance/{name} per instance — deletes from cluster one by one
  4. POST /api/deployedblueprints/{id}/decommission/complete — finalizes record

Progress modal shows per-instance deletion status. Blueprints with active deployments cannot be deleted from the Blueprint Design page.

Deployed blueprint API

Method Endpoint Purpose
GET /api/deployedblueprints List all deployments
POST /api/deployedblueprints/{id}/change-owner Transfer ownership
POST /api/deployedblueprints/{id}/decommission/start Start teardown
DELETE /api/deployedblueprints/{id}/decommission/instance/{name} Delete instance
POST /api/deployedblueprints/{id}/decommission/complete Finalize

Managing Instances

Navigate to Manage VMs/Containers (/infra). Shows all instances across connected clusters with full lifecycle management.

Instance table columns

Column Width Content
Manage 8% Opens management popup
Name 15% Instance name
Type 8% virtual-machine or container
Status 8% Running, Stopped, Frozen
Location 12% Cluster member node
IPs 22% NIC name: IP address
Instance Tags 27% Key-value metadata (user.* namespace)

Management actions

State control:

  • Running → Restart, Stop, Force Stop, Freeze
  • Stopped → Start
  • Frozen → Unfreeze

Resource editing:

  • Edit CPU — number or range (e.g., 0-3). Running VMs offer "Stop & Apply"
  • Edit Memory — value + unit. Warns about OOM (containers) or hotplug limits (VMs)
  • Resize Disk — grow only, warns about manual filesystem expansion for VMs
  • Add/Remove Disk — non-root disks only

Snapshots:

  • Take Snapshot — named, optional expiration (default 7 days)
  • Restore Snapshot — reverts instance to snapshot state
  • Delete Snapshot

Console access:

  • Text Console — WebSocket terminal (xterm.js)
  • Exec Shell — shell exec via WebSocket
  • VGA Console — SPICE graphical console (VMs only, with Ctrl+Alt+Del)

Instance Tags — add, edit, delete key-value metadata. Reserved keys excluded. Changes committed in batch.

ACLs — view applied firewall rules per NIC, with ingress/egress filtering.

Instance management API

Method Endpoint Purpose
GET /infra/details/{clusterID}/{name} Full instance details
POST /infra/state/{clusterID}/{name} Change state
DELETE /infra/instance/{clusterID}/{name} Delete instance
POST /infra/snapshot/create/{clusterID}/{name} Create snapshot
POST /infra/snapshot/action/{clusterID}/{name}/{snap} Restore/delete
POST /infra/resources/{clusterID}/{name} Edit CPU/memory/disk
POST /infra/disk/{clusterID}/{name} Add disk
DELETE /infra/disk/{clusterID}/{name}/{disk} Remove disk
POST /infra/metadata/{clusterID}/{name} Update tags
WS /infra/console/{clusterID}/{name} Text console
WS /infra/exec/{clusterID}/{name} Shell exec
WS /infra/vga-console/{clusterID}/{name} SPICE VGA

Recovering OVN After ovn-central Container Move

Moving the ovn-central container between nodes causes the OVN northbound (NB) database to be reset/empty. The southbound (SB) database retains chassis entries but becomes stale. The net-prod OVN network exists in Incus but has no backing OVN objects. incus network delete hangs because Incus tries to clean non-existent OVN resources.

Root causes discovered

  1. OVN NB database loss: moving the ovn-central container creates a fresh NB database. All logical switches, routers, DNS records, NAT rules, DHCP options, and HA chassis groups are lost.
  2. Stale OVS bridges: when Incus creates a physical uplink (UPLINK), it creates an OVS bridge named incusovnN (where N is the network DB ID). Deleting and recreating the network assigns a new ID, creating a new bridge while the old one persists in OVS. The old bridge still holds the physical parent interface mapping.
  3. OVN chassis UUID mismatch: toggling the OVN service on IncusOS nodes can change the system-id in the OVN SB. But the system-id stored in the local OVS external_ids persists. This creates a mismatch where Incus writes requested-chassis=<OVS-system-id> but the SB chassis has a different name. Ports can't be bound.

Recovery procedure (tested)

Phase 1: Remove stale network from Incus DB

# Get the stale network ID
incus admin sql <remote>:global \
  "SELECT id, name FROM networks WHERE name = 'net-prod'"

# Delete all related DB entries (children first)
incus admin sql <remote>:global "DELETE FROM networks_load_balancers WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_forwards WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_nodes WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_config WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks WHERE id = <ID>"

Phase 2: Proxmox stop/start all nodes

Required to clear the Incus daemon's in-memory cache of stale OVN state. Guest reboot is NOT safe on OC-managed nodes (see CLAUDE.md). Proxmox stop/start IS safe.

# Stop each VM via Proxmox API, wait 10s, start
# One node at a time, wait for rejoin before next

Phase 3: Clean stale OVS bridges on each node

Use a privileged container with OVS tools to access the host's OVS database. Critical: mount the host's OVS socket at a different path to avoid the container's own OVS service masking it.

incus launch images:debian/12 <remote>:ovs-fix --target <node> \
    -c security.privileged=true
incus config device add <remote>:ovs-fix ovs-run disk \
    source=/run/openvswitch path=/host-ovs

# Install OVS with auto-start prevention
incus exec <remote>:ovs-fix -- bash -c "
    echo -e '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d
    chmod +x /usr/sbin/policy-rc.d
    apt-get update -qq && apt-get install -y -qq openvswitch-switch
    rm /usr/sbin/policy-rc.d
    systemctl stop openvswitch-switch"

# Delete old stale bridge (e.g., incusovn3 from the old UPLINK)
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock del-br incusovn3

# Fix bridge mapping to point to new bridge only
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock set Open_vSwitch . \
    external-ids:ovn-bridge-mappings="UPLINK:incusovn<NEW_ID>"

# Move container to next node and repeat

Phase 4: Fix chassis UUID mismatch (node-01 specific)

If a node's OVS system-id doesn't match the SB chassis name, delete the stale SB chassis and let ovn-controller re-register:

# Check OVS system-id
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock get Open_vSwitch . external-ids:system-id

# Delete stale chassis from SB
incus exec <remote>:ovn-central -- ovn-sbctl chassis-del "<stale-name>"
# ovn-controller re-registers with the correct system-id within seconds

Phase 5: Recreate networks

# Recreate UPLINK (must use --target per node, talk to each node directly)
incus network create oc-node-01:UPLINK --type physical parent=mgmt --target oc-node-01
incus network create oc-node-02:UPLINK --type physical parent=mgmt --target oc-node-02
incus network create oc-node-03:UPLINK --type physical parent=mgmt --target oc-node-03
# Finalize
incus network create <remote>:UPLINK --type physical \
    ipv4.gateway=192.168.100.1/22 \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    dns.nameservers=192.168.100.1

# Create OVN network
incus network create <remote>:net-prod --type=ovn network=UPLINK \
    ipv4.address=10.10.10.1/24 ipv4.nat=true \
    ipv6.address=auto ipv6.nat=true

Key lessons

  • Never toggle OVN service on IncusOS nodes unnecessarily — it can change the system-id, creating chassis UUID mismatches.
  • Mount host OVS at /host-ovs (not /run/openvswitch) — the container's own OVS service masks the host's socket at the default path.
  • incus network create --target must be sent directly to each node — cluster-forwarded --target calls can time out.
  • Proxmox stop/start is safe for OC-managed nodes; guest reboot is NOT.

Resizing IncusOS Nodes (Proxmox)

Growing IncusOS VMs requires three steps: Proxmox resize, boot (partition auto-expand), and manual ZFS pool expansion.

Step 1: Proxmox resize (VM must be stopped)

# Stop VM via Proxmox API
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/stop"

# Resize disk (non-destructive, grow only)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X PUT -d "disk=scsi0&size=100G" \
    "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/resize"

# Set memory (in MiB)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X PUT -d "memory=20480" \
    "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/config"

# Start VM
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/start"

Step 2: Partition auto-expand (automatic on boot)

IncusOS uses systemd-repart which runs during initrd on every boot. The local-data partition (partition 11) has no SizeMaxBytes, so it automatically grows to fill all remaining disk space. This is automatic — no action needed.

Verify via the IncusOS API:

incus query <remote>:/os/1.0/system/resources | python3 -c "
import json,sys; d=json.load(sys.stdin)['storage']
for disk in d['disks']:
    for part in disk['partitions']:
        if part['partition'] == 11:
            print(f'Part 11: {part[\"size\"]/1073741824:.1f} GiB')"

Step 3: ZFS pool expansion (manual — not automatic)

Critical discovery: IncusOS creates the ZFS pool without autoexpand=on. After the partition grows, the ZFS pool does NOT automatically use the new space. The pool's EXPANDSZ column shows the available expansion, but it must be triggered manually.

Since IncusOS has no shell access and no API for zpool online -e, use a privileged container to run the ZFS expansion:

# Create privileged container on the target node
incus launch images:debian/12 <remote>:zfs-expand --target <node> \
    -c security.privileged=true

# Add devices for ZFS control and the partition
incus config device add <remote>:zfs-expand zfs unix-char \
    source=/dev/zfs path=/dev/zfs
incus config device add <remote>:zfs-expand sda11 unix-block \
    source=/dev/sda11 path=/dev/sda11

# Install ZFS tools and create device symlink
incus exec <remote>:zfs-expand -- bash -c "
    apt-get update -qq && apt-get install -y -qq zfsutils-linux
    mkdir -p /dev/disk/by-id
    ln -sf /dev/sda11 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11"

# Verify pool sees expansion available
incus exec <remote>:zfs-expand -- zpool list
# Output shows EXPANDSZ column with available expansion (e.g., 50G)

# Expand the pool
incus exec <remote>:zfs-expand -- zpool online -e local \
    scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11

# Verify expansion
incus exec <remote>:zfs-expand -- zpool list
# EXPANDSZ should now show "-" and SIZE should reflect the new total

# Clean up
incus delete <remote>:zfs-expand --force

Why this works: privileged containers share the host kernel. The /dev/zfs character device is the ZFS kernel module's control interface. The container's zpool commands talk directly to the host kernel's ZFS, which manages the local pool. The /dev/disk/by-id/ symlink must be created inside the container because the container doesn't have udev populating /dev/disk/.

Warning: do NOT run zpool online -e with a device path that doesn't exist inside the container. This causes the ZFS pool to go SUSPENDED (I/O errors). Recovery requires a Proxmox stop/start of the VM (ZFS re-imports cleanly on boot, no data loss).

Rolling resize procedure (cluster)

Resize one node at a time to maintain quorum (2 of 3 nodes):

  1. Node-03 first (lowest risk — fewest workloads)
  2. Node-02 — if it hosts OVN central, move it to another node first and update OVN config on all nodes
  3. Node-01 last — if it hosts Aether, stop it, resize, restart, then start Aether

After each node: verify cluster status (incus cluster list), storage size (incus storage info <remote>:local --target <node>), and port 8443 reachability.