incus-contrib/notes/aether-guide.md

# Aether — Management Platform for Incus + OVN

Aether is a web-based management application that connects to Incus clusters
and Operations Center instances. It provides VM/container deployment via
blueprints, OVN network management, NSX firewall rule translation to OVN ACLs,
HAProxy load balancer management, Ansible automation, and RBAC.

## Deployment

### Requirements

- **Image**: Ubuntu Noble (24.04) golden image (~6.6 GiB compressed, 200 GiB
  virtual disk)
- **Resources**: 8 GiB RAM minimum
- **Networking**: needs direct LAN/VLAN access (macvlan recommended)
- **Services**: PostgreSQL (bundled), FFSDN Go application on port 8443

### Deployment on Incus cluster

```bash
# Import image
incus image import sources/aether-golden-image-v6.tar.gz \
    --alias aether-golden-image-v6 <remote>:

# Create VM (200 GiB disk required — qcow2 virtual size)
incus init <remote>:aether-golden-image-v6 <remote>:aether --vm \
    --target <node> --config limits.memory=8GiB -d root,size=200GiB

# Configure macvlan networking for direct VLAN access
incus config device remove <remote>:aether eth0 2>/dev/null
incus config device add <remote>:aether eth0 nic nictype=macvlan parent=mgmt

# Start and wait ~30s for boot
incus start <remote>:aether

# Run post-deploy (configures static IP, regenerates SSH keys, sets up DB)
incus exec <remote>:aether -- /home/ffsdn/post_deploy.sh <IP/PREFIX> <GATEWAY> <DNS>
```

### Disk size considerations

The golden image is a qcow2 with 200 GiB virtual size but only ~11 GiB actual
data. The target node must have at least 200 GiB of allocatable storage (even
if thin-provisioned). On a fresh IncusOS node with a 64 GiB disk, only ~29 GiB
is free after the OS. Nodes with 50 GiB disks (~8.5 GiB free) cannot host
Aether. Target the node with the most available storage.

### Lab deployment details

| Setting | Value |
|---------|-------|
| VM name | `aether` |
| Location | oc-node-01 |
| IP | 192.168.102.160/22 (VLAN 69) |
| Gateway | 192.168.100.1 |
| DNS | 192.168.100.1 |
| Port | 8443 (HTTPS) |
| RAM | 8 GiB |
| Disk | 200 GiB (virtual), ~11 GiB actual |

### Web UI

Access at `https://<IP>:8443`. Default credentials: admin / (set during setup).

### Post-deploy script

`/home/ffsdn/post_deploy.sh` performs 7 steps:
1. Configure network (netplan)
2. Apply network configuration
3. Regenerate SSH host keys
4. Wait for FFSDN to create PostgreSQL user
5. Transfer database ownership
6. Transfer table and sequence ownership
7. Restart FFSDN service

The script **deletes itself** after successful execution.

## Navigation

The sidebar provides access to all features:

| Menu Item | URL Path | Description |
|-----------|----------|-------------|
| Home | `/home` | Dashboard, welcome page |
| Manage Global FW rules | `/global-acls` | Global firewall / ACL management |
| Manage Cluster FW rules | (submenu) | Per-cluster firewall rules |
| Manage RBAC | (submenu) | Role-based access control |
| Manage VMs/Containers | `/infra` | View and manage instances |
| Deploy VM/Container/Blueprint | `/deploy` | Launch new workloads |
| Deployed Blueprints | `/deployedblueprints` | Track blueprint deployments |
| Blueprint Design | `/blueprintdesign` | Create deployment templates |
| Ansible Automation | `/awx-endpoints` | Ansible playbook management |
| Manage INCUS Clusters | `/incus-infra` | Connect and manage Incus clusters |
| Operations Center | `/operationcenter` | Connect to OC instances |
| HAProxy Load Balancers | `/haproxy` | Load balancer configuration |
| Trace Network Flow | `/traceflow` | Network flow analysis (NSX → OVN ACL translation?) |
| View Live GO Log | `/logs/live` | Real-time application logs |
| AETHER Ledger | `/logs` | Audit/activity ledger |
| View Sync Logs | `/synclogs` | Synchronization logs |
| AETHER Health | `/health` | System health checks |
| Change Password | `/change-password` | Change admin password |
| Licensing | `/licensing` | License management |
| Settings | `/settings` | Application settings |

## Initial Setup

### Step 1: Connect Incus Cluster

Navigate to **Manage INCUS Clusters** (`/incus-infra`) → click
**Add/Edit/Delete INCUS clusters from/to AETHER**.

The form requires:
- **Cluster Name**: label for this cluster (e.g., `oc-lab-cluster`)
- **URL**: Incus API endpoint (e.g., `https://192.168.102.141:8443`)
- **Trust Token**: generated on the cluster

Generate a trust token on the Incus cluster:
```bash
incus config trust add <remote>:AETHER
```

Paste the token into the form and click **Add Cluster**. The cluster appears
in the "Current INCUS Clusters" table with its TLS certificate fingerprint
and expiry date.

After adding, select the cluster from the **Select Cluster** dropdown on
the INCUS Infrastructure Management page. This loads the cluster dashboard
with tabs:

| Tab | Description |
|-----|-------------|
| Clustering: Members | Cluster nodes with status, memory, load, roles |
| Clustering: Cluster Groups | Logical groupings of nodes |
| Storage: Pools | Storage pool configuration |
| Storage: Volumes | Storage volume management |
| Profiles | Instance profiles with devices and config |
| Operations | Running and completed operations |
| Warnings | Cluster warnings |
| Settings | Cluster configuration |
| Images | Cached images across the cluster |
| Configuration | Cluster-level config keys |
| Instances | All instances with actions (Start/Stop/Migrate/Console/etc.) |
| Networking | Networks (bridge, OVN, physical) with View/Edit |
| ACLs | Network ACL management |
| Address Sets | Address set management for ACLs |
| OS | IncusOS node management |

The **Instances** tab provides per-instance action buttons: Start, Restart,
Freeze, Unfreeze, Stop, Migrate, Snapshot, Console, Logs, Delete. Instances
can be filtered by type (VM/Container) and status. Each row shows name, type,
status, IPv4, memory usage, image, snapshots, and location (cluster member).

The **Cluster Members** view shows real-time memory and load bars per node,
OVN roles, and an Evacuate action button.

### Step 2: Connect Operations Center

Navigate to **Operations Center** (`/operationcenter`) → click
**+ Add Operations Center**.

The form requires:
- **Name**: label for this OC (e.g., `oc-lab`)
- **URL**: OC API endpoint (e.g., `https://192.168.102.120:8443`)
- **Certificate (PEM)**: client TLS certificate (PEM format)
- **Private Key (PEM)**: matching private key (PEM format)

Unlike Incus (which uses trust tokens), OC uses **mutual TLS** — Aether
connects with a client certificate that must already be in the OC trust store.
Use the same `client.crt` and `client.key` that were injected into the OC
seed during deployment (typically at `~/.config/incus/client.crt` and
`~/.config/incus/client.key`).

Click **Test Connection** to verify — a successful test shows:
`Connected! API: 1.0 (devel)`.

Click **Save** to add the OC. The OC appears in the "Configured Operations
Centers" table with status, fingerprint, last connected timestamp, and
Edit/Test/Delete actions. A green **Manage** badge indicates an active
connection.

### Connection comparison

| | Incus Cluster | Operations Center |
|---|---|---|
| Auth method | Trust token | Mutual TLS (client cert + key) |
| Token/cert source | `incus config trust add` | Existing client cert from `~/.config/incus/` |
| API version | Incus REST API | OC API 1.0 |
| Port | 8443 | 8443 |

## Lab State After Setup

### Connected infrastructure

| Connection | Name | URL | Status |
|------------|------|-----|--------|
| Incus Cluster | oc-lab-cluster | https://192.168.102.141:8443 | Connected |
| Operations Center | oc-lab | https://192.168.102.120:8443 | Connected |

### Cluster members (as seen in Aether)

| Node | URL | Status | RAM | Disk | Roles |
|------|-----|--------|-----|------|-------|
| oc-node-01 | https://192.168.102.140:8443 | Online | 20 GiB | 64G (29.5G pool) | ovn-chassis, database |
| oc-node-02 | https://192.168.102.141:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database |
| oc-node-03 | https://192.168.102.142:8443 | Online | 20 GiB | 100G (64.4G pool) | ovn-chassis, database-leader |

### Instances visible in Aether

| Name | Type | Status | IPv4 | Network | Location | Tags |
|------|------|--------|------|---------|----------|------|
| aether | VM | Running | 192.168.102.160 | macvlan (mgmt) | oc-node-01 | - |
| aether-test-ct | Container | Running | 10.207.217.22 | incusbr0 | oc-node-02 | owner: admin |
| aether-test-vm | VM | Running | 10.207.217.64 | incusbr0 | oc-node-03 | owner: admin |
| demo-web-tier-web-1 | Container | Running | 10.207.217.2 | incusbr0 | oc-node-02 | blueprint: web-tier, ha-group: demo-web-tier-web |
| demo-web-tier-web-2 | Container | Running | 10.207.217.3 | incusbr0 | oc-node-01 | blueprint: web-tier, ha-group: demo-web-tier-web |
| demo-web-tier-app-1 | Container | Running | 10.207.217.4 | incusbr0 | oc-node-03 | blueprint: web-tier |
| ovn-central | Container | Running | 10.207.217.23 | incusbr0 | oc-node-03 | - |

### Storage usage

| Node | Pool Size | Used | Available |
|------|-----------|------|-----------|
| oc-node-01 | 29.5 GiB | 8.9 GiB | 20.6 GiB |
| oc-node-02 | 64.4 GiB | 7.4 GiB | 57.1 GiB |
| oc-node-03 | 64.4 GiB | 7.9 GiB | 56.5 GiB |

### Networks visible in Aether

| Name | Type | IPv4 | Uplink | NAT | Used By |
|------|------|------|--------|-----|---------|
| UPLINK | physical | 192.168.100.1/22 | - | - | 1 |
| incusbr0 | bridge | 10.207.217.1/24 | - | IPv4, IPv6 | 7 |
| meshbr0 | bridge | none | - | IPv6 | 1 |
| net-prod | ovn | 10.10.10.1/24 | UPLINK | IPv4, IPv6 | 0 |

## Deploying Instances

Navigate to **Deploy VM/Container/Blueprint** (`/deploy`). The form uses
progressive disclosure — each section appears after the preceding selection.

### Deploy flow (single instance)

1. **Select Cluster** — dropdown populated from `/deploy/clusters`
2. **Select Type** — radio: Virtual Machine, Container, or Blueprint
3. **Select Image** — dropdown filtered by type, fetched from
   `/deploy/images/{clusterID}?type={type}`
4. **Name** — validated in real-time against the cluster
   (`/deploy/check-name/{clusterID}/{name}`)
5. **Network** — dropdown shows `name (TYPE) - ipv4_network`
6. **IP Address** — available IPs from the chosen network, ping-verified
   before assignment (`/deploy/verify-ip/{clusterID}`)
7. **Resources** — CPU (1-64, default 2), Memory (default 4 GiB), Root Disk
   (default 50 GiB). Memory validated against cluster available memory with
   warning at 90-95% and block at >95%
8. **Storage Pool** — auto-selects pool with most free space
9. **Additional Disks** — optional, VM only. Each disk has size + pool
10. **Instance Tags** — optional key-value pairs (max 10). Reserved keys:
    `owner`, `deployed_by`, `deployed_at` (set automatically)
11. **Deploy** — reserves IP → creates instance → optionally triggers AWX
    post-deploy playbook → releases IP reservation

### Deploy form fields

| Field | Type | Default | Constraints |
|-------|------|---------|-------------|
| Cluster | select | - | Required |
| Instance type | radio | - | virtual-machine, container, blueprint |
| Image | select | - | Filtered by type |
| Name | text | - | `[a-zA-Z][a-zA-Z0-9-]{0,62}`, unique on cluster |
| Network | select | - | Lists all cluster networks with CIDR |
| IP Address | select | - | Available IPs in network, ping-verified |
| CPU | select | 2 | 1, 2, 4, 8, 16, 32, 64 |
| Memory | number + unit | 4 GiB | 1-1024 GiB or 64-1048576 MiB |
| Root Disk | number | 50 GiB | 1-10000 GiB |
| Storage Pool | select | (most free) | Shows driver + available GiB |
| Additional Disks | number + pool | 100 GiB | 1-10000 GiB each, VM only |
| Tags | key-value | - | Max 10, key: `[a-zA-Z0-9._-]+` |

### Instance creation payload

```json
{
    "cluster_id": 52,
    "name": "my-instance",
    "type": "virtual-machine",
    "image_fingerprint": "abc123...",
    "image_alias": "debian-12",
    "network_name": "incusbr0",
    "ip_address": "10.207.217.50",
    "cpu": 2,
    "memory_value": 4,
    "memory_unit": "GiB",
    "root_disk_gib": 50,
    "storage_pool": "local",
    "additional_disks": [{"size_gib": 100, "pool": "local"}],
    "custom_tags": {"department": "engineering"}
}
```

### Tested: deploy container

Deployed `aether-test-ct` (container, 1 CPU, 1 GiB RAM, 10 GiB disk) on
incusbr0 via the deploy form. Instance appeared immediately in the Instances
tab and via `incus list`. Aether auto-tags: `owner`, `deployed_by`,
`deployed_at`.

### Tested: deploy VM

Deployed `aether-test-vm` (VM, 2 CPU, 2 GiB RAM, 20 GiB disk) on incusbr0.
Took ~30s to start (QEMU boot). Same auto-tagging. Both test instances are
on the bridge network (node-local only).

### AWX integration

Aether can trigger Ansible (AWX) playbooks after instance creation and
before decommission. The deploy page shows a "Your Recent Ansible Automation
Jobs" table that auto-refreshes every 10s. AWX template IDs are configured
per blueprint or globally. If AWX is not configured, the step is skipped.

## Blueprint Design

Navigate to **Blueprint Design** (`/blueprintdesign`). Blueprints are
reusable deployment templates that define multiple instance components.

### Blueprint structure

A blueprint contains:
- **Name** — unique, `[a-zA-Z][a-zA-Z0-9-]{0,99}`
- **Description** — free text, max 500 chars
- **Components** — 1+ component definitions (max 20 total instances)
- **AWX Post-Deploy Template ID** — optional, runs after all instances deploy
- **AWX Decommission Template ID** — optional, runs before teardown

Each component defines:

| Field | Type | Default | Constraints |
|-------|------|---------|-------------|
| Name | text | - | Max 50 chars, required |
| Count | number | 1 | 1-10 per component, max 20 total |
| Type | select | virtual-machine | virtual-machine or container |
| Image | select | - | From reference cluster, filtered by type |
| CPU | number | 2 | 1-128 |
| Memory | number + unit | 4 GiB | 1-1024 GiB or 1-1048576 MiB |
| Root Disk | number | 50 GiB | 1-10000 GiB |
| Additional Disks | number[] | - | VM only, 1-10000 GiB each |

Blueprints are **cluster-agnostic** — they store image aliases, not
fingerprints. The "Reference Cluster" is only used to populate the image
dropdown during design. The same blueprint can deploy to any cluster that
has matching image aliases.

### Blueprint JSON

```json
{
    "name": "web-tier",
    "description": "2 web containers + 1 app container",
    "components": [
        {
            "name": "web",
            "count": 2,
            "type": "container",
            "image_alias": "alpine-3.21",
            "cpu": 1,
            "memory_value": 256,
            "memory_unit": "MiB",
            "root_disk_gib": 2
        },
        {
            "name": "app",
            "count": 1,
            "type": "container",
            "image_alias": "alpine-3.21",
            "cpu": 1,
            "memory_value": 512,
            "memory_unit": "MiB",
            "root_disk_gib": 4
        }
    ]
}
```

### Blueprint API

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/blueprints` | List all blueprints |
| POST | `/api/blueprints` | Create blueprint |
| PUT | `/api/blueprints/{id}` | Update blueprint |
| DELETE | `/api/blueprints/{id}` | Delete (blocked if deployments exist) |
| GET | `/api/blueprints/images/{clusterID}` | List images for design |

### Tested: web-tier blueprint

Created blueprint `web-tier` with 2 components:
- `web`: 2x Alpine 3.21 containers, 1 CPU, 256 MiB, 2 GiB disk
- `app`: 1x Alpine 3.21 container, 1 CPU, 512 MiB, 4 GiB disk

Total: 3 instances, 3 vCPUs, 1 GiB memory, 8 GiB storage.

## Deploying Blueprints

When "Blueprint" is selected as instance type on the deploy page, the form
changes:

1. **Select Blueprint** — eligible blueprints for the cluster (checks image
   availability). Shows component preview cards with resource totals
2. **Base Name** — prefix for all instances. Names generated as
   `{base}-{blueprint}-{component}-{N}` (e.g., `demo-web-tier-web-1`).
   Validated against cluster in real-time
3. **Network** — same as single instance, but no individual IP selection.
   Verifies enough available IPs for all instances
4. **Storage Pool** — validates total storage fits
5. **Deploy** — three-phase process:
   - `POST /deploy/blueprint/start` — resolves images, assigns IPs
   - Sequential `POST /deploy/blueprint/{id}/instance` per instance
   - `POST /deploy/blueprint/{id}/complete` — finalizes, triggers AWX

### Blueprint deploy progress

A modal shows real-time step-by-step progress with per-instance status.
Users can:
- **Cancel** — stops creating remaining instances, server rolls back
- **Run in Background** — hides modal, shows indicator in bottom-right
  corner (click to re-open)

### Instance naming convention

```
{base_name}-{blueprint_name}-{component_name}-{N}
```

Example with base `demo`, blueprint `web-tier`:
- `demo-web-tier-web-1` (web component, instance 1)
- `demo-web-tier-web-2` (web component, instance 2)
- `demo-web-tier-app-1` (app component, instance 1)

### Blueprint instance tags

Blueprint-deployed instances receive automatic tags:

| Tag | Example |
|-----|---------|
| `blueprint` | web-tier |
| `blueprint_base` | demo |
| `blueprint_component` | web |
| `ha-group` | demo-web-tier-web |
| `deployed_by` | admin |
| `deployed_at` | 2026-02-23T20:52:19Z |
| `owner` | admin |

The `ha-group` tag groups instances of the same component for HA tracking.

### Tested: deploy web-tier blueprint

Deployed with base name `demo` on `incusbr0` network, `local` storage pool.
All 3 instances created successfully on different nodes (scheduler spread them
across oc-node-01, oc-node-02, oc-node-03). Each instance received correct
tags. IPs auto-assigned from incusbr0 DHCP range.

## Deployed Blueprints (Lifecycle)

Navigate to **Deployed Blueprints** (`/deployedblueprints`). Tracks all
blueprint deployments with lifecycle management.

### Table columns

| Column | Description |
|--------|-------------|
| Blueprint | Blueprint template name |
| Base Name | Instance name prefix |
| Cluster | Target cluster |
| Network | Deployment network |
| Instances | Count (clickable — shows detail modal) |
| Status | deploying, deployed, decommissioning, decommissioned, failed |
| Deployed By | Username |
| Deployed At | Timestamp |
| Actions | Change Owner (admin), Decommission |

### Decommission workflow

1. Click **Decommission** → confirmation dialog
2. `POST /api/deployedblueprints/{id}/decommission/start` — triggers AWX
   decommission playbook (if configured)
3. Sequential `DELETE /api/deployedblueprints/{id}/decommission/instance/{name}`
   per instance — deletes from cluster one by one
4. `POST /api/deployedblueprints/{id}/decommission/complete` — finalizes record

Progress modal shows per-instance deletion status. Blueprints with active
deployments cannot be deleted from the Blueprint Design page.

### Deployed blueprint API

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/deployedblueprints` | List all deployments |
| POST | `/api/deployedblueprints/{id}/change-owner` | Transfer ownership |
| POST | `/api/deployedblueprints/{id}/decommission/start` | Start teardown |
| DELETE | `/api/deployedblueprints/{id}/decommission/instance/{name}` | Delete instance |
| POST | `/api/deployedblueprints/{id}/decommission/complete` | Finalize |

## Managing Instances

Navigate to **Manage VMs/Containers** (`/infra`). Shows all instances across
connected clusters with full lifecycle management.

### Instance table columns

| Column | Width | Content |
|--------|-------|---------|
| Manage | 8% | Opens management popup |
| Name | 15% | Instance name |
| Type | 8% | virtual-machine or container |
| Status | 8% | Running, Stopped, Frozen |
| Location | 12% | Cluster member node |
| IPs | 22% | NIC name: IP address |
| Instance Tags | 27% | Key-value metadata (`user.*` namespace) |

### Management actions

**State control:**
- Running → Restart, Stop, Force Stop, Freeze
- Stopped → Start
- Frozen → Unfreeze

**Resource editing:**
- Edit CPU — number or range (e.g., `0-3`). Running VMs offer "Stop & Apply"
- Edit Memory — value + unit. Warns about OOM (containers) or hotplug limits (VMs)
- Resize Disk — grow only, warns about manual filesystem expansion for VMs
- Add/Remove Disk — non-root disks only

**Snapshots:**
- Take Snapshot — named, optional expiration (default 7 days)
- Restore Snapshot — reverts instance to snapshot state
- Delete Snapshot

**Console access:**
- Text Console — WebSocket terminal (xterm.js)
- Exec Shell — shell exec via WebSocket
- VGA Console — SPICE graphical console (VMs only, with Ctrl+Alt+Del)

**Instance Tags** — add, edit, delete key-value metadata. Reserved keys
excluded. Changes committed in batch.

**ACLs** — view applied firewall rules per NIC, with ingress/egress filtering.

### Instance management API

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/infra/details/{clusterID}/{name}` | Full instance details |
| POST | `/infra/state/{clusterID}/{name}` | Change state |
| DELETE | `/infra/instance/{clusterID}/{name}` | Delete instance |
| POST | `/infra/snapshot/create/{clusterID}/{name}` | Create snapshot |
| POST | `/infra/snapshot/action/{clusterID}/{name}/{snap}` | Restore/delete |
| POST | `/infra/resources/{clusterID}/{name}` | Edit CPU/memory/disk |
| POST | `/infra/disk/{clusterID}/{name}` | Add disk |
| DELETE | `/infra/disk/{clusterID}/{name}/{disk}` | Remove disk |
| POST | `/infra/metadata/{clusterID}/{name}` | Update tags |
| WS | `/infra/console/{clusterID}/{name}` | Text console |
| WS | `/infra/exec/{clusterID}/{name}` | Shell exec |
| WS | `/infra/vga-console/{clusterID}/{name}` | SPICE VGA |

## Recovering OVN After ovn-central Container Move

Moving the `ovn-central` container between nodes causes the OVN northbound
(NB) database to be reset/empty. The southbound (SB) database retains chassis
entries but becomes stale. The `net-prod` OVN network exists in Incus but has
no backing OVN objects. `incus network delete` hangs because Incus tries to
clean non-existent OVN resources.

### Root causes discovered

1. **OVN NB database loss**: moving the ovn-central container creates a fresh
   NB database. All logical switches, routers, DNS records, NAT rules, DHCP
   options, and HA chassis groups are lost.
2. **Stale OVS bridges**: when Incus creates a physical uplink (UPLINK), it
   creates an OVS bridge named `incusovnN` (where N is the network DB ID).
   Deleting and recreating the network assigns a new ID, creating a new bridge
   while the old one persists in OVS. The old bridge still holds the physical
   parent interface mapping.
3. **OVN chassis UUID mismatch**: toggling the OVN service on IncusOS nodes
   can change the `system-id` in the OVN SB. But the `system-id` stored in
   the local OVS `external_ids` persists. This creates a mismatch where Incus
   writes `requested-chassis=<OVS-system-id>` but the SB chassis has a
   different name. Ports can't be bound.

### Recovery procedure (tested)

**Phase 1: Remove stale network from Incus DB**

```bash
# Get the stale network ID
incus admin sql <remote>:global \
  "SELECT id, name FROM networks WHERE name = 'net-prod'"

# Delete all related DB entries (children first)
incus admin sql <remote>:global "DELETE FROM networks_load_balancers WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_forwards WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_nodes WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks_config WHERE network_id = <ID>"
incus admin sql <remote>:global "DELETE FROM networks WHERE id = <ID>"
```

**Phase 2: Proxmox stop/start all nodes**

Required to clear the Incus daemon's in-memory cache of stale OVN state.
Guest reboot is NOT safe on OC-managed nodes (see CLAUDE.md). Proxmox
stop/start IS safe.

```bash
# Stop each VM via Proxmox API, wait 10s, start
# One node at a time, wait for rejoin before next
```

**Phase 3: Clean stale OVS bridges on each node**

Use a privileged container with OVS tools to access the host's OVS database.
**Critical**: mount the host's OVS socket at a different path to avoid the
container's own OVS service masking it.

```bash
incus launch images:debian/12 <remote>:ovs-fix --target <node> \
    -c security.privileged=true
incus config device add <remote>:ovs-fix ovs-run disk \
    source=/run/openvswitch path=/host-ovs

# Install OVS with auto-start prevention
incus exec <remote>:ovs-fix -- bash -c "
    echo -e '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d
    chmod +x /usr/sbin/policy-rc.d
    apt-get update -qq && apt-get install -y -qq openvswitch-switch
    rm /usr/sbin/policy-rc.d
    systemctl stop openvswitch-switch"

# Delete old stale bridge (e.g., incusovn3 from the old UPLINK)
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock del-br incusovn3

# Fix bridge mapping to point to new bridge only
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock set Open_vSwitch . \
    external-ids:ovn-bridge-mappings="UPLINK:incusovn<NEW_ID>"

# Move container to next node and repeat
```

**Phase 4: Fix chassis UUID mismatch (node-01 specific)**

If a node's OVS `system-id` doesn't match the SB chassis name, delete the
stale SB chassis and let ovn-controller re-register:

```bash
# Check OVS system-id
incus exec <remote>:ovs-fix -- \
    ovs-vsctl --db=unix:/host-ovs/db.sock get Open_vSwitch . external-ids:system-id

# Delete stale chassis from SB
incus exec <remote>:ovn-central -- ovn-sbctl chassis-del "<stale-name>"
# ovn-controller re-registers with the correct system-id within seconds
```

**Phase 5: Recreate networks**

```bash
# Recreate UPLINK (must use --target per node, talk to each node directly)
incus network create oc-node-01:UPLINK --type physical parent=mgmt --target oc-node-01
incus network create oc-node-02:UPLINK --type physical parent=mgmt --target oc-node-02
incus network create oc-node-03:UPLINK --type physical parent=mgmt --target oc-node-03
# Finalize
incus network create <remote>:UPLINK --type physical \
    ipv4.gateway=192.168.100.1/22 \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    dns.nameservers=192.168.100.1

# Create OVN network
incus network create <remote>:net-prod --type=ovn network=UPLINK \
    ipv4.address=10.10.10.1/24 ipv4.nat=true \
    ipv6.address=auto ipv6.nat=true
```

### Key lessons

- **Never toggle OVN service on IncusOS nodes unnecessarily** — it can change
  the `system-id`, creating chassis UUID mismatches.
- **Mount host OVS at `/host-ovs`** (not `/run/openvswitch`) — the container's
  own OVS service masks the host's socket at the default path.
- **`incus network create --target` must be sent directly to each node** —
  cluster-forwarded `--target` calls can time out.
- **Proxmox stop/start is safe** for OC-managed nodes; guest reboot is NOT.

## Resizing IncusOS Nodes (Proxmox)

Growing IncusOS VMs requires three steps: Proxmox resize, boot (partition
auto-expand), and manual ZFS pool expansion.

### Step 1: Proxmox resize (VM must be stopped)

```bash
# Stop VM via Proxmox API
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/stop"

# Resize disk (non-destructive, grow only)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X PUT -d "disk=scsi0&size=100G" \
    "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/resize"

# Set memory (in MiB)
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X PUT -d "memory=20480" \
    "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/config"

# Start VM
curl -fsSk -H "Authorization: PVEAPIToken=..." \
    -X POST "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/start"
```

### Step 2: Partition auto-expand (automatic on boot)

IncusOS uses `systemd-repart` which runs during initrd on every boot. The
`local-data` partition (partition 11) has no `SizeMaxBytes`, so it
automatically grows to fill all remaining disk space. This is automatic —
no action needed.

Verify via the IncusOS API:
```bash
incus query <remote>:/os/1.0/system/resources | python3 -c "
import json,sys; d=json.load(sys.stdin)['storage']
for disk in d['disks']:
    for part in disk['partitions']:
        if part['partition'] == 11:
            print(f'Part 11: {part[\"size\"]/1073741824:.1f} GiB')"
```

### Step 3: ZFS pool expansion (manual — not automatic)

**Critical discovery**: IncusOS creates the ZFS pool without `autoexpand=on`.
After the partition grows, the ZFS pool does NOT automatically use the new
space. The pool's `EXPANDSZ` column shows the available expansion, but it
must be triggered manually.

Since IncusOS has no shell access and no API for `zpool online -e`, use a
**privileged container** to run the ZFS expansion:

```bash
# Create privileged container on the target node
incus launch images:debian/12 <remote>:zfs-expand --target <node> \
    -c security.privileged=true

# Add devices for ZFS control and the partition
incus config device add <remote>:zfs-expand zfs unix-char \
    source=/dev/zfs path=/dev/zfs
incus config device add <remote>:zfs-expand sda11 unix-block \
    source=/dev/sda11 path=/dev/sda11

# Install ZFS tools and create device symlink
incus exec <remote>:zfs-expand -- bash -c "
    apt-get update -qq && apt-get install -y -qq zfsutils-linux
    mkdir -p /dev/disk/by-id
    ln -sf /dev/sda11 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11"

# Verify pool sees expansion available
incus exec <remote>:zfs-expand -- zpool list
# Output shows EXPANDSZ column with available expansion (e.g., 50G)

# Expand the pool
incus exec <remote>:zfs-expand -- zpool online -e local \
    scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part11

# Verify expansion
incus exec <remote>:zfs-expand -- zpool list
# EXPANDSZ should now show "-" and SIZE should reflect the new total

# Clean up
incus delete <remote>:zfs-expand --force
```

**Why this works**: privileged containers share the host kernel. The `/dev/zfs`
character device is the ZFS kernel module's control interface. The container's
`zpool` commands talk directly to the host kernel's ZFS, which manages the
`local` pool. The `/dev/disk/by-id/` symlink must be created inside the
container because the container doesn't have udev populating `/dev/disk/`.

**Warning**: do NOT run `zpool online -e` with a device path that doesn't
exist inside the container. This causes the ZFS pool to go SUSPENDED (I/O
errors). Recovery requires a Proxmox stop/start of the VM (ZFS re-imports
cleanly on boot, no data loss).

### Rolling resize procedure (cluster)

Resize one node at a time to maintain quorum (2 of 3 nodes):

1. **Node-03 first** (lowest risk — fewest workloads)
2. **Node-02** — if it hosts OVN central, move it to another node first
   and update OVN config on all nodes
3. **Node-01 last** — if it hosts Aether, stop it, resize, restart, then
   start Aether

After each node: verify cluster status (`incus cluster list`), storage
size (`incus storage info <remote>:local --target <node>`), and port 8443
reachability.