incus-contrib/notes/production-lab-guide.md

# Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center
dashboard, OC-managed 3-node cluster with OVN overlay networking, Aether
management platform, HAProxy load balancing, AWX lifecycle automation,
Prometheus/Grafana/Loki observability stack, live migration, network
security, and cluster lifecycle management.

All commands and output in this guide are from actual deployments on
Proxmox VE 9.1.5 with IncusOS 202602230420, Incus client 6.21,
Operations Center v0.3.0, and Aether v6.4.317.


## Section 0: Architecture Overview

### Network Topology

```mermaid
flowchart TD
    vlan(("VLAN 69<br/>192.168.100.0/22"))

    subgraph mgmt["Management"]
        oc["oc-server<br/>VMID 920 · .120<br/>Operations Center"]
    end

    subgraph cluster["Incus Cluster"]
        n1["oc-node-01<br/>VMID 400 · .140<br/>Aether · OVN central"]
        n2["oc-node-02<br/>VMID 401 · .141<br/>AWX · Monitoring"]
        n3["oc-node-03<br/>VMID 402 · .142<br/>HAProxy backends"]
    end

    subgraph services["Macvlan Services"]
        aether["Aether · .160"]
        awx["AWX · .161"]
    end

    subgraph ovn["OVN · net-prod 10.10.10.0/24"]
        haproxy["HAProxy HA<br/>.50 · .51"]
        backends["nginx backends<br/>.60 · .61 · .62"]
        monitoring["Observability<br/>.70 + node-exp .71-.73"]
    end

    subgraph uplink["UPLINK · 192.168.103.x"]
        vip[".200 HAProxy VIP"]
        fwd[".201 Grafana / Prometheus"]
    end

    vlan --- mgmt & cluster
    cluster --> services & ovn
    ovn -.-> uplink

    classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
    classDef serviceClass fill:#E69F00,color:#fff,stroke:#b87d00
    classDef ovnClass fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 nodeClass
    class oc mgmtClass
    class aether,awx serviceClass
    class haproxy,backends,monitoring ovnClass
    class vip,fwd,vlan networkClass

    style mgmt fill:#f5e6f0,stroke:#CC79A7
    style cluster fill:#e6f5f0,stroke:#009E73
    style services fill:#fef3e0,stroke:#E69F00
    style ovn fill:#e0f2fe,stroke:#56B4E9
    style uplink fill:#e0eef8,stroke:#0072B2
```

### Infrastructure

| Component | VMID | IP | Cores | RAM | Disk | Role |
|-----------|------|-----|-------|-----|------|------|
| oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center |
| oc-node-01 | 400 | 192.168.102.140/22 | 4 | 20 GiB | 250G | Cluster init + Aether host |
| oc-node-02 | 401 | 192.168.102.141/22 | 4 | 20 GiB | 100G | AWX + monitoring host |
| oc-node-03 | 402 | 192.168.102.142/22 | 4 | 20 GiB | 100G | HAProxy backends |

**RAM budget**: 64 GiB of 94 GiB (68%). Host: i9-13900HK, 94 GiB RAM,
881 GiB ZFS pool. Leaves 30 GiB headroom for other VMs on the host.

### Inner Cluster Services

| Instance | Network | IP | Node | RAM | Description |
|----------|---------|-----|------|-----|-------------|
| ovn-central | incusbr0 | DHCP | node-01 | 512 MiB | OVN NB/SB databases |
| aether | macvlan mgmt | 192.168.102.160 | node-01 | 8 GiB | Management platform |
| awx | macvlan mgmt | 192.168.102.161 | node-02 | 8 GiB | Ansible automation |
| ha-web-01 | net-prod | 10.10.10.60 | node-01 | 256 MiB | Nginx backend |
| ha-web-02 | net-prod | 10.10.10.61 | node-02 | 256 MiB | Nginx backend |
| ha-web-03 | net-prod | 10.10.10.62 | node-03 | 256 MiB | Nginx backend |
| haproxy-01 | net-prod | 10.10.10.50 | varies | 512 MiB | HA load balancer |
| haproxy-02 | net-prod | 10.10.10.51 | varies | 512 MiB | HA load balancer |
| monitoring | net-prod | 10.10.10.70 | node-02 | 2 GiB | Prometheus + Grafana + Loki |
| node-exp-01 | net-prod | 10.10.10.71 | node-01 | 128 MiB | Host metrics exporter |
| node-exp-02 | net-prod | 10.10.10.72 | node-02 | 128 MiB | Host metrics exporter |
| node-exp-03 | net-prod | 10.10.10.73 | node-03 | 128 MiB | Host metrics exporter |

### External IP Allocation

OVN external addresses from UPLINK range (192.168.103.200-210). Exclude
these from your DHCP server's range:

| IP | Purpose |
|-----|---------|
| 192.168.103.200 | HAProxy VIP (OVN load balancer → haproxy-01/02) |
| 192.168.103.201 | Observability forward (Grafana :3000, Prometheus :9090) |

### Decision Rationale

**Why OC-managed clustering?** OC `provisioning cluster add` is the
production path for Incus deployments. It handles cluster formation,
update management, and inventory centrally. The deploy scripts
(`deploy-haproxy`, `deploy-awx`, `deploy-observability`) are built
for the `oc-node-*` naming and IP scheme.

**Why 20 GiB RAM per node?** Aether requires 8 GiB. AWX requires
4-8 GiB. Monitoring + HAProxy + backends need ~3 GiB total.
Leaves headroom for mixed workloads and live migration.

**Why 250 GiB disk for node-01?** Aether's golden image is 200 GiB
virtual (qcow2). With ZFS thin provisioning only ~11 GiB is used
initially, but the pool needs 200 GiB allocatable space.

**Why OVN?** Bridge networks are node-local — instances on different
nodes cannot communicate. OVN provides cross-node L2 overlay with
sub-ms latency, network isolation, ACLs, load balancers, and network
forwards — essential for HAProxy HA and distributed workloads.

**Why VLAN 69?** Isolates lab traffic from the production LAN. All VMs
share VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the
Proxmox level — IncusOS and workloads are unaware of it.

### Cross-References

This guide integrates techniques from the deep-dive guides:

- [Clustering Guide](clustering-guide.md) — manual cluster formation reference
- [Networking Guide](networking-guide.md) — OVN setup, ACLs, peering
- [Operations Center Guide](operations-center-guide.md) — OC provisioning, CLI, web UI
- [Aether Guide](aether-guide.md) — management platform deployment and API
- [HAProxy Guide](haproxy-guide.md) — HA load balancing with Aether
- [AWX Guide](awx-guide.md) — Ansible lifecycle automation
- [Observability Guide](observability-guide.md) — Prometheus, Grafana, Loki stack


## Section 1: Prerequisites

### Required Tools

Verify all tools are available before starting:

```bash
incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1
```

**Minimum versions**: Incus client 6.3+, Operations Center v0.3.0+.

### Aether Browser Automation

Several Aether features (HAProxy management, blueprint deployment) are
not in the JWT API — they use session-authenticated routes with CSRF
protection. Playwright browser automation is required for Sections 7-9:

```bash
node --version         # Node.js 18+
npx playwright --version
```

Install if missing:

```bash
npm install playwright @playwright/mcp
npx playwright install chromium
```

The Playwright MCP server (configured in `.mcp.json`) provides browser
tools when available. The `incusos/helpers/aether-browser` script is
the standalone alternative.

### Aether Golden Image

The Aether golden image must be available locally before Section 7:

```bash
ls -la sources/aether-golden-image-v6.tar.gz
```

This is an Ubuntu Noble (24.04) image with 200 GiB virtual disk (~6.6
GiB compressed). Obtain it from the Aether distribution.

### Proxmox Configuration

Your `incusos/proxmox.yaml` should contain:

```yaml
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab
```

The `env` file at the repository root must export `PROXMOX_TOKEN_SECRET`
and `AETHER_ADMIN_PASSWORD`. Scripts auto-discover them.

### Client Certificates

Incus client certificates are used for Incus, OC, and Prometheus
(metrics scraping) connections:

```bash
# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key
```

Copy certs for OC CLI:

```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```

For OC web UI browser access, generate a PKCS#12 bundle:

```bash
openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"
```

Import `client.pfx` into your browser's certificate store (Firefox:
Settings → Privacy & Security → View Certificates → Import).

### Doctor Check

Run the environment check to verify everything is in order:

```bash
cd incusos
./incusos-proxmox --doctor
```

Expected output includes tool versions, IncusOS CDN reachability,
`proxmox.yaml` discovery, and Proxmox API connectivity.


## Section 2: Deploy OC Server

### Configuration File

```yaml
# incusos/examples/lab-oc-deploy.yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
    ip: 192.168.102.120/22
```

### Deploy

```bash
./incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
```

Actual output (key lines):

```
[ok]  VM 'oc-server' created (VMID 920)
[ok]  VM 'oc-server' installed and running at 192.168.102.120
```

### Set Up OC CLI Remote

```bash
# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab
```

**Important**: The OC CLI does **not** support the `remote:` suffix syntax
that the Incus CLI uses. Switch to the remote first, then run commands
without a remote suffix.

### Verify OC

```bash
operations-center admin os show
```

Actual output (uptime will vary):

```
WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602230420"
  os_version_next: ""
  uptime: 63
```

### Wait for Updates

OC downloads IncusOS update packages from upstream. At least one update
must reach `ready` state before ISOs can be generated:

```bash
# Poll until at least one update shows "ready"
operations-center provisioning update list
```

Actual output (after ~8 minutes; UUIDs are stable across deployments):

| UUID | Origin | Channels | Version | Severity | Status |
|------|--------|----------|---------|----------|--------|
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready |
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready |
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready |

The table also includes `Upstream Channels` and `Published At` columns
(omitted for width). Not all updates may be ready simultaneously — at
least one `ready` is sufficient to proceed.

### Web UI Access

Open `https://192.168.102.120:8443/ui/` in your browser (with client.pfx
imported from Section 1). The web UI provides a dashboard view of the OC
server, update status, provisioning tokens, and system configuration.

---

## Section 3: Provision Nodes

### 3.1 Create Provisioning Token

```bash
operations-center provisioning token add --uses 5 --description "Production lab cluster"
operations-center provisioning token list
```

Actual output (UUID changes every run):

| UUID | Uses Remaining | Expire At | Channel | Description |
|------|----------------|-----------|---------|-------------|
| <token-UUID> | 5 | <30 days from now> | stable | Production lab cluster |

Save the `<token-UUID>` — you'll need it for the next steps.

### 3.2 Create Token Seed (No force_reboot)

**Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
`incusos-proxmox` handles the install lifecycle externally (blockstat
detection + media removal). `force_reboot` triggers SysRq-B which causes
the crontab bug (~50% failure rate).

```yaml
# /tmp/oc-preseed.yaml
install:
  version: "1"
  force_install: true
```

**Important**: use the structured format with section keys (`install:`). A
flat format (`version: "1"` at root) maps fields to empty `{}` and they
don't get assigned to any section.

```bash
operations-center provisioning token seed add <token-UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
```

### 3.3 Generate OC-Provisioned ISO (Older Version)

**Critical discovery**: nodes deployed from an ISO matching the latest OC
update version are tracked as `needs_update: true` by OC because the OS
was never delivered through OC's update pipeline. The fix: generate the ISO
from an older channel so OC can push the real update after deployment.

```bash
# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
    --description "Older stable versions for initial provisioning"

# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable

# Generate ISO from the older channel
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
```

Actual output:

```
Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"
```

The ISO contains IncusOS 202602210344 (one version behind). OC will push
the latest (202602230420) after nodes register.

### 3.4 Node Configuration

```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 400

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false
    disk: 250
    ip: 192.168.102.140/22

  - name: oc-node-02
    app: incus
    apply_defaults: false
    ip: 192.168.102.141/22

  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22
```

**Key decisions**:

- **20 GiB RAM** per node: Aether needs 8 GiB, AWX needs 4-8 GiB
- **250 GiB disk** for node-01: hosts Aether's 200 GiB virtual image
- **100 GiB disk** for nodes 02-03: sufficient for AWX, monitoring, HAProxy
- **`apply_defaults: false`** for all nodes: OC's Terraform handles
  resource creation during cluster formation

### 3.5 Deploy Nodes (Hybrid Approach)

The hybrid approach uses `incusos-proxmox --iso` to combine OC
auto-registration (from the boot ISO token) with `incusos-proxmox` VM
creation, per-node SEED_DATA (hostname, static IP), install monitoring,
and media cleanup.

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml
```

Actual output (key lines):

```
[ok]  ISO uploaded: IncusOS-oc.iso
[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
[ok]  Remote 'oc-node-01' added (192.168.102.140)
[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
[ok]  Remote 'oc-node-02' added (192.168.102.141)
[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
[ok]  Remote 'oc-node-03' added (192.168.102.142)
[ok]  All post-deployment checks passed
```

All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.

### 3.6 Verify Auto-Registration

Nodes auto-register with OC within ~30 seconds of first boot. The update
from 202602210344 to 202602230420 happens automatically:

```bash
operations-center provisioning server list
```

Actual output (key columns; full table includes Type, Channel, Certificate
Fingerprint, Public Connection URL, Last Updated, Last Seen):

| Cluster | Name | Connection URL | Status | Update Status |
|---------|------|----------------|--------|---------------|
| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pending |

**Key**: all 3 nodes show **"up to date"** because OC delivered the
202602230420 update through its pipeline. This is what unlocks clustering.
Nodes may already be up to date by the time the last node finishes
deploying — the update gets pushed while `incusos-proxmox` deploys
subsequent nodes sequentially.

### 3.7 Verify Scrub Schedules

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${node}:/os/1.0/system/storage | python3 -c \
        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done
```

Actual output:

```
oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0
```

All healthy. No crontab bug (force_reboot was not used).

---

## Section 4: Form Cluster via Operations Center

### 4.1 The needs_update Blocker

OC requires all nodes to show `needs_update: false` before clustering.
Nodes deployed from an ISO matching the latest version are tracked as
`needs_update: true` because the OS was never delivered through OC's update
pipeline. The `needs_update` flag is server-side computed and cannot be
overridden via REST API PUT.

**Solution**: deploy from an older ISO version (Section 3.3). OC then pushes
the real update to nodes through its pipeline, clearing the flag.

### 4.2 Form Cluster

**Important**: if the client certificate was already injected via SEED_DATA,
use an empty application seed config to avoid "Certificate already in trust
store" Terraform errors:

```bash
echo '{}' > /tmp/oc-app-config.yaml

operations-center provisioning cluster add oc-cluster \
    https://192.168.102.140:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on oc-node-01
3. Joins oc-node-02 and oc-node-03
4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`)
5. Runs Terraform/OpenTofu for post-cluster configuration

### 4.3 Fix Remotes After Clustering

Clustering regenerates TLS certificates. Re-add the remotes:

```bash
incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
```

### 4.4 Verify Cluster

```bash
incus cluster list oc-node-01:
```

Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):

| NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE |
|------|-----|-------|--------------|--------|---------|
| oc-node-01 | https://192.168.102.140:8443 | database-leader, database | x86_64 | ONLINE | Fully operational |
| oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational |
| oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational |

All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added
later in Section 6.4.

### 4.5 Cluster Resources Created by OC

```bash
incus storage list oc-node-01:
incus network list oc-node-01:
```

Actual output (incusbr0 subnet varies per deployment):

| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
|------|--------|-------------|---------|-------|
| local | zfs | Local storage pool (on system drive) | 8 | CREATED |

| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|------|------|---------|------|-------------|---------|
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |

OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
`meshbr0` (OC-specific mesh network for inter-node communication). The
table also includes IPv6 and STATE columns.

---

## Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are
node-local. This demonstrates why OVN is needed.

### Same-Node Communication

Launch 2 containers on the same node. **Important**: use `--target` to force
placement — without it, the cluster scheduler may place containers on
different nodes automatically.

**Important**: launch containers **one at a time**, not chained with `&&`.
The first launch on a fresh cluster downloads the image (~1 GB), which
takes 2-3 minutes. Subsequent launches on the same node use the cached
image and are instant. Launches targeting a different node trigger another
image transfer to that node.

```bash
incus launch images:debian/12 oc-node-01:test-bridge-a --target oc-node-01
incus launch images:debian/12 oc-node-01:test-bridge-b --target oc-node-01
```

Wait for them to get IPs:

```bash
incus list oc-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping between them:

```bash
IP_B=$(incus list oc-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 "$IP_B"
```

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

### Cross-Node Communication (Fails)

Launch a container on a different node:

```bash
incus launch images:debian/12 oc-node-01:test-bridge-c --target oc-node-02
```

Wait for IP:

```bash
incus list oc-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping from node-01 to node-02:

```bash
IP_C=$(incus list oc-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"
```

Actual result: **100% packet loss**. Bridge networks are node-local — there is
no L2 path between `incusbr0` on node-01 and `incusbr0` on node-02. Each
node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are
separate L2 domains.

### Internet Access

NAT to the internet works from any node:

```bash
incus exec oc-node-01:test-bridge-a -- ping -c 3 1.1.1.1
```

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via
the host's management interface.

### Cleanup

```bash
incus delete oc-node-01:test-bridge-a --force
incus delete oc-node-01:test-bridge-b --force
incus delete oc-node-01:test-bridge-c --force
```


## Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this
section, containers on any node can communicate transparently.

### 6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

```bash
incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
```

Install OVN:

```bash
incus exec oc-node-01:ovn-central -- bash -c \
    "apt-get update -qq && apt-get install -y -qq ovn-central"
```

Configure OVN to listen on all interfaces:

```bash
incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
```

Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:

```bash
incus config device add oc-node-01:ovn-central \
    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642
```

### 6.2 Enable OVN on All IncusOS Nodes

Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`).
The `database` field is the **southbound** DB (port 6642), not northbound.

```bash
for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
        \"config\": {
            \"database\": \"tcp:192.168.102.140:6642\",
            \"enabled\": true,
            \"tunnel_address\": \"${node_ip}\",
            \"tunnel_protocol\": \"geneve\"
        },
        \"state\": {}
    }"
done
```

Each call should return `{}` on success.

### 6.3 Configure Incus OVN Connection

Point Incus to the **northbound** DB (port 6641):

```bash
incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641
```

### 6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the `ovn-chassis` role:

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role add oc-node-01:${node} ovn-chassis
done
```

Verify:

```bash
incus cluster list oc-node-01:
```

The ROLES column should now include `ovn-chassis` for each member.

### 6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and
the physical LAN. It uses the two-step cluster pattern: per-member
`--target` first, then cluster-wide create.

**Important**: IncusOS names its management NIC `mgmt`, NOT `ens18`. Using
`parent=ens18` will fail with "Parent interface 'ens18' not found".

```bash
# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done

# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1
```

### 6.6 Create OVN Network (net-prod)

```bash
incus network create oc-node-01:net-prod --type=ovn \
    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true
```

Actual output:

```
Network net-prod created
```

net-prod is assigned external IP `192.168.103.200` from the UPLINK range.

### 6.7 Verify Cross-Node OVN Connectivity

```bash
incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
```

Actual output:

```
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms
```

Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
containers after verification:

```bash
incus delete oc-node-01:test-1 oc-node-01:test-2 --force
```

### 6.8 Final Network State

```bash
incus network list oc-node-01:
```

Actual output:

| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|------|------|---------|------|-------------|---------|
| UPLINK | physical | YES | | | 1 |
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
| net-prod | ovn | YES | 10.10.10.1/24 | | 0 |

The incusbr0 subnet is randomly assigned per deployment. The USED BY count
for net-prod is 0 at this point (test containers deleted); it increases as
workloads are added in subsequent sections.

---

## Section 7: Mixed Workloads

Deploy a realistic workload mix: web servers, application containers, and
VMs configured for live migration.

### 7.1 Containers on net-prod

Deploy containers with targeted placement across nodes. Run each launch
command **one at a time** — each new target node needs to download the
image from the cluster (~1 GB transfer, 2-3 minutes per node):

```bash
incus launch images:debian/12 oc-node-01:prod-web-01 --network net-prod --target oc-node-01
incus launch images:debian/12 oc-node-01:prod-web-02 --network net-prod --target oc-node-02
incus launch images:debian/12 oc-node-01:prod-api-01 --network net-prod --target oc-node-03
```

Install nginx on the web servers:

```bash
incus exec oc-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec oc-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx"
```

Set distinct content to verify load balancing later:

```bash
incus exec oc-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html"
incus exec oc-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html"
```

Install nginx on the API container:

```bash
incus exec oc-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec oc-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html"
```

### 7.2 VMs (Migration-Ready)

Deploy VMs with live migration configuration:

```bash
incus launch images:debian/12 oc-node-01:prod-db-01 --vm --network net-prod --target oc-node-01
incus launch images:debian/12 oc-node-01:prod-app-01 --vm --network net-prod --target oc-node-02
```

VMs may take longer to boot than containers (~30-60s for image download +
boot). If the VMs show as STOPPED, start them explicitly:

```bash
incus start oc-node-01:prod-db-01
incus start oc-node-01:prod-app-01
```

Wait for the VM agent to become available, then verify:

```bash
# Check VM agent is running
incus exec oc-node-01:prod-db-01 -- uname -a
incus exec oc-node-01:prod-app-01 -- uname -a
```

### 7.3 Configure VMs for Live Migration

**Critical**: use `limits.cpu` as a **range** (e.g., `0-1`), not an integer.
Without the range, QEMU sets `maxcpus` based on the host's CPU count, which
varies across nodes and breaks migration with `Missing section footer for
ICH9LPC`.

Stop VMs before configuring `migration.stateful`:

```bash
incus stop oc-node-01:prod-db-01
incus stop oc-node-01:prod-app-01
```

Configure migration settings:

```bash
# prod-db-01
incus config set oc-node-01:prod-db-01 limits.cpu=0-1
incus config set oc-node-01:prod-db-01 migration.stateful=true
incus config device override oc-node-01:prod-db-01 root size.state=2GiB

# prod-app-01
incus config set oc-node-01:prod-app-01 limits.cpu=0-1
incus config set oc-node-01:prod-app-01 migration.stateful=true
incus config device override oc-node-01:prod-app-01 root size.state=2GiB
```

**Important**: use `device override` (not `device set`) because the `root`
device comes from the default profile. `device set` fails with "Device from
profile(s) cannot be modified for individual instance".

Start the VMs:

```bash
incus start oc-node-01:prod-db-01
incus start oc-node-01:prod-app-01
```

### 7.4 Workload Distribution

View the full workload distribution:

```bash
incus list oc-node-01: --columns nstL4 --format table
```

Expected layout:

| NAME        | STATE   | TYPE            | LOCATION    | IPV4                  |
|-------------|---------|-----------------|-------------|-----------------------|
| ovn-central | RUNNING | CONTAINER       | oc-node-01 | ...                   |
| prod-web-01 | RUNNING | CONTAINER       | oc-node-01 | 10.10.10.x (net-prod) |
| prod-db-01  | RUNNING | VIRTUAL-MACHINE | oc-node-01 | 10.10.10.x (net-prod) |
| prod-web-02 | RUNNING | CONTAINER       | oc-node-02 | 10.10.10.x (net-prod) |
| prod-app-01 | RUNNING | VIRTUAL-MACHINE | oc-node-02 | 10.10.10.x (net-prod) |
| prod-api-01 | RUNNING | CONTAINER       | oc-node-03 | 10.10.10.x (net-prod) |


## Section 8: Network Isolation & Security

### 8.1 Create Isolated Network

```bash
incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none
```

### 8.2 Launch Isolated Containers

```bash
incus launch images:debian/12 oc-node-01:iso-app-01 --network net-isolated --target oc-node-01
incus launch images:debian/12 oc-node-01:iso-app-02 --network net-isolated --target oc-node-02
```

### 8.3 Verify Network Isolation

Containers on net-isolated can reach each other:

```bash
IP_ISO2=$(incus list oc-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the
same OVN network can reach each other across nodes.

But net-prod **cannot** reach net-isolated:

```bash
incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Actual result: **100% packet loss**. Different OVN networks are fully
isolated — separate L2 domains, no routing between them.

### 8.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

```bash
incus network acl create oc-node-01:block-ping
incus network acl rule add oc-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.10.0/24 \
    description="Block ICMP from net-prod subnet"
```

### 8.5 Apply and Test ACL

Apply the ACL to net-isolated:

```bash
incus network set oc-node-01:net-isolated security.acls=block-ping
```

Verify ICMP is blocked between net-isolated containers (since they match
the source range — adjust the ACL source for targeted blocking):

```bash
incus exec oc-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Remove the ACL:

```bash
incus network unset oc-node-01:net-isolated security.acls
```

Verify ICMP works again:

```bash
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

### 8.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can
communicate. Peering is bilateral — create a peer on both sides:

```bash
# From net-prod's perspective
incus network peer create oc-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create oc-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"
```

### 8.7 Verify Peering

Cross-network ping (prod → isolated):

```bash
incus exec oc-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops),
confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

```bash
IP_WEB1=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1"
```

### 8.8 Remove Peering

```bash
incus network peer delete oc-node-01:net-prod peer-to-isolated
incus network peer delete oc-node-01:net-isolated peer-to-prod
```

Verify isolation is restored:

```bash
incus exec oc-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

```bash
incus delete oc-node-01:iso-app-01 --force
incus delete oc-node-01:iso-app-02 --force
```


## Section 9: Load Balancers & Network Forwards

### 9.1 Create OVN Load Balancer

Create a load balancer with a VIP from the UPLINK range:

```bash
incus network load-balancer create oc-node-01:net-prod 192.168.103.200
```

Add backend servers. **Important**: backends require the instance's **IP
address**, not its name. Get the IPs first:

```bash
WEB1_IP=$(incus list oc-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
WEB2_IP=$(incus list oc-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1)
echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP"
```

Add backends using IP addresses:

```bash
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \
    web-01 "$WEB1_IP" 80
incus network load-balancer backend add oc-node-01:net-prod 192.168.103.200 \
    web-02 "$WEB2_IP" 80
```

Add a port mapping:

```bash
incus network load-balancer port add oc-node-01:net-prod 192.168.103.200 \
    tcp 80 web-01,web-02
```

### 9.2 Test Load Balancer

From your dev machine (must be on the same VLAN or have routing to
192.168.103.0/24):

```bash
for i in $(seq 1 6); do
    curl -s http://192.168.103.200
done
```

Actual output:

```
Server: prod-web-01
Server: prod-web-01
Server: prod-web-01
Server: prod-web-02
Server: prod-web-02
Server: prod-web-02
```

OVN uses connection-based hashing (not round-robin). Multiple requests from
the same source will typically hit the same backend. Different source ports
or connections may hit different backends.

### 9.3 Create Network Forward

Network forwards expose internal services on LAN IPs. Forward
tcp:8080 → prod-api-01:80. **Like LB backends, forwards require IP addresses**:

```bash
API_IP=$(incus list oc-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1)

incus network forward create oc-node-01:net-prod 192.168.103.201
incus network forward port add oc-node-01:net-prod 192.168.103.201 \
    tcp 8080 "$API_IP" 80
```

### 9.4 Test Network Forward

```bash
curl -s http://192.168.103.201:8080
```

Actual output: `API: prod-api-01`

### 9.5 DNS Resolution

OVN provides per-network DNS. Containers can resolve each other by
hostname:

```bash
incus exec oc-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus"
```

Actual output: `10.10.10.3` — OVN DNS resolves instance names within
each network.


## Section 10: Live Migration

### 10.1 Verify Migration Readiness

Check that VMs have the required configuration:

```bash
for vm in prod-db-01 prod-app-01; do
    echo "=== $vm ==="
    incus config get oc-node-01:$vm limits.cpu
    incus config get oc-node-01:$vm migration.stateful
    incus config device get oc-node-01:$vm root size.state
done
```

Expected: `0-1`, `true`, `2GiB` for each VM.

### 10.2 Create Heartbeat Service

Create a simple counter in prod-db-01 to verify state continuity across
migration:

```bash
incus exec oc-node-01:prod-db-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'
```

Read the counter:

```bash
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Note the value. After migration, the counter should continue from where
it left off (live migration preserves running state).

### 10.3 Live Migration Round-Trip

Check current location:

```bash
incus list oc-node-01:prod-db-01 --columns nL --format csv
```

**Migrate node-01 → node-02:**

```bash
time incus move oc-node-01:prod-db-01 --target oc-node-02
```

Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect:

```bash
sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Counter went from 9 → 25. The heartbeat process was never interrupted —
it continued counting during migration.

**Migrate node-02 → node-03:**

```bash
time incus move oc-node-01:prod-db-01 --target oc-node-03
sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Actual result: 7.379s. Counter went to 41.

**Migrate node-03 → node-01 (back to origin):**

```bash
time incus move oc-node-01:prod-db-01 --target oc-node-01
sleep 4
incus exec oc-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Actual result: 6.896s. Counter went to 56.

Verify the VM is back on node-01:

```bash
incus list oc-node-01:prod-db-01 --columns nL --format csv
```

### 10.4 Active I/O During Migration

Start a continuous write inside the VM:

```bash
incus exec oc-node-01:prod-db-01 -- bash -c '
    dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 &
    echo "Write started, PID: $!"
'
```

Migrate while I/O is active:

```bash
time incus move oc-node-01:prod-db-01 --target oc-node-02
sleep 4
```

Verify the file exists and is intact:

```bash
incus exec oc-node-01:prod-db-01 -- ls -la /tmp/testfile
incus exec oc-node-01:prod-db-01 -- md5sum /tmp/testfile
```

Move back:

```bash
incus move oc-node-01:prod-db-01 --target oc-node-01
sleep 4
```

### 10.5 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly
where it was:

```bash
# Note the heartbeat counter
incus exec oc-node-01:prod-app-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    echo 42 > /tmp/heartbeat/counter
    cat /tmp/heartbeat/counter
'
```

Stateful stop:

```bash
incus stop oc-node-01:prod-app-01 --stateful
```

Start (resumes from saved state):

```bash
incus start oc-node-01:prod-app-01
sleep 4
incus exec oc-node-01:prod-app-01 -- cat /tmp/heartbeat/counter
```

Expected: `42` — the file (and entire VM state) is preserved.

**If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the
saved state:

```bash
incus start oc-node-01:prod-app-01 --stateless
```


## Section 11: Cluster Lifecycle

### 11.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

```bash
incus cluster evacuate oc-node-01:oc-node-02 --force
```

Check workload distribution — nothing on node-02:

```bash
incus list oc-node-01: --columns nstL --format table
```

Actual behavior: VMs with `migration.stateful=true` are live-migrated
(prod-app-01 migrated to oc-node-03). Containers are stopped and moved
(prod-web-02 stopped, moved to oc-node-03, then started). The `--force`
flag skips confirmation prompts.

**Note**: if VMs lack the `limits.cpu` range fix, use `--action stop`
instead to avoid migration failures:
```bash
incus cluster evacuate oc-node-01:oc-node-02 --force --action stop
```

Verify node-02 shows EVACUATED:

```bash
incus cluster list oc-node-01:
```

Restore node-02 (workloads return):

```bash
incus cluster restore oc-node-01:oc-node-02 --force
```

Verify all workloads are back:

```bash
incus list oc-node-01: --columns nstL --format table
incus cluster list oc-node-01:
```

All nodes should show ONLINE.

### 11.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat
detects the failure in ~40 seconds. After the node is restarted:

1. The node auto-rejoins the cluster (~60s)
2. Containers auto-start
3. VMs that were running resume

**Procedure** (document only — do not execute while OVN is running unless
you can tolerate temporary network disruption):

```bash
# Simulate crash: hard-stop via Proxmox API
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/stop

# Wait for heartbeat detection (~40s)
# incus cluster list oc-node-01:
# → oc-node-02 shows OFFLINE

# Restart via Proxmox
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/start

# Wait for auto-rejoin (~60s)
# incus cluster list oc-node-01:
# → oc-node-02 shows ONLINE
```

### 11.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the
VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

**Step 1: Evacuate node-03:**

```bash
incus cluster evacuate oc-node-01:oc-node-03 --force --action stop
```

**Step 2: Remove from cluster:**

```bash
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
```

**Note**: `incus cluster remove` prompts "Are you really sure?" even with
`--force`. The `printf` pipes `yes` for automation.

**Step 3: Clean up the remote:**

```bash
incus remote remove oc-node-03
```

**Step 4: Destroy and redeploy the VM.** Use `incusos-proxmox` to destroy
just node-03 (VMID 402) and redeploy it. Create a single-VM config:

```yaml
# /tmp/lab-replace-node03.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 402

vms:
  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22
```

```bash
./incusos-proxmox --cleanup --yes /tmp/lab-replace-node03.yaml
./incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/lab-replace-node03.yaml
```

**Step 5: Join the fresh node to the cluster:**

```bash
# Set specific IP
incus config set oc-node-03: core.https_address 192.168.102.142:8443

# Generate join token
incus cluster add oc-node-01:oc-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:

# Fix remote
incus remote remove oc-node-03
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
```

**Step 6: Re-enable OVN on the replacement node:**

```bash
incus query oc-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.140:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.142",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add oc-node-01:oc-node-03 ovn-chassis
```

**Step 7: Verify:**

```bash
incus cluster list oc-node-01:
```

All 3 nodes should be ONLINE with `ovn-chassis` role.

### 11.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads
are unevenly distributed), Incus redistributes VMs:

```bash
incus config set oc-node-01: cluster.rebalance.interval=1
incus config set oc-node-01: cluster.rebalance.threshold=10
incus config set oc-node-01: cluster.rebalance.batch=2
incus config set oc-node-01: cluster.rebalance.cooldown=5m
```

**Important**: only VMs with `migration.stateful=true` are rebalanced.
Containers are NOT auto-rebalanced.

Monitor rebalancing:

```bash
incus list oc-node-01: --columns nstL --format table
```

Disable rebalancing when done testing:

```bash
incus config unset oc-node-01: cluster.rebalance.interval
incus config unset oc-node-01: cluster.rebalance.threshold
```


## Section 12: OC Dashboard

**Important**: Switch to the OC remote first. The OC CLI does not support
`remote:` suffix syntax:

```bash
operations-center remote switch oc-lab
```

### OC Server Information

```bash
operations-center admin os show
```

Actual output (version and uptime will vary):

```
environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602240349"
  os_version_next: ""
  uptime: 3600
```

### Provisioning Status

OC manages all 3 cluster nodes. Verify they're registered and up to date:

```bash
operations-center provisioning server list
```

All nodes should show `ready` status and `up to date` update status.

### Web UI

The OC web UI at `https://192.168.102.120:8443/ui/` provides:

- **Dashboard**: server overview with resource utilization
- **Updates**: available IncusOS updates and delivery status
- **Provisioning**: token management, server list, cluster formation
- **System**: OC configuration and certificates

Because the nodes were deployed from an OC-provisioned ISO (Section 3),
OC has full visibility and management of the cluster — including update
delivery, server inventory, and cluster formation.


## Section 13: Cleanup

### Delete All Workloads

```bash
# Delete containers
for c in prod-web-01 prod-web-02 prod-api-01; do
    incus delete oc-node-01:$c --force
done

# Delete VMs
for vm in prod-db-01 prod-app-01; do
    incus delete oc-node-01:$vm --force
done
```

### Remove OVN Networks

```bash
# Delete OVN networks
incus network delete oc-node-01:net-prod
incus network delete oc-node-01:net-isolated 2>/dev/null || true

# Delete UPLINK
incus network delete oc-node-01:UPLINK
```

### Remove OVN Control Plane

```bash
incus delete oc-node-01:ovn-central --force
```

### Disable OVN Services

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done
```

### Infrastructure Options

**Keep infrastructure** (stop VMs, keep on disk for later):

```bash
./incusos-proxmox --lab-down examples/lab-oc-nodes.yaml
./incusos-proxmox --lab-down examples/lab-oc-deploy.yaml
```

Restart later with:

```bash
./incusos-proxmox --lab-up examples/lab-oc-deploy.yaml
./incusos-proxmox --lab-up examples/lab-oc-nodes.yaml
```

**Full teardown** (destroy all VMs, remove ISOs, remotes, cache):

```bash
./incusos-proxmox --cleanup-all --deep --yes
```


## Section 14: Verification Checklist

| # | Check | Command | Expected |
|---|-------|---------|----------|
| 1 | All VMs running | `incusos-proxmox --status examples/lab-oc-nodes.yaml` | 4 VMs running, port 8443 open |
| 2 | Scrub schedule healthy | `incus query oc-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` |
| 3 | OC accessible | `operations-center remote switch oc-lab && operations-center admin os show` | Shows version, uptime |
| 4 | Cluster formed | `incus cluster list oc-node-01:` | 3 nodes ONLINE |
| 5 | Storage pool | `incus storage list oc-node-01:` | `local` pool on all members |
| 6 | Bridge isolation | Ping cross-node on incusbr0 | 100% loss (expected) |
| 7 | OVN connectivity | Ping cross-node on net-prod | 0% loss |
| 8 | Internet via OVN | `ping 1.1.1.1` from OVN container | 0% loss |
| 9 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) |
| 10 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 |
| 11 | Load balancer | `curl http://192.168.103.200` | Backend response |
| 12 | Network forward | `curl http://192.168.103.201:8080` | API response |
| 13 | DNS resolution | `dig prod-web-02.incus` from container | Resolves to 10.10.10.x |
| 14 | VM live migration | `incus move` VM between nodes | State preserved |
| 15 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned |
| 16 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved |


## Section 15: Quick Reference

### Cluster Command Syntax

| Command | Arguments | Notes |
|---------|-----------|-------|
| `incus cluster enable` | `remote: member-name` | TWO args (space between) |
| `incus cluster add` | `remote:member-name` | ONE arg (no space) |
| `incus cluster join` | `init-remote: joining-remote:` | TWO args (space between) |
| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts even with `--force` |
| `incus cluster evacuate` | `remote:member-name` | ONE arg (no space) |
| `incus cluster restore` | `remote:member-name` | ONE arg (no space) |
| `incus config set` | `remote: key value` | Remote with trailing colon + space |
| `incus storage show` | `remote:pool` | ONE arg (no space) |
| `incus storage show` | `remote:pool --target member` | `--target` for member-specific |

### OVN Setup Cheat Sheet

```bash
# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true
```

### Migration Readiness Checklist

| Setting | Value | Why |
|---------|-------|-----|
| `limits.cpu` | Range (e.g., `0-1`) | Fixed QEMU topology across hosts |
| `migration.stateful` | `true` | Enables live migration |
| `root size.state` | `2GiB` (or `4GiB` for 3-4 vCPUs) | Space for memory state file |

Configure while VM is **stopped**:

```bash
incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device override REMOTE:VM root size.state=2GiB
incus start REMOTE:VM
```

### Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| Port 8443 not reachable after boot | Boot still in progress or crontab bug | Wait 180s; check scrub_schedule via API |
| `scrub_schedule` empty | Crontab race condition | `incusos-proxmox --status` auto-heals |
| `Missing section footer for ICH9LPC` on migration | `limits.cpu` set as integer | Set as range: `limits.cpu=0-1` |
| `VM agent isn't currently running` after migration | Agent reconnecting | `sleep 4` after migration |
| `db.sock not found` on OVN config | OVN service not enabled on IncusOS | Enable via `/os/1.0/services/ovn` API |
| Cross-node ping fails (bridge) | Bridge networks are node-local | Use OVN network instead |
| `zfs load-key: Raw key too short` | TPM corruption from premature VM stop | Destroy and redeploy VM |
| Cluster join fails with "pool already exists" | `apply_defaults: true` on joining node | Use `apply_defaults: false` or run 8-command cleanup |
| OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration |
| `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore |
| "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network |
| "Invalid target address" on LB backend | Backend needs IP, not instance name | Use instance IP address (e.g., `10.10.10.2`) |
| OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first |
| Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement |
| "Device from profile(s) cannot be modified" | `root` device comes from default profile | Use `incus config device override` instead of `device set` |
| `incus launch` hangs or times out | Image download to new node takes 2-3 min | Run launches one at a time, not chained with `&&` |