# Shared Storage Guide — iSCSI + lvmcluster on IncusOS

Add shared storage to an existing IncusOS cluster using iSCSI targets and the
`lvmcluster` Incus storage driver. Shared storage eliminates disk data copy
during migration — live migration only transfers RAM state, and non-live
migration becomes a sub-second metadata operation.

All commands and output in this guide are from an actual deployment on
2026-02-23. Tested on an OC-managed 3-node IncusOS cluster (build 202602230420)
on Proxmox VE 9.1.5, Incus client 6.21.

This guide covers two iSCSI target options: a self-contained lab target
container (no external hardware) and a QNAP NAS (production-like). Both use
IncusOS's built-in iSCSI initiator and LVM services — no packages to install.

---

## Section 0: Architecture Overview

### Why shared storage?

With local ZFS pools, migration must copy the full root disk between nodes.
With shared storage, the disk is already accessible from all nodes:

```
Non-live migration (stop/move/start):
  Local ZFS:       copy disk data at ~140 MB/s ≈ varies with disk size
  Shared storage:  metadata update only        ≈ 0.1-2 seconds

Live migration (VM stays running):
  Local ZFS:       copy disk + RAM at ~140 MB/s ≈ 7s+ (grows with disk)
  Shared storage:  copy RAM only at ~140 MB/s   ≈ 5-7s (constant)
```

**Key insight from testing:** live migration always transfers VM RAM (~1 GiB
at ~140 MB/s ≈ 5-7 seconds). Shared storage eliminates the DISK transfer —
the win grows with disk size. A 100 GiB VM on local ZFS takes minutes to
migrate; on shared storage it's still just 5-7 seconds (RAM only).

Non-live migration (stop/move/start) is where shared storage truly shines:
**0.1-2 seconds** regardless of disk size, compared to minutes for large VMs
on local ZFS.

### How iSCSI + lvmcluster works

```mermaid
flowchart LR
    target["iSCSI Target<br/>QNAP / VM / any target<br/>Exposes LUN as block device"]

    subgraph cluster["Incus Cluster Nodes"]
        n1["oc-node-01"]
        n2["oc-node-02"]
        n3["oc-node-03"]
        stack["iSCSI initiator (built-in)<br/>lvmlockd + sanlock (built-in)<br/>Incus lvmcluster pool"]
    end

    target <-->|"iSCSI (1GbE)"| n1
    target <-->|"iSCSI (1GbE)"| n2
    target <-->|"iSCSI (1GbE)"| n3
    n1 & n2 & n3 ~~~ stack

    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf

    class target network
    class n1,n2,n3 node
    class stack instance

    style cluster fill:#e6f5f0,stroke:#009E73
```

1. **iSCSI target** exports a block device (LUN) over the network
2. **iSCSI initiators** (IncusOS built-in) connect from each node — the LUN
   appears as a local block device (`/dev/sd*`)
3. **LVM with sanlock** (`lvmlockd`) provides distributed locking so multiple
   nodes can safely use the shared block device
4. **Incus `lvmcluster` driver** creates logical volumes (LVs) on the shared
   volume group for instance root disks and custom volumes
5. **Live migration** is metadata-only — the destination node already has
   access to the same LV on the same shared device

### Hybrid architecture: local ZFS + shared lvmcluster

The recommended setup uses **both** pools:

| Pool | Type | Use case | Provisioning |
|------|------|----------|--------------|
| `local` | ZFS (per-node) | OS images, containers, non-HA workloads | Thin (copy-on-write) |
| `shared` | lvmcluster (shared) | HA VMs, workloads needing instant migration | Thick (full allocation) |

**Why hybrid?**
- Local ZFS has thin provisioning, snapshots, and fast local I/O
- lvmcluster has no thin provisioning (10 GiB VM = 10 GiB on LUN)
- Keep OS images and ephemeral containers on local ZFS
- Put only HA VMs that need instant migration on the shared pool

### Decision matrix: when to use which pool

| Scenario | Pool | Why |
|----------|------|-----|
| Development container | `local` | Fast, thin provisioned, no HA needed |
| Production VM with HA | `shared` | Instant migration, zero data copy |
| OVN control plane container | `local` | Pinned to one node, no migration |
| Database VM (HA) | `shared` | Needs failover without data copy |
| Temporary test instance | `local` | Ephemeral, don't waste shared space |

### Network topology (tested)

```mermaid
flowchart TD
    subgraph proxmox["Proxmox Host"]
        subgraph n1["oc-node-01 · VMID 400 · .140"]
            target["iscsi-target<br/>10.207.217.19 (bridge)<br/>192.168.102.150 (macvlan)<br/>LUN: 20 GiB"]
        end

        n2["oc-node-02<br/>VMID 401 · .141<br/>lvmcluster"]
        n3["oc-node-03<br/>VMID 402 · .142<br/>lvmcluster"]
    end

    target <-->|"bridge"| n1
    target <-->|"macvlan"| n2
    target <-->|"macvlan"| n3
    vlan(("VLAN 69<br/>192.168.100.0/22"))
    proxmox --- vlan

    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef lb fill:#E69F00,color:#fff,stroke:#b87d00

    class target network
    class n2,n3 node
    class vlan network

    style proxmox fill:#f5f5f5,stroke:#999
    style n1 fill:#e6f5f0,stroke:#009E73
```

**Lab target container** (Option A, tested): a Debian container on oc-node-01
running `tgt` (userspace iSCSI target). Uses two network paths: bridge IP for
same-node access, macvlan on `mgmt` for cross-node access.

**QNAP NAS** (Option B, not yet tested): replace the target container with
the QNAP at 192.168.1.x. Requires routing between VLAN 69 and the native LAN.

### Infrastructure (Option A: Lab target container)

| Component | IP | Role |
|-----------|-----|------|
| iscsi-target (container on oc-node-01) | 10.207.217.19 (bridge), 192.168.102.150 (macvlan) | iSCSI target (tgt) |
| oc-node-01 (VMID 400) | 192.168.102.140/22 | Cluster init + iSCSI via bridge |
| oc-node-02 (VMID 401) | 192.168.102.141/22 | Cluster member + iSCSI via macvlan |
| oc-node-03 (VMID 402) | 192.168.102.142/22 | Cluster member + iSCSI via macvlan |

**Additional RAM**: negligible (~20 MiB for the container). No extra Proxmox VM needed.

### Cross-references

- [Clustering guide](clustering-guide.md) — manual cluster formation
- [Networking guide](networking-guide.md) — OVN overlay tutorial
- [Operations Center guide](operations-center-guide.md) — OC-managed clusters
- [Production lab guide](production-lab-guide.md) — manual cluster + OVN + HA

---

## Section 1: Prerequisites

### Existing cluster

This guide assumes you have a working 3-node IncusOS cluster (OC-managed or
manual). The examples use the OC-managed cluster from the
[Operations Center guide](operations-center-guide.md):

```bash
incus cluster list oc-node-01:
```

Actual output:

| NAME       | URL                          | ROLES                                      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE           |
|------------|------------------------------|---------------------------------------------|--------------|----------------|-------------|--------|-------------------|
| oc-node-01 | https://192.168.102.140:8443 | ovn-chassis, database-leader, database      | x86_64       | default        |             | ONLINE | Fully operational |
| oc-node-02 | https://192.168.102.141:8443 | ovn-chassis, database                       | x86_64       | default        |             | ONLINE | Fully operational |
| oc-node-03 | https://192.168.102.142:8443 | ovn-chassis, database                       | x86_64       | default        |             | ONLINE | Fully operational |

The `ovn-chassis` roles are from the [networking guide](networking-guide.md) —
they don't affect shared storage setup.

### IncusOS built-in services

IncusOS is immutable — you cannot install packages. But these services are
built in and enabled via the REST API:

| Service | API Endpoint | Purpose |
|---------|-------------|---------|
| **iSCSI** | `/os/1.0/services/iscsi` | Initiator — connects to external iSCSI targets |
| **LVM** | `/os/1.0/services/lvm` | Enables `lvmlockd` + `sanlock` for clustered LVM |
| **OVN** | `/os/1.0/services/ovn` | OVN controller (already configured if using OVN) |
| **Ceph** | `/os/1.0/services/ceph` | Ceph client (alternative to iSCSI, not covered here) |
| **Multipath** | `/os/1.0/services/multipath` | Redundant I/O paths (not needed for single-path lab) |

Check available services on a node:

```bash
incus query oc-node-01:/os/1.0/services
```

### Required tools

```bash
# Incus client
incus version
# Client version: 6.21

# Optional: incusos-proxmox for deploying the target VM
./incusos/incusos-proxmox --doctor
```

### Network requirements

- All cluster nodes must reach the iSCSI target on TCP port 3260
- For lab target VM: all on VLAN 69 — no routing needed
- For QNAP NAS: routing between VLAN 69 (192.168.102.x) and native LAN
  (192.168.1.x) via gateway or dedicated NIC

---

## Section 2: Option A — Lab Target Container (Self-Contained, Tested)

Deploy a Debian container on the Incus cluster running `tgt` (userspace iSCSI
target). No external hardware or Proxmox VM needed — everything runs inside
the existing cluster.

**Why a container instead of a Proxmox VM?** Faster to deploy, no Proxmox
manual steps, and validates the same iSCSI concepts. The container uses `tgt`
(a userspace iSCSI target daemon) which works in unprivileged containers
without kernel modules.

**Why not use Incus proxy devices?** Tested and failed. The IncusOS iSCSI
service uses `iscsiadm -m discovery -t sendtargets` before login. Through a
proxy device, the target responds with its container IP in the SendTargets
response, creating node records with the wrong portal address. The subsequent
login to the proxy IP fails with "No records found" (exit status 21).

**Solution: dual network paths.** The container gets two interfaces:
- `eth0` on `incusbr0` (bridge, 10.207.217.x) — used by oc-node-01 (same node)
- `eth1` as macvlan on `mgmt` (192.168.102.150) — used by oc-node-02/03

### 2.1 Launch the iSCSI target container

```bash
incus launch images:debian/12 oc-node-01:iscsi-target --target oc-node-01
```

### 2.2 Install tgt

```bash
incus exec oc-node-01:iscsi-target -- apt-get update -qq
incus exec oc-node-01:iscsi-target -- apt-get install -y tgt
```

Actual output (key lines):

```
Setting up tgt (1:1.0.85-1+deb12u1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/tgt.service → ...
```

The RDMA and oom_score warnings in the service log are expected in a container
and can be safely ignored:

```
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
can't adjust oom-killer's pardon /proc/self/oom_score_adj, Permission denied
```

### 2.3 Create the backing store and configure the target

```bash
incus exec oc-node-01:iscsi-target -- bash -c '
# Create sparse file (only uses actual written data on disk)
mkdir -p /srv/iscsi
truncate -s 20G /srv/iscsi/shared-lun.img

# Create iSCSI target
tgtadm --lld iscsi --op new --mode target --tid 1 \
    -T iqn.2026-02.lab.incus:storage.shared

# Add LUN (LUN 0 is reserved for the controller, use LUN 1)
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 \
    -b /srv/iscsi/shared-lun.img

# Allow all initiators (lab environment)
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

# Verify
tgtadm --lld iscsi --op show --mode target
'
```

Actual output:

```
Target 1: iqn.2026-02.lab.incus:storage.shared
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 21475 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /srv/iscsi/shared-lun.img
            Backing store flags:
    Account information:
    ACL information:
        ALL
```

### 2.4 Make the target persistent

```bash
incus exec oc-node-01:iscsi-target -- bash -c 'cat > /etc/tgt/conf.d/shared-lun.conf << '\''EOF'\''
<target iqn.2026-02.lab.incus:storage.shared>
    backing-store /srv/iscsi/shared-lun.img
    initiator-address ALL
</target>
EOF'
```

### 2.5 Add macvlan NIC for cross-node access

The container is on `incusbr0` which is node-local — oc-node-02 and oc-node-03
cannot reach it. Add a macvlan NIC on the management interface:

```bash
# Add macvlan NIC on IncusOS management interface
incus config device add oc-node-01:iscsi-target mgmt-nic nic \
    nictype=macvlan parent=mgmt

# Configure static IP inside the container
incus exec oc-node-01:iscsi-target -- ip link set eth1 up
incus exec oc-node-01:iscsi-target -- ip addr add 192.168.102.150/22 dev eth1
```

**macvlan limitation**: the host (oc-node-01) cannot reach 192.168.102.150
through the macvlan (kernel filters traffic between macvlan and parent).
That's why oc-node-01 connects via the bridge IP (10.207.217.x) instead.

### 2.6 Verify the target is listening and reachable

```bash
# Verify tgt is listening
incus exec oc-node-01:iscsi-target -- ss -tlnp | grep 3260
```

Actual output:

```
LISTEN 0  4096  0.0.0.0:3260  0.0.0.0:*  users:(("tgtd",pid=840,fd=6))
LISTEN 0  4096     [::]:3260     [::]:*  users:(("tgtd",pid=840,fd=7))
```

**Note on LUN sizing**: 20 GiB is sufficient for testing (2-3 small VMs).
lvmcluster uses thick provisioning — a 4 GiB VM root + 2 GiB state area =
6.25 GiB on the LUN (including LVM/sanlock metadata). For production, use
200+ GiB on the QNAP.

---

## Section 3: Option B — QNAP iSCSI Target (Production-Like)

Use the QNAP NAS's built-in iSCSI target service instead of a VM. This is
more realistic for production but requires physical access and VLAN routing.

### 3.1 Create iSCSI target on QNAP

Via the QNAP web UI (QTS):

1. Open **iSCSI & Fibre Channel** app (install from App Center if missing)
2. **Storage** > Create a new LUN:
   - Name: `incus-shared`
   - Size: 200 GiB
   - Provisioning: Thick (recommended) or Thin
3. **Target** > Create a new target:
   - Name: `iqn.2026-02.nas.qnap:incus-shared`
   - CHAP: disabled (lab environment)
4. Map the LUN to the target
5. Note the QNAP's IP address (e.g., `192.168.1.100`)

### 3.2 Network routing (VLAN 69 to native LAN)

IncusOS VMs on VLAN 69 (192.168.102.x/22) need to reach the QNAP on the
native LAN (192.168.1.x). Options:

**Option 1: Route via gateway (simplest)**

If your router/gateway (192.168.100.1) handles both VLANs, the VMs can
already reach 192.168.1.x. Test from a node:

```bash
incus exec oc-node-01:test-ping -- ping -c 3 192.168.1.100
```

Or test via the IncusOS API (since IncusOS has no shell):

```bash
# Check if the QNAP port is reachable from a container on the node
incus launch images:debian/12 oc-node-01:test-ping --target oc-node-01
incus exec oc-node-01:test-ping -- apt-get update -qq
incus exec oc-node-01:test-ping -- apt-get install -y -qq iputils-ping
incus exec oc-node-01:test-ping -- ping -c 3 192.168.1.100
incus delete oc-node-01:test-ping --force
```

**Option 2: Dedicated NIC on QNAP**

If the QNAP has a second NIC port, connect it to the VLAN 69 network
and assign an IP in the 192.168.102.x/22 range (e.g., 192.168.102.200).

### 3.3 Verify connectivity

From each IncusOS node, verify the iSCSI target port is reachable. Since
IncusOS has no shell, verify after enabling the iSCSI service (Section 4)
by checking if the target discovery succeeds.

---

## Section 4: Enable IncusOS iSCSI Service

Enable the built-in iSCSI initiator on every cluster node. This must be done
before any iSCSI target can be connected.

### 4.1 Check current iSCSI service state

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== $node ==="
    incus query ${node}:/os/1.0/services/iscsi
done
```

Output (before enabling):

```json
{
    "config": {
        "enabled": false
    },
    "state": {}
}
```

### 4.2 Enable iSCSI and connect to the target

**Critical: the API field is `target`, not `iqn`.** The JSON key for the IQN
is `"target"`. Using `"iqn"` silently saves an empty target string, and the
service enables but cannot connect. This is the most common mistake.

**For lab target container (Option A):**

oc-node-01 connects via the bridge IP (same node); oc-node-02 and oc-node-03
connect via the macvlan IP (cross-node):

```bash
TARGET_IQN="iqn.2026-02.lab.incus:storage.shared"

# oc-node-01: connect via bridge IP (macvlan doesn't work to own parent)
echo "=== Enabling iSCSI on oc-node-01 ==="
incus query oc-node-01:/os/1.0/services/iscsi --request PUT --data "{
    \"config\": {
        \"enabled\": true,
        \"targets\": [
            {
                \"address\": \"10.207.217.19\",
                \"port\": 3260,
                \"target\": \"${TARGET_IQN}\"
            }
        ]
    },
    \"state\": {}
}"

# oc-node-02 and oc-node-03: connect via macvlan IP
for node in oc-node-02 oc-node-03; do
    echo "=== Enabling iSCSI on $node ==="
    incus query ${node}:/os/1.0/services/iscsi --request PUT --data "{
        \"config\": {
            \"enabled\": true,
            \"targets\": [
                {
                    \"address\": \"192.168.102.150\",
                    \"port\": 3260,
                    \"target\": \"${TARGET_IQN}\"
                }
            ]
        },
        \"state\": {}
    }"
done
```

**For QNAP (Option B):**

All nodes use the same QNAP IP:

```bash
TARGET_IP="192.168.1.100"                        # Or 192.168.102.200 if dedicated NIC
TARGET_IQN="iqn.2026-02.nas.qnap:incus-shared"  # From QNAP web UI

for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== Enabling iSCSI on $node ==="
    incus query ${node}:/os/1.0/services/iscsi --request PUT --data "{
        \"config\": {
            \"enabled\": true,
            \"targets\": [
                {
                    \"address\": \"${TARGET_IP}\",
                    \"port\": 3260,
                    \"target\": \"${TARGET_IQN}\"
                }
            ]
        },
        \"state\": {}
    }"
done
```

### 4.3 Verify the iSCSI connection

After enabling, verify each node connected successfully:

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== $node ==="
    incus query ${node}:/os/1.0/services/iscsi
done
```

Actual output (oc-node-01 — via bridge IP):

```json
{
    "config": {
        "enabled": true,
        "targets": [
            {
                "address": "10.207.217.19",
                "port": 3260,
                "target": "iqn.2026-02.lab.incus:storage.shared"
            }
        ]
    },
    "state": {
        "initiator_name": "iqn.2004-10.org.linuxcontainers:01:2390dbbbee72"
    }
}
```

Actual output (oc-node-02 — via macvlan IP):

```json
{
    "config": {
        "enabled": true,
        "targets": [
            {
                "address": "192.168.102.150",
                "port": 3260,
                "target": "iqn.2026-02.lab.incus:storage.shared"
            }
        ]
    },
    "state": {
        "initiator_name": "iqn.2004-10.org.linuxcontainers:01:3c58d1566dc1"
    }
}
```

**How to tell it worked:** the `state` section contains `initiator_name`
(auto-generated unique IQN). If `state` is empty `{}`, the connection
failed — check the target address and that the `target` field (not `iqn`)
was used.

Each node gets a unique `initiator_name` — this is expected and correct.
The iSCSI target sees 3 separate initiators sharing the same LUN.

### 4.4 Identify the iSCSI block device

The iSCSI LUN appears as a SCSI disk on each node. Check via the IncusOS
storage API:

```bash
incus query oc-node-01:/os/1.0/system/storage | python3 -c "
import sys, json
d = json.load(sys.stdin)
for drive in d['state']['drives']:
    size_gib = drive['capacity_in_bytes'] / (1024**3)
    print(f\"  {drive['id']} - {size_gib:.0f} GiB - {drive['model_name']}\")"
```

Actual output:

```
  /dev/disk/by-id/lvm-pv-uuid-rUETjP-5LQr-0g9y-KWD8-cxB9-QsOv-ieIcru - 20 GiB - VIRTUAL-DISK
  /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 - 64 GiB - QEMU HARDDISK
```

The 20 GiB `VIRTUAL-DISK` is the iSCSI LUN. The 64 GiB `QEMU HARDDISK` is
the system drive. Note the LVM PV UUID path — this means the device is already
an LVM physical volume (Incus created it when we set up the pool).

**The device also appears as `/dev/sdb` on all 3 nodes** — consistent because
all VMs have the same hardware layout (scsi0 = system, sdb = iSCSI LUN).

You don't need to note the exact path — Incus handles device references
internally when creating the lvmcluster pool (Section 7).

---

## Section 5: Enable IncusOS LVM Service

Enable the LVM service with `lvmlockd` and `sanlock` for clustered LVM.
This provides distributed locking so multiple nodes can safely share the
same volume group.

### 5.1 Check current LVM service state

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== $node ==="
    incus query ${node}:/os/1.0/services/lvm
done
```

Output (before enabling):

```json
{
    "config": {
        "enabled": false
    },
    "state": {}
}
```

### 5.2 Enable LVM with unique system_id per node

**Critical: each node must have a unique `system_id` between 1 and 2000.**
This is the sanlock host ID. Using `0` or duplicates will cause pool creation
to fail with `"Invalid host_id 0, use 1-2000"`.

```bash
id=1
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== Enabling LVM on $node (system_id=$id) ==="
    incus query ${node}:/os/1.0/services/lvm --request PUT --data "{
        \"config\": {
            \"enabled\": true,
            \"system_id\": $id
        },
        \"state\": {}
    }"
    id=$((id + 1))
done
```

### 5.3 Verify LVM services are running

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== $node ==="
    incus query ${node}:/os/1.0/services/lvm
done
```

Actual output (oc-node-01, after pool creation — `state` shows VG info):

```json
{
    "config": {
        "enabled": true,
        "system_id": 1
    },
    "state": {
        "pvs": [
            {
                "pv_attr": "a--",
                "pv_fmt": "lvm2",
                "pv_free": "<19.75g",
                "pv_name": "/dev/sdb",
                "pv_size": "<20.00g",
                "vg_name": "shared"
            }
        ],
        "vgs": [
            {
                "lv_count": 0,
                "pv_count": 1,
                "snap_count": 0,
                "vg_attr": "wz--ns",
                "vg_free": "<19.75g",
                "vg_name": "shared",
                "vg_size": "<20.00g"
            }
        ]
    }
}
```

**Before pool creation**, the `state` section will show empty `pvs` and `vgs`
arrays. That's normal — the VG doesn't exist yet.

**Key fields**: `system_id` is unique per node (1, 2, 3). The `vg_attr`
`wz--ns` means: writeable, resizable, no allocation policy, not partial,
shared (`s`). The `s` at the end confirms sanlock locking is active.

The `vgs` section showing the same VG name on all 3 nodes confirms shared
access — all nodes see the same volume group through their iSCSI connections.

---

## Section 6: Create Shared Volume Group

**You don't need to create the VG manually.** The Incus `lvmcluster` driver
handles `pvcreate`, `vgcreate --shared`, and `vgchange --lock-start`
automatically during pool creation (Section 7).

This section is included for reference — skip to Section 7 unless you need
to understand the underlying LVM operations.

### 6.1 What Incus does automatically

When you create an `lvmcluster` pool with `source=/dev/disk/by-id/scsi-...`,
Incus runs the equivalent of:

```bash
# 1. Create physical volume on the iSCSI device (one node)
pvcreate /dev/sdb

# 2. Create shared VG with sanlock lock type (one node)
vgcreate --shared <pool-name> /dev/sdb

# 3. Start the lock on all nodes (each node)
vgchange --lock-start <pool-name>
```

The `--shared` flag creates a VG with `locktype=sanlock`, which uses sanlock
for distributed locking. `vgchange --lock-start` activates the lock manager
on each node that needs access to the VG.

### 6.2 Verify VG visibility (after pool creation in Section 7)

After the pool is created, verify all nodes see the shared VG via the LVM
service API:

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== $node ==="
    incus query ${node}:/os/1.0/services/lvm | python3 -c "
import sys, json
d = json.load(sys.stdin)
for vg in d.get('state', {}).get('vgs', []):
    print(f\"  VG: {vg['vg_name']}  Size: {vg['vg_size']}  Free: {vg['vg_free']}  Attr: {vg['vg_attr']}\")
"
done
```

Actual output:

```
=== oc-node-01 ===
  VG: shared  Size: <20.00g  Free: <19.75g  Attr: wz--ns
=== oc-node-02 ===
  VG: shared  Size: <20.00g  Free: <19.75g  Attr: wz--ns
=== oc-node-03 ===
  VG: shared  Size: <20.00g  Free: <19.75g  Attr: wz--ns
```

All 3 nodes see the same VG `shared` (20 GiB, ~256 MiB used for LVM/sanlock
metadata). The `s` in `vg_attr` confirms sanlock locking is active.

---

## Section 7: Create lvmcluster Pool in Incus

Create the shared storage pool using Incus's `lvmcluster` driver. This
follows the **two-step cluster pattern**: create a pending entry per member
with `--target`, then finalize without `--target`.

### 7.1 Identify the iSCSI device path

Get the stable device path from the storage API (Section 4.4). In our lab:

```
/dev/disk/by-id/scsi-360000000000000000e00000000010001
```

This is a stable SCSI ID path — it won't change across reboots (unlike
`/dev/sdb` which could shift). You can also use `/dev/sdb` if your setup
is simple.

### 7.2 Create pending pool on each member

```bash
ISCSI_DEVICE="/dev/disk/by-id/scsi-360000000000000000e00000000010001"

# Create pending pool entry for each cluster member
for node in oc-node-01 oc-node-02 oc-node-03; do
    echo "=== Creating pending pool on $node ==="
    incus storage create oc-node-01:shared lvmcluster \
        source=${ISCSI_DEVICE} \
        --target ${node}
done
```

Each `--target` call creates a **pending** entry for that member. The pool
does not become active until finalized. Output for each:

```
Storage pool shared pending on member oc-node-01
Storage pool shared pending on member oc-node-02
Storage pool shared pending on member oc-node-03
```

### 7.3 Finalize the pool

```bash
incus storage create oc-node-01:shared lvmcluster
```

Output:

```
Storage pool shared created
```

Incus handles all LVM operations automatically:
1. `pvcreate` on the iSCSI device
2. `vgcreate --shared` to create the volume group with sanlock
3. `vgchange --lock-start` on all members
4. Pool metadata registration in the cluster database

### 7.4 Verify the pool

```bash
incus storage list oc-node-01:
```

Actual output:

| NAME   | DRIVER     | DESCRIPTION                          | USED BY | STATE   |
|--------|------------|--------------------------------------|---------|---------|
| local  | zfs        | Local storage pool (on system drive) | 17      | CREATED |
| shared | lvmcluster |                                      | 1       | CREATED |

```bash
incus storage show oc-node-01:shared
```

Actual output:

```yaml
config: {}
description: ""
name: shared
driver: lvmcluster
used_by:
- /1.0/profiles/shared-pool
status: Created
locations:
- oc-node-01
- oc-node-02
- oc-node-03
```

All 3 locations listed = pool is active on all cluster members.

```bash
# Per-member config (shows the device source and VG name)
incus storage show oc-node-01:shared --target oc-node-01
```

Actual output:

```yaml
config:
  lvm.vg_name: shared
  source: shared
  volatile.initial_source: /dev/disk/by-id/scsi-360000000000000000e00000000010001
description: ""
name: shared
driver: lvmcluster
used_by:
- /1.0/profiles/shared-pool
status: Created
locations:
- oc-node-01
- oc-node-02
- oc-node-03
```

```bash
# Pool space info
incus storage info oc-node-01:shared --target oc-node-01
```

Actual output:

```
info:
  description: ""
  driver: lvmcluster
  name: shared
  space used: 256.00MiB
  total space: 20.00GiB
used by:
  profiles:
  - shared-pool
```

256 MiB used = LVM metadata + sanlock lease area. 19.75 GiB available for
instance storage.

### 7.5 lvmcluster limitations

Be aware of these limitations compared to local ZFS:

| Feature | Local ZFS | lvmcluster |
|---------|-----------|------------|
| Thin provisioning | Yes (copy-on-write) | **No** (thick LVs only) |
| Snapshots (custom volumes) | Yes | **No** (sanlock limitation) |
| Snapshots (VM disks) | Yes | Yes (QEMU internal) |
| Compression | Yes (LZ4) | **No** |
| Image caching | Yes | Yes |
| Container support | Yes | Yes |
| VM support | Yes | Yes |

**Thick provisioning means disk space is allocated up front.** A 4 GiB VM root
disk uses 4 GiB on the LUN immediately. A 10 GiB root disk uses 10 GiB. Plan
LUN sizing accordingly — a 20 GiB test LUN fits ~3 small VMs with 4 GiB roots.

---

## Section 8: Test — Launch and Migrate

### 8.1 Create a profile for the shared pool

```bash
incus profile create oc-node-01:shared-pool

incus profile device add oc-node-01:shared-pool root disk \
    path=/ pool=shared

incus profile device add oc-node-01:shared-pool eth0 nic \
    network=incusbr0 name=eth0
```

Verify the profile:

```bash
incus profile show oc-node-01:shared-pool
```

Actual output:

```yaml
config: {}
description: ""
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: shared
    type: disk
name: shared-pool
used_by: []
project: default
```

### 8.2 Test container migration (stop/move/start)

Start with a container — it's fast to launch and validates the shared pool
before testing VMs.

```bash
# Launch a container on the shared pool
incus launch images:debian/12 oc-node-01:test-shared-ct \
    --profile shared-pool \
    --target oc-node-01
```

Wait for startup, then create a test file at a **persistent** location
(`/root/`, not `/tmp/` — tmpfs gets cleared on stop/start):

```bash
sleep 5
incus exec oc-node-01:test-shared-ct -- bash -c \
    'echo "shared storage works" > /root/test.txt'
```

Stop, move, start:

```bash
incus stop oc-node-01:test-shared-ct
time incus move oc-node-01:test-shared-ct --target oc-node-02
incus start oc-node-01:test-shared-ct
```

Actual timing:

```
real    0m0.121s
```

**0.121 seconds** — metadata only, no data copy. Verify data persists:

```bash
incus exec oc-node-01:test-shared-ct -- cat /root/test.txt
```

```
shared storage works
```

Move again (node-02 → node-03):

```bash
incus stop oc-node-01:test-shared-ct
time incus move oc-node-01:test-shared-ct --target oc-node-03
incus start oc-node-01:test-shared-ct
```

```
real    0m0.152s
```

**0.152 seconds.** Data verified intact. Container migration on shared storage
is effectively instant — the container's root filesystem LV is already
accessible from all nodes.

### 8.3 Launch a VM on the shared pool

```bash
incus launch images:debian/12 oc-node-01:test-shared-vm --vm \
    --profile shared-pool \
    --target oc-node-01 \
    -c limits.cpu=0-1 \
    -c migration.stateful=true \
    -d root,size=4GiB \
    -d root,size.state=2GiB
```

**Important flags explained:**
- `limits.cpu=0-1` — CPU range (not integer!) required for live migration
  compatibility across nodes with different core counts
- `migration.stateful=true` — enables live migration (QEMU state transfer)
- `size=4GiB` — explicit root disk size (thick provisioned, uses 4 GiB on LUN)
- `size.state=2GiB` — space for VM RAM state during migration

Wait for the VM to boot (~20 seconds for first image download):

```bash
incus list oc-node-01: --format compact | grep test-shared-vm
```

Actual output:

```
 test-shared-vm  RUNNING  10.207.217.54 (enp5s0)  fd42:...:fe9e:b52a (enp5s0)  VIRTUAL-MACHINE  0  oc-node-01
```

### 8.4 Live migrate the VM

```bash
time incus move oc-node-01:test-shared-vm --target oc-node-02
```

Actual output:

```
Transferring instance: Live migration: 1.05GB remaining (141.85MB/s)
Transferring instance: Live migration: 879.56MB remaining (141.86MB/s)
Transferring instance: Live migration: 457.06MB remaining (142.02MB/s)
Transferring instance: Live migration: 0B remaining (109.96MB/s)

real    0m6.011s
```

**6.0 seconds** — only RAM was transferred (~1 GiB at ~140 MB/s). No disk
data was copied. The progress shows RAM transfer only.

```bash
sleep 4  # Wait for VM agent to reconnect
incus list oc-node-01: --format compact | grep test-shared-vm
```

```
 test-shared-vm  RUNNING  10.207.217.54 (enp5s0)  ...  VIRTUAL-MACHINE  0  oc-node-02
```

### 8.5 Continue migrating across all nodes

```bash
# node-02 → node-03
time incus move oc-node-01:test-shared-vm --target oc-node-03
```

```
real    0m6.083s
```

```bash
# node-03 → node-01
sleep 4
time incus move oc-node-01:test-shared-vm --target oc-node-01
```

```
real    0m6.132s
```

All live migrations consistent at **~6 seconds** (RAM transfer only).

**Known issue: first live migration after stop/start may fail.** If a VM was
stopped and started (non-live migration), the first live migration attempt
may fail with `exit status 1` (QEMU on the destination cannot start). This
appears to be a transient sanlock lease issue. **Workaround:** retry the
migration — subsequent attempts succeed. Non-live migration (stop/move/start)
always works reliably.

### 8.6 Non-live migration comparison

For completeness, test stop/move/start with the VM:

```bash
incus stop oc-node-01:test-shared-vm
time incus move oc-node-01:test-shared-vm --target oc-node-03
incus start oc-node-01:test-shared-vm
```

Actual timing:

```
real    0m1.756s
```

**1.8 seconds** — mostly LVM metadata operations. No disk data copied.
Compare this to local ZFS where the full root disk must be transferred.

### 8.7 Cleanup test instances

```bash
incus delete oc-node-01:test-shared-vm --force
incus delete oc-node-01:test-shared-ct --force
```

---

## Section 9: Performance Comparison

### 9.1 Migration time comparison (tested)

The primary benefit of shared storage is migration speed. Tested results:

| Migration type | Local ZFS | Shared lvmcluster | Improvement |
|---------------|-----------|-------------------|-------------|
| **VM live migration** | 7.0s | **6.0-6.4s** | ~14% faster (RAM only vs RAM+disk) |
| **VM non-live (stop/move/start)** | 2.1s | **1.8s** | ~15% faster (metadata only) |
| **Container non-live** | N/A | **0.12-0.15s** | Near-instant |

**Key insight:** with a small 4 GiB VM root disk, the improvement is modest
because local ZFS with thin provisioning only transfers actual data (not the
full allocated size). The real win comes with **larger VMs**: a 50 GiB root
disk on local ZFS must transfer all used data (~minutes); on shared storage
it's still 6 seconds (RAM only).

Tested migration details:

```
Shared pool VM live migration (1 GiB RAM):
  node-01 → node-02:  6.011s  (141 MB/s, ~1 GiB RAM transferred)
  node-02 → node-03:  6.083s  (141 MB/s)
  node-03 → node-01:  6.132s  (141 MB/s)

Local ZFS VM live migration (1 GiB RAM, 4 GiB disk):
  node-01 → node-02:  7.027s  (141 MB/s, RAM + disk data)

Shared pool container (stop/move/start):
  node-01 → node-02:  0.121s  (metadata only)
  node-02 → node-03:  0.152s  (metadata only)

Shared pool VM (stop/move/start):
  node-03 → node-01:  1.756s  (LVM metadata update)

Local ZFS VM (stop/move/start):
  node-02 → node-03:  2.078s  (data copy)
```

### 9.2 I/O benchmarks (optional)

To compare I/O performance between the pools, launch VMs and run `fio`:

```bash
# Launch VMs on each pool
incus launch images:debian/12 oc-node-01:bench-local --vm \
    --target oc-node-01 -c limits.cpu=0-1

incus launch images:debian/12 oc-node-01:bench-shared --vm \
    --profile shared-pool --target oc-node-01 \
    -c limits.cpu=0-1 -d root,size=4GiB

# Wait for boot, install fio
sleep 30
for vm in bench-local bench-shared; do
    incus exec oc-node-01:${vm} -- apt-get update -qq
    incus exec oc-node-01:${vm} -- apt-get install -y -qq fio
done

# Sequential write (1M blocks, 1 GiB)
for vm in bench-local bench-shared; do
    echo "=== $vm ==="
    incus exec oc-node-01:${vm} -- fio --name=seqwrite \
        --ioengine=libaio --direct=1 --bs=1M --size=1G \
        --numjobs=1 --rw=write --group_reporting
done

# Random 4K read (30 seconds)
for vm in bench-local bench-shared; do
    echo "=== $vm ==="
    incus exec oc-node-01:${vm} -- fio --name=rand4k \
        --ioengine=libaio --direct=1 --bs=4k --size=256M \
        --numjobs=4 --rw=randread --group_reporting \
        --runtime=30 --time_based
done

# Cleanup
incus delete oc-node-01:bench-local --force
incus delete oc-node-01:bench-shared --force
```

### 9.3 Expected I/O performance (1GbE)

| Test | Local ZFS | Shared lvmcluster (1GbE) | Notes |
|------|-----------|--------------------------|-------|
| Sequential write | ~500-800 MB/s | ~100-110 MB/s | Network-bound on iSCSI |
| Sequential read | ~500-800 MB/s | ~100-110 MB/s | Network-bound on iSCSI |
| Random 4K read IOPS | ~10,000-50,000 | ~5,000-15,000 | Latency-sensitive |
| VM boot time | ~5 seconds | ~8-12 seconds | Acceptable for lab |

Local ZFS will always win on raw I/O throughput (local disk vs network).
The trade-off is clear: **use local ZFS for I/O-heavy workloads, shared
lvmcluster for HA workloads needing fast migration.**

---

## Section 10: Hybrid Architecture in Practice

Use both pools together. Keep the local ZFS pool as the default for general
workloads. Use the shared pool only for instances that need HA migration.

### 10.1 Profiles

Two profiles exist — one for each pool:

```bash
incus profile show oc-node-01:default
```

```yaml
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: local      # ← local ZFS
    type: disk
```

```bash
incus profile show oc-node-01:shared-pool
```

```yaml
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: shared     # ← shared lvmcluster
    type: disk
```

### 10.2 Launching with explicit pool choice

```bash
# Regular container (local ZFS, default profile)
incus launch images:debian/12 oc-node-01:app-web --target oc-node-01

# HA VM (shared lvmcluster, explicit profile + migration settings)
incus launch images:debian/12 oc-node-01:app-db --vm \
    --profile shared-pool \
    --target oc-node-01 \
    -c limits.cpu=0-1 \
    -c migration.stateful=true \
    -d root,size=4GiB \
    -d root,size.state=2GiB
```

**Remember:** shared pool VMs need explicit `size` because of thick
provisioning (default 10 GiB may be too large for a small test LUN), and
`size.state` for stateful migration.

### 10.3 Moving instances between pools

To move an instance from local to shared (or vice versa), use `incus move`
with `--storage`:

```bash
incus stop oc-node-01:app-db
incus move oc-node-01:app-db oc-node-01:app-db --storage shared
incus start oc-node-01:app-db
```

**Note:** This copies all data from the source pool to the destination pool.
For large VMs, this can take time depending on disk size and pool speeds.

---

## Section 11: Future — Multi-Host and Upgrades

### Adding a second Proxmox host

With shared iSCSI storage, adding a second Proxmox host enables true cross-
host live migration:

```mermaid
flowchart TD
    subgraph hostA["Proxmox Host A"]
        a1["oc-node-01"]
        a2["oc-node-02"]
    end

    subgraph hostB["Proxmox Host B"]
        b1["oc-node-03"]
        b2["oc-node-04"]
    end

    qnap["QNAP NAS<br/>iSCSI target<br/>Shared LUN"]

    hostA <-->|"iSCSI"| qnap
    hostB <-->|"iSCSI"| qnap

    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e

    class a1,a2,b1,b2 node
    class qnap network

    style hostA fill:#f5f5f5,stroke:#999
    style hostB fill:#f5f5f5,stroke:#999
```

Requirements:
- QNAP NAS (or other external iSCSI target) accessible from both hosts
- Same iSCSI IQN and LUN on all nodes
- Nodes on the same Incus cluster (already the case)

### 2.5GbE upgrade path

When the 2.5GbE switch arrives:
- iSCSI throughput increases from ~110 MB/s to ~275 MB/s
- Random IOPS may improve slightly (lower network latency)
- Migration time (already sub-second) is unchanged
- Consider a dedicated storage VLAN for iSCSI traffic separation

### Dedicated storage VLAN

For production, isolate iSCSI traffic on its own VLAN:

```
VLAN 69  (192.168.102.x) — management + OVN tunnels
VLAN 70  (10.69.0.x)     — iSCSI storage traffic (dedicated)
```

This prevents storage I/O from competing with management and OVN traffic.
Requires a second NIC or VLAN trunking on each node.

### Ceph (reference only)

IncusOS also includes a Ceph client service. Ceph provides distributed
storage with replication and is the standard for large-scale deployments.
However, it requires:

- Minimum 3 OSD nodes with dedicated disks
- Significant RAM and CPU overhead
- More complexity to operate

For a home lab with a NAS already available, iSCSI + lvmcluster is simpler
and more resource-efficient. Ceph is better suited for multi-rack deployments
where you need storage to survive node failures without a single NAS.

---

## Section 12: Troubleshooting

### Wrong API field name (`iqn` vs `target`)

**Symptom:** iSCSI service shows `enabled: true` but no `initiator_name` in
the state, and no block device appears.

**Cause:** Using `"iqn"` instead of `"target"` in the PUT request. The API
silently accepts the wrong field and stores an empty target string.

**How to check:**

```bash
incus query oc-node-01:/os/1.0/services/iscsi
```

If the target field is empty (`"target": ""`), you used the wrong field name.

**Fix:** Re-submit the PUT request with `"target"` (not `"iqn"`):

```json
{
    "config": {
        "enabled": true,
        "targets": [{
            "address": "10.207.217.19",
            "port": 3260,
            "target": "iqn.2026-02.lab.incus:storage.shared"
        }]
    },
    "state": {}
}
```

### Proxy devices don't work for iSCSI

**Symptom:** iSCSI service returns "No records found" (exit status 21) when
using an Incus proxy device to expose the target's port 3260.

**Cause:** IncusOS runs `iscsiadm -m discovery -t sendtargets` before login.
The iSCSI target (tgt/targetcli) responds with its own IP in the SendTargets
response. Through a proxy device, the target returns the container's internal
IP (e.g., `10.207.217.19`) — but the node connected to the proxy's host IP
(e.g., `192.168.102.140`). The portal mismatch causes the login to fail.

**Fix:** Use direct network connectivity instead of proxy devices. The lab
target container uses dual network paths (bridge + macvlan) as described in
Section 2.

### LVM system_id must be 1-2000

**Symptom:** `incus storage create` fails during pool finalization with
`"Invalid host_id 0, use 1-2000"`.

**Cause:** LVM service enabled without specifying `system_id`, or `system_id`
set to 0. Sanlock requires a unique host ID in the range 1-2000.

**Fix:** Set unique `system_id` values on each node:

```bash
incus query oc-node-01:/os/1.0/services/lvm --request PUT --data '{
    "config": { "enabled": true, "system_id": 1 },
    "state": {}
}'
# system_id: 2 for node-02, 3 for node-03
```

### First live migration fails after stop/start

**Symptom:** Live migration fails with `exit status 1` (QEMU cannot start on
the destination). The error includes a long QEMU command line ending with
`-incoming defer ... exit status 1`.

**Cause:** Appears to be a transient sanlock lease timing issue. After a VM
is stopped and restarted (or after a non-live migration), the first live
migration attempt may fail because the destination node's sanlock lease for
the LV hasn't fully registered.

**Fix:** Retry the live migration — subsequent attempts succeed. Non-live
migration (stop/move/start) always works as a fallback.

### Insufficient space on shared pool

**Symptom:** Instance launch fails with space-related errors.

**Cause:** lvmcluster uses **thick provisioning**. A 10 GiB VM root disk
immediately allocates 10 GiB on the LUN. With a 20 GiB test LUN, you can
fit ~3 VMs with 4 GiB roots (accounting for LVM/sanlock metadata overhead
of ~256 MiB).

**Fix:** Use explicit small sizes for test VMs:

```bash
incus launch ... -d root,size=4GiB -d root,size.state=2GiB
```

Or use a larger LUN (200+ GiB for production).

### lvmcluster pool creation errors

| Error | Cause | Fix |
|-------|-------|-----|
| `device not found` | Wrong device path | Verify via storage API (Section 4.4) |
| `VG already exists` | Previous attempt left VG | Use existing VG name in `source=` |
| `lockd not running` | LVM service not enabled | Enable with `system_id` (Section 5) |
| `Invalid host_id 0` | Missing `system_id` | Set unique `system_id` 1-2000 per node |

### Recovery from target container/VM/NAS failure

If the iSCSI target goes down:

1. **All instances on `shared` pool hang** — I/O operations block until the
   target returns or the iSCSI timeout expires
2. **Instances on `local` pool are unaffected** — this is why the hybrid
   architecture matters
3. **When the target recovers**: iSCSI initiators reconnect automatically
4. **If the target is permanently lost**: shared pool data is gone. Delete
   the pool and recreate from a new target

### Target container maintenance

```bash
# 1. List instances on the shared pool
incus list oc-node-01: --format json | python3 -c "
import sys, json
for inst in json.load(sys.stdin):
    for dev in inst.get('expanded_devices', {}).values():
        if dev.get('pool') == 'shared':
            print(f\"  {inst['name']} ({inst['status']}) on {inst['location']}\")
            break
"

# 2. Stop all shared-pool instances
# 3. Stop the target container/VM/NAS
# 4. Perform maintenance
# 5. Start the target
# 6. Wait ~30s for iSCSI reconnection
# 7. Start instances
```

---

## Section 13: 1GbE Performance Assessment

### Theoretical limits

- 1GbE raw: 125 MB/s
- TCP/iSCSI overhead: ~10-15%
- Practical maximum: **106-112 MB/s** sequential

### What we actually measured

| Operation | Local ZFS | Shared lvmcluster | Observation |
|-----------|-----------|-------------------|-------------|
| VM live migration (1 GiB RAM) | 7.0s | **6.0-6.4s** | RAM transfer at ~141 MB/s |
| VM non-live migration | 2.1s | **1.8s** | Metadata only on shared |
| Container non-live migration | N/A | **0.12-0.15s** | Near-instant |
| VM boot time | ~20s | ~20s | No noticeable difference |
| Migration transfer rate | ~141 MB/s | ~141 MB/s | Same network, same rate |

**The migration speed difference grows with disk size.** Our test used a small
4 GiB VM root. With a 50 GiB root disk on local ZFS, migration must transfer
all used data — potentially minutes. On shared storage, it's always 6 seconds
(only RAM).

### Expected I/O performance (not yet benchmarked)

| Scenario | Throughput | Notes |
|----------|-----------|-------|
| Single node sequential I/O | ~100-110 MB/s | Near wire speed |
| 3 nodes concurrent I/O | ~33-37 MB/s per node | Shared bandwidth |
| Random 4K IOPS | 5,000-15,000 | Target-dependent, not network-bound |
| I/O latency | 0.5-1.5ms | vs ~0.1ms for local ZFS |

### Verdict

**1GbE is viable for a home lab.** The primary goal — eliminating data copy
during migration — is fully achieved regardless of network speed. Day-to-day
I/O is slower than local ZFS but acceptable for lab workloads. The hybrid
architecture (local ZFS for general use, shared for HA) minimizes the impact.

2.5GbE will improve throughput ~2.5x when the switch arrives but is not a
prerequisite for this setup.

### sanlock overhead

Minimal in testing. sanlock uses ~100 IOPS for lease renewal (every 20
seconds) and only holds locks during LVM metadata operations (create, delete,
resize LV). Normal I/O to the logical volumes bypasses sanlock entirely. The
256 MiB overhead on the 20 GiB LUN is all metadata + lease storage.