1614 lines
51 KiB
Markdown
1614 lines
51 KiB
Markdown
# Shared Storage Guide — iSCSI + lvmcluster on IncusOS
|
|
|
|
Add shared storage to an existing IncusOS cluster using iSCSI targets and the
|
|
`lvmcluster` Incus storage driver. Shared storage eliminates disk data copy
|
|
during migration — live migration only transfers RAM state, and non-live
|
|
migration becomes a sub-second metadata operation.
|
|
|
|
All commands and output in this guide are from an actual deployment on
|
|
2026-02-23. Tested on an OC-managed 3-node IncusOS cluster (build 202602230420)
|
|
on Proxmox VE 9.1.5, Incus client 6.21.
|
|
|
|
This guide covers two iSCSI target options: a self-contained lab target
|
|
container (no external hardware) and a QNAP NAS (production-like). Both use
|
|
IncusOS's built-in iSCSI initiator and LVM services — no packages to install.
|
|
|
|
---
|
|
|
|
## Section 0: Architecture Overview
|
|
|
|
### Why shared storage?
|
|
|
|
With local ZFS pools, migration must copy the full root disk between nodes.
|
|
With shared storage, the disk is already accessible from all nodes:
|
|
|
|
```
|
|
Non-live migration (stop/move/start):
|
|
Local ZFS: copy disk data at ~140 MB/s ≈ varies with disk size
|
|
Shared storage: metadata update only ≈ 0.1-2 seconds
|
|
|
|
Live migration (VM stays running):
|
|
Local ZFS: copy disk + RAM at ~140 MB/s ≈ 7s+ (grows with disk)
|
|
Shared storage: copy RAM only at ~140 MB/s ≈ 5-7s (constant)
|
|
```
|
|
|
|
**Key insight from testing:** live migration always transfers VM RAM (~1 GiB
|
|
at ~140 MB/s ≈ 5-7 seconds). Shared storage eliminates the DISK transfer —
|
|
the win grows with disk size. A 100 GiB VM on local ZFS takes minutes to
|
|
migrate; on shared storage it's still just 5-7 seconds (RAM only).
|
|
|
|
Non-live migration (stop/move/start) is where shared storage truly shines:
|
|
**0.1-2 seconds** regardless of disk size, compared to minutes for large VMs
|
|
on local ZFS.
|
|
|
|
### How iSCSI + lvmcluster works
|
|
|
|
```
|
|
┌──────────────────┐ iSCSI (1GbE) ┌──────────────────┐
|
|
│ iSCSI Target │◄───────────────────────►│ oc-node-01 │
|
|
│ (QNAP / VM / │◄───────────────────────►│ oc-node-02 │
|
|
│ any target) │◄───────────────────────►│ oc-node-03 │
|
|
│ │ │ │
|
|
│ Exposes LUN(s) │ │ iSCSI initiator │
|
|
│ as block device │ │ (IncusOS built-in)│
|
|
└──────────────────┘ │ │
|
|
│ lvmlockd+sanlock│
|
|
│ (IncusOS built-in)│
|
|
│ │
|
|
│ Incus lvmcluster│
|
|
│ storage pool │
|
|
└──────────────────┘
|
|
```
|
|
|
|
1. **iSCSI target** exports a block device (LUN) over the network
|
|
2. **iSCSI initiators** (IncusOS built-in) connect from each node — the LUN
|
|
appears as a local block device (`/dev/sd*`)
|
|
3. **LVM with sanlock** (`lvmlockd`) provides distributed locking so multiple
|
|
nodes can safely use the shared block device
|
|
4. **Incus `lvmcluster` driver** creates logical volumes (LVs) on the shared
|
|
volume group for instance root disks and custom volumes
|
|
5. **Live migration** is metadata-only — the destination node already has
|
|
access to the same LV on the same shared device
|
|
|
|
### Hybrid architecture: local ZFS + shared lvmcluster
|
|
|
|
The recommended setup uses **both** pools:
|
|
|
|
| Pool | Type | Use case | Provisioning |
|
|
|------|------|----------|--------------|
|
|
| `local` | ZFS (per-node) | OS images, containers, non-HA workloads | Thin (copy-on-write) |
|
|
| `shared` | lvmcluster (shared) | HA VMs, workloads needing instant migration | Thick (full allocation) |
|
|
|
|
**Why hybrid?**
|
|
- Local ZFS has thin provisioning, snapshots, and fast local I/O
|
|
- lvmcluster has no thin provisioning (10 GiB VM = 10 GiB on LUN)
|
|
- Keep OS images and ephemeral containers on local ZFS
|
|
- Put only HA VMs that need instant migration on the shared pool
|
|
|
|
### Decision matrix: when to use which pool
|
|
|
|
| Scenario | Pool | Why |
|
|
|----------|------|-----|
|
|
| Development container | `local` | Fast, thin provisioned, no HA needed |
|
|
| Production VM with HA | `shared` | Instant migration, zero data copy |
|
|
| OVN control plane container | `local` | Pinned to one node, no migration |
|
|
| Database VM (HA) | `shared` | Needs failover without data copy |
|
|
| Temporary test instance | `local` | Ephemeral, don't waste shared space |
|
|
|
|
### Network topology (tested)
|
|
|
|
```
|
|
┌─ Proxmox host ──────────────────────────────────────────────────────┐
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────────────┐ │
|
|
│ │ oc-node-01 (VMID 400, 192.168.102.140) │ │
|
|
│ │ ┌──────────────┐ │ │
|
|
│ │ │iscsi-target │ Debian container running tgt │ │
|
|
│ │ │10.207.217.19 │ incusbr0 (bridge, same-node access) │ │
|
|
│ │ │.102.150 │ macvlan on mgmt (cross-node access) │ │
|
|
│ │ │LUN: 20 GiB │◄─── iSCSI initiator (oc-node-01 via bridge)│ │
|
|
│ │ └──────────────┘◄─── iSCSI initiator (oc-node-02 via macvlan)│ │
|
|
│ │ ◄─── iSCSI initiator (oc-node-03 via macvlan)│ │
|
|
│ └──────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌──────────┐ ┌──────────┐ │
|
|
│ │oc-node-02│ │oc-node-03│ │
|
|
│ │ VMID 401 │ │ VMID 402 │ │
|
|
│ │ .102.141 │ │ .102.142 │ │
|
|
│ │ lvmclust │ │ lvmclust │ │
|
|
│ └──────────┘ └──────────┘ │
|
|
│ │
|
|
│ vmbr0 (VLAN 69) ── 192.168.100.0/22 │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Lab target container** (Option A, tested): a Debian container on oc-node-01
|
|
running `tgt` (userspace iSCSI target). Uses two network paths: bridge IP for
|
|
same-node access, macvlan on `mgmt` for cross-node access.
|
|
|
|
**QNAP NAS** (Option B, not yet tested): replace the target container with
|
|
the QNAP at 192.168.1.x. Requires routing between VLAN 69 and the native LAN.
|
|
|
|
### Infrastructure (Option A: Lab target container)
|
|
|
|
| Component | IP | Role |
|
|
|-----------|-----|------|
|
|
| iscsi-target (container on oc-node-01) | 10.207.217.19 (bridge), 192.168.102.150 (macvlan) | iSCSI target (tgt) |
|
|
| oc-node-01 (VMID 400) | 192.168.102.140/22 | Cluster init + iSCSI via bridge |
|
|
| oc-node-02 (VMID 401) | 192.168.102.141/22 | Cluster member + iSCSI via macvlan |
|
|
| oc-node-03 (VMID 402) | 192.168.102.142/22 | Cluster member + iSCSI via macvlan |
|
|
|
|
**Additional RAM**: negligible (~20 MiB for the container). No extra Proxmox VM needed.
|
|
|
|
### Cross-references
|
|
|
|
- [Clustering guide](clustering-guide.md) — manual cluster formation
|
|
- [Networking guide](networking-guide.md) — OVN overlay tutorial
|
|
- [Operations Center guide](operations-center-guide.md) — OC-managed clusters
|
|
- [Production lab guide](production-lab-guide.md) — manual cluster + OVN + HA
|
|
|
|
---
|
|
|
|
## Section 1: Prerequisites
|
|
|
|
### Existing cluster
|
|
|
|
This guide assumes you have a working 3-node IncusOS cluster (OC-managed or
|
|
manual). The examples use the OC-managed cluster from the
|
|
[Operations Center guide](operations-center-guide.md):
|
|
|
|
```bash
|
|
incus cluster list oc-node-01:
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
+------------+------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|
|
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
|
|
+------------+------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|
|
| oc-node-01 | https://192.168.102.140:8443 | ovn-chassis | x86_64 | default | | ONLINE | Fully operational |
|
|
| | | database-leader | | | | | |
|
|
| | | database | | | | | |
|
|
+------------+------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|
|
| oc-node-02 | https://192.168.102.141:8443 | ovn-chassis | x86_64 | default | | ONLINE | Fully operational |
|
|
| | | database | | | | | |
|
|
+------------+------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|
|
| oc-node-03 | https://192.168.102.142:8443 | ovn-chassis | x86_64 | default | | ONLINE | Fully operational |
|
|
| | | database | | | | | |
|
|
+------------+------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|
|
```
|
|
|
|
The `ovn-chassis` roles are from the [networking guide](networking-guide.md) —
|
|
they don't affect shared storage setup.
|
|
|
|
### IncusOS built-in services
|
|
|
|
IncusOS is immutable — you cannot install packages. But these services are
|
|
built in and enabled via the REST API:
|
|
|
|
| Service | API Endpoint | Purpose |
|
|
|---------|-------------|---------|
|
|
| **iSCSI** | `/os/1.0/services/iscsi` | Initiator — connects to external iSCSI targets |
|
|
| **LVM** | `/os/1.0/services/lvm` | Enables `lvmlockd` + `sanlock` for clustered LVM |
|
|
| **OVN** | `/os/1.0/services/ovn` | OVN controller (already configured if using OVN) |
|
|
| **Ceph** | `/os/1.0/services/ceph` | Ceph client (alternative to iSCSI, not covered here) |
|
|
| **Multipath** | `/os/1.0/services/multipath` | Redundant I/O paths (not needed for single-path lab) |
|
|
|
|
Check available services on a node:
|
|
|
|
```bash
|
|
incus query oc-node-01:/os/1.0/services
|
|
```
|
|
|
|
### Required tools
|
|
|
|
```bash
|
|
# Incus client
|
|
incus version
|
|
# Client version: 6.21
|
|
|
|
# Optional: incusos-proxmox for deploying the target VM
|
|
./incusos/incusos-proxmox --doctor
|
|
```
|
|
|
|
### Network requirements
|
|
|
|
- All cluster nodes must reach the iSCSI target on TCP port 3260
|
|
- For lab target VM: all on VLAN 69 — no routing needed
|
|
- For QNAP NAS: routing between VLAN 69 (192.168.102.x) and native LAN
|
|
(192.168.1.x) via gateway or dedicated NIC
|
|
|
|
---
|
|
|
|
## Section 2: Option A — Lab Target Container (Self-Contained, Tested)
|
|
|
|
Deploy a Debian container on the Incus cluster running `tgt` (userspace iSCSI
|
|
target). No external hardware or Proxmox VM needed — everything runs inside
|
|
the existing cluster.
|
|
|
|
**Why a container instead of a Proxmox VM?** Faster to deploy, no Proxmox
|
|
manual steps, and validates the same iSCSI concepts. The container uses `tgt`
|
|
(a userspace iSCSI target daemon) which works in unprivileged containers
|
|
without kernel modules.
|
|
|
|
**Why not use Incus proxy devices?** Tested and failed. The IncusOS iSCSI
|
|
service uses `iscsiadm -m discovery -t sendtargets` before login. Through a
|
|
proxy device, the target responds with its container IP in the SendTargets
|
|
response, creating node records with the wrong portal address. The subsequent
|
|
login to the proxy IP fails with "No records found" (exit status 21).
|
|
|
|
**Solution: dual network paths.** The container gets two interfaces:
|
|
- `eth0` on `incusbr0` (bridge, 10.207.217.x) — used by oc-node-01 (same node)
|
|
- `eth1` as macvlan on `mgmt` (192.168.102.150) — used by oc-node-02/03
|
|
|
|
### 2.1 Launch the iSCSI target container
|
|
|
|
```bash
|
|
incus launch images:debian/12 oc-node-01:iscsi-target --target oc-node-01
|
|
```
|
|
|
|
### 2.2 Install tgt
|
|
|
|
```bash
|
|
incus exec oc-node-01:iscsi-target -- apt-get update -qq
|
|
incus exec oc-node-01:iscsi-target -- apt-get install -y tgt
|
|
```
|
|
|
|
Actual output (key lines):
|
|
|
|
```
|
|
Setting up tgt (1:1.0.85-1+deb12u1) ...
|
|
Created symlink /etc/systemd/system/multi-user.target.wants/tgt.service → ...
|
|
```
|
|
|
|
The RDMA and oom_score warnings in the service log are expected in a container
|
|
and can be safely ignored:
|
|
|
|
```
|
|
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
|
|
can't adjust oom-killer's pardon /proc/self/oom_score_adj, Permission denied
|
|
```
|
|
|
|
### 2.3 Create the backing store and configure the target
|
|
|
|
```bash
|
|
incus exec oc-node-01:iscsi-target -- bash -c '
|
|
# Create sparse file (only uses actual written data on disk)
|
|
mkdir -p /srv/iscsi
|
|
truncate -s 20G /srv/iscsi/shared-lun.img
|
|
|
|
# Create iSCSI target
|
|
tgtadm --lld iscsi --op new --mode target --tid 1 \
|
|
-T iqn.2026-02.lab.incus:storage.shared
|
|
|
|
# Add LUN (LUN 0 is reserved for the controller, use LUN 1)
|
|
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 \
|
|
-b /srv/iscsi/shared-lun.img
|
|
|
|
# Allow all initiators (lab environment)
|
|
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
|
|
|
|
# Verify
|
|
tgtadm --lld iscsi --op show --mode target
|
|
'
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
Target 1: iqn.2026-02.lab.incus:storage.shared
|
|
System information:
|
|
Driver: iscsi
|
|
State: ready
|
|
I_T nexus information:
|
|
LUN information:
|
|
LUN: 0
|
|
Type: controller
|
|
SCSI ID: IET 00010000
|
|
SCSI SN: beaf10
|
|
Size: 0 MB, Block size: 1
|
|
Online: Yes
|
|
Removable media: No
|
|
Prevent removal: No
|
|
Readonly: No
|
|
SWP: No
|
|
Thin-provisioning: No
|
|
Backing store type: null
|
|
Backing store path: None
|
|
Backing store flags:
|
|
LUN: 1
|
|
Type: disk
|
|
SCSI ID: IET 00010001
|
|
SCSI SN: beaf11
|
|
Size: 21475 MB, Block size: 512
|
|
Online: Yes
|
|
Removable media: No
|
|
Prevent removal: No
|
|
Readonly: No
|
|
SWP: No
|
|
Thin-provisioning: No
|
|
Backing store type: rdwr
|
|
Backing store path: /srv/iscsi/shared-lun.img
|
|
Backing store flags:
|
|
Account information:
|
|
ACL information:
|
|
ALL
|
|
```
|
|
|
|
### 2.4 Make the target persistent
|
|
|
|
```bash
|
|
incus exec oc-node-01:iscsi-target -- bash -c 'cat > /etc/tgt/conf.d/shared-lun.conf << '\''EOF'\''
|
|
<target iqn.2026-02.lab.incus:storage.shared>
|
|
backing-store /srv/iscsi/shared-lun.img
|
|
initiator-address ALL
|
|
</target>
|
|
EOF'
|
|
```
|
|
|
|
### 2.5 Add macvlan NIC for cross-node access
|
|
|
|
The container is on `incusbr0` which is node-local — oc-node-02 and oc-node-03
|
|
cannot reach it. Add a macvlan NIC on the management interface:
|
|
|
|
```bash
|
|
# Add macvlan NIC on IncusOS management interface
|
|
incus config device add oc-node-01:iscsi-target mgmt-nic nic \
|
|
nictype=macvlan parent=mgmt
|
|
|
|
# Configure static IP inside the container
|
|
incus exec oc-node-01:iscsi-target -- ip link set eth1 up
|
|
incus exec oc-node-01:iscsi-target -- ip addr add 192.168.102.150/22 dev eth1
|
|
```
|
|
|
|
**macvlan limitation**: the host (oc-node-01) cannot reach 192.168.102.150
|
|
through the macvlan (kernel filters traffic between macvlan and parent).
|
|
That's why oc-node-01 connects via the bridge IP (10.207.217.x) instead.
|
|
|
|
### 2.6 Verify the target is listening and reachable
|
|
|
|
```bash
|
|
# Verify tgt is listening
|
|
incus exec oc-node-01:iscsi-target -- ss -tlnp | grep 3260
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
LISTEN 0 4096 0.0.0.0:3260 0.0.0.0:* users:(("tgtd",pid=840,fd=6))
|
|
LISTEN 0 4096 [::]:3260 [::]:* users:(("tgtd",pid=840,fd=7))
|
|
```
|
|
|
|
**Note on LUN sizing**: 20 GiB is sufficient for testing (2-3 small VMs).
|
|
lvmcluster uses thick provisioning — a 4 GiB VM root + 2 GiB state area =
|
|
6.25 GiB on the LUN (including LVM/sanlock metadata). For production, use
|
|
200+ GiB on the QNAP.
|
|
|
|
---
|
|
|
|
## Section 3: Option B — QNAP iSCSI Target (Production-Like)
|
|
|
|
Use the QNAP NAS's built-in iSCSI target service instead of a VM. This is
|
|
more realistic for production but requires physical access and VLAN routing.
|
|
|
|
### 3.1 Create iSCSI target on QNAP
|
|
|
|
Via the QNAP web UI (QTS):
|
|
|
|
1. Open **iSCSI & Fibre Channel** app (install from App Center if missing)
|
|
2. **Storage** > Create a new LUN:
|
|
- Name: `incus-shared`
|
|
- Size: 200 GiB
|
|
- Provisioning: Thick (recommended) or Thin
|
|
3. **Target** > Create a new target:
|
|
- Name: `iqn.2026-02.nas.qnap:incus-shared`
|
|
- CHAP: disabled (lab environment)
|
|
4. Map the LUN to the target
|
|
5. Note the QNAP's IP address (e.g., `192.168.1.100`)
|
|
|
|
### 3.2 Network routing (VLAN 69 to native LAN)
|
|
|
|
IncusOS VMs on VLAN 69 (192.168.102.x/22) need to reach the QNAP on the
|
|
native LAN (192.168.1.x). Options:
|
|
|
|
**Option 1: Route via gateway (simplest)**
|
|
|
|
If your router/gateway (192.168.100.1) handles both VLANs, the VMs can
|
|
already reach 192.168.1.x. Test from a node:
|
|
|
|
```bash
|
|
incus exec oc-node-01:test-ping -- ping -c 3 192.168.1.100
|
|
```
|
|
|
|
Or test via the IncusOS API (since IncusOS has no shell):
|
|
|
|
```bash
|
|
# Check if the QNAP port is reachable from a container on the node
|
|
incus launch images:debian/12 oc-node-01:test-ping --target oc-node-01
|
|
incus exec oc-node-01:test-ping -- apt-get update -qq
|
|
incus exec oc-node-01:test-ping -- apt-get install -y -qq iputils-ping
|
|
incus exec oc-node-01:test-ping -- ping -c 3 192.168.1.100
|
|
incus delete oc-node-01:test-ping --force
|
|
```
|
|
|
|
**Option 2: Dedicated NIC on QNAP**
|
|
|
|
If the QNAP has a second NIC port, connect it to the VLAN 69 network
|
|
and assign an IP in the 192.168.102.x/22 range (e.g., 192.168.102.200).
|
|
|
|
### 3.3 Verify connectivity
|
|
|
|
From each IncusOS node, verify the iSCSI target port is reachable. Since
|
|
IncusOS has no shell, verify after enabling the iSCSI service (Section 4)
|
|
by checking if the target discovery succeeds.
|
|
|
|
---
|
|
|
|
## Section 4: Enable IncusOS iSCSI Service
|
|
|
|
Enable the built-in iSCSI initiator on every cluster node. This must be done
|
|
before any iSCSI target can be connected.
|
|
|
|
### 4.1 Check current iSCSI service state
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== $node ==="
|
|
incus query ${node}:/os/1.0/services/iscsi
|
|
done
|
|
```
|
|
|
|
Output (before enabling):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": false
|
|
},
|
|
"state": {}
|
|
}
|
|
```
|
|
|
|
### 4.2 Enable iSCSI and connect to the target
|
|
|
|
**Critical: the API field is `target`, not `iqn`.** The JSON key for the IQN
|
|
is `"target"`. Using `"iqn"` silently saves an empty target string, and the
|
|
service enables but cannot connect. This is the most common mistake.
|
|
|
|
**For lab target container (Option A):**
|
|
|
|
oc-node-01 connects via the bridge IP (same node); oc-node-02 and oc-node-03
|
|
connect via the macvlan IP (cross-node):
|
|
|
|
```bash
|
|
TARGET_IQN="iqn.2026-02.lab.incus:storage.shared"
|
|
|
|
# oc-node-01: connect via bridge IP (macvlan doesn't work to own parent)
|
|
echo "=== Enabling iSCSI on oc-node-01 ==="
|
|
incus query oc-node-01:/os/1.0/services/iscsi --request PUT --data "{
|
|
\"config\": {
|
|
\"enabled\": true,
|
|
\"targets\": [
|
|
{
|
|
\"address\": \"10.207.217.19\",
|
|
\"port\": 3260,
|
|
\"target\": \"${TARGET_IQN}\"
|
|
}
|
|
]
|
|
},
|
|
\"state\": {}
|
|
}"
|
|
|
|
# oc-node-02 and oc-node-03: connect via macvlan IP
|
|
for node in oc-node-02 oc-node-03; do
|
|
echo "=== Enabling iSCSI on $node ==="
|
|
incus query ${node}:/os/1.0/services/iscsi --request PUT --data "{
|
|
\"config\": {
|
|
\"enabled\": true,
|
|
\"targets\": [
|
|
{
|
|
\"address\": \"192.168.102.150\",
|
|
\"port\": 3260,
|
|
\"target\": \"${TARGET_IQN}\"
|
|
}
|
|
]
|
|
},
|
|
\"state\": {}
|
|
}"
|
|
done
|
|
```
|
|
|
|
**For QNAP (Option B):**
|
|
|
|
All nodes use the same QNAP IP:
|
|
|
|
```bash
|
|
TARGET_IP="192.168.1.100" # Or 192.168.102.200 if dedicated NIC
|
|
TARGET_IQN="iqn.2026-02.nas.qnap:incus-shared" # From QNAP web UI
|
|
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== Enabling iSCSI on $node ==="
|
|
incus query ${node}:/os/1.0/services/iscsi --request PUT --data "{
|
|
\"config\": {
|
|
\"enabled\": true,
|
|
\"targets\": [
|
|
{
|
|
\"address\": \"${TARGET_IP}\",
|
|
\"port\": 3260,
|
|
\"target\": \"${TARGET_IQN}\"
|
|
}
|
|
]
|
|
},
|
|
\"state\": {}
|
|
}"
|
|
done
|
|
```
|
|
|
|
### 4.3 Verify the iSCSI connection
|
|
|
|
After enabling, verify each node connected successfully:
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== $node ==="
|
|
incus query ${node}:/os/1.0/services/iscsi
|
|
done
|
|
```
|
|
|
|
Actual output (oc-node-01 — via bridge IP):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": true,
|
|
"targets": [
|
|
{
|
|
"address": "10.207.217.19",
|
|
"port": 3260,
|
|
"target": "iqn.2026-02.lab.incus:storage.shared"
|
|
}
|
|
]
|
|
},
|
|
"state": {
|
|
"initiator_name": "iqn.2004-10.org.linuxcontainers:01:2390dbbbee72"
|
|
}
|
|
}
|
|
```
|
|
|
|
Actual output (oc-node-02 — via macvlan IP):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": true,
|
|
"targets": [
|
|
{
|
|
"address": "192.168.102.150",
|
|
"port": 3260,
|
|
"target": "iqn.2026-02.lab.incus:storage.shared"
|
|
}
|
|
]
|
|
},
|
|
"state": {
|
|
"initiator_name": "iqn.2004-10.org.linuxcontainers:01:3c58d1566dc1"
|
|
}
|
|
}
|
|
```
|
|
|
|
**How to tell it worked:** the `state` section contains `initiator_name`
|
|
(auto-generated unique IQN). If `state` is empty `{}`, the connection
|
|
failed — check the target address and that the `target` field (not `iqn`)
|
|
was used.
|
|
|
|
Each node gets a unique `initiator_name` — this is expected and correct.
|
|
The iSCSI target sees 3 separate initiators sharing the same LUN.
|
|
|
|
### 4.4 Identify the iSCSI block device
|
|
|
|
The iSCSI LUN appears as a SCSI disk on each node. Check via the IncusOS
|
|
storage API:
|
|
|
|
```bash
|
|
incus query oc-node-01:/os/1.0/system/storage | python3 -c "
|
|
import sys, json
|
|
d = json.load(sys.stdin)
|
|
for drive in d['state']['drives']:
|
|
size_gib = drive['capacity_in_bytes'] / (1024**3)
|
|
print(f\" {drive['id']} - {size_gib:.0f} GiB - {drive['model_name']}\")"
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
/dev/disk/by-id/lvm-pv-uuid-rUETjP-5LQr-0g9y-KWD8-cxB9-QsOv-ieIcru - 20 GiB - VIRTUAL-DISK
|
|
/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 - 64 GiB - QEMU HARDDISK
|
|
```
|
|
|
|
The 20 GiB `VIRTUAL-DISK` is the iSCSI LUN. The 64 GiB `QEMU HARDDISK` is
|
|
the system drive. Note the LVM PV UUID path — this means the device is already
|
|
an LVM physical volume (Incus created it when we set up the pool).
|
|
|
|
**The device also appears as `/dev/sdb` on all 3 nodes** — consistent because
|
|
all VMs have the same hardware layout (scsi0 = system, sdb = iSCSI LUN).
|
|
|
|
You don't need to note the exact path — Incus handles device references
|
|
internally when creating the lvmcluster pool (Section 7).
|
|
|
|
---
|
|
|
|
## Section 5: Enable IncusOS LVM Service
|
|
|
|
Enable the LVM service with `lvmlockd` and `sanlock` for clustered LVM.
|
|
This provides distributed locking so multiple nodes can safely share the
|
|
same volume group.
|
|
|
|
### 5.1 Check current LVM service state
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== $node ==="
|
|
incus query ${node}:/os/1.0/services/lvm
|
|
done
|
|
```
|
|
|
|
Output (before enabling):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": false
|
|
},
|
|
"state": {}
|
|
}
|
|
```
|
|
|
|
### 5.2 Enable LVM with unique system_id per node
|
|
|
|
**Critical: each node must have a unique `system_id` between 1 and 2000.**
|
|
This is the sanlock host ID. Using `0` or duplicates will cause pool creation
|
|
to fail with `"Invalid host_id 0, use 1-2000"`.
|
|
|
|
```bash
|
|
id=1
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== Enabling LVM on $node (system_id=$id) ==="
|
|
incus query ${node}:/os/1.0/services/lvm --request PUT --data "{
|
|
\"config\": {
|
|
\"enabled\": true,
|
|
\"system_id\": $id
|
|
},
|
|
\"state\": {}
|
|
}"
|
|
id=$((id + 1))
|
|
done
|
|
```
|
|
|
|
### 5.3 Verify LVM services are running
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== $node ==="
|
|
incus query ${node}:/os/1.0/services/lvm
|
|
done
|
|
```
|
|
|
|
Actual output (oc-node-01, after pool creation — `state` shows VG info):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": true,
|
|
"system_id": 1
|
|
},
|
|
"state": {
|
|
"pvs": [
|
|
{
|
|
"pv_attr": "a--",
|
|
"pv_fmt": "lvm2",
|
|
"pv_free": "<19.75g",
|
|
"pv_name": "/dev/sdb",
|
|
"pv_size": "<20.00g",
|
|
"vg_name": "shared"
|
|
}
|
|
],
|
|
"vgs": [
|
|
{
|
|
"lv_count": 0,
|
|
"pv_count": 1,
|
|
"snap_count": 0,
|
|
"vg_attr": "wz--ns",
|
|
"vg_free": "<19.75g",
|
|
"vg_name": "shared",
|
|
"vg_size": "<20.00g"
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
**Before pool creation**, the `state` section will show empty `pvs` and `vgs`
|
|
arrays. That's normal — the VG doesn't exist yet.
|
|
|
|
**Key fields**: `system_id` is unique per node (1, 2, 3). The `vg_attr`
|
|
`wz--ns` means: writeable, resizable, no allocation policy, not partial,
|
|
shared (`s`). The `s` at the end confirms sanlock locking is active.
|
|
|
|
The `vgs` section showing the same VG name on all 3 nodes confirms shared
|
|
access — all nodes see the same volume group through their iSCSI connections.
|
|
|
|
---
|
|
|
|
## Section 6: Create Shared Volume Group
|
|
|
|
**You don't need to create the VG manually.** The Incus `lvmcluster` driver
|
|
handles `pvcreate`, `vgcreate --shared`, and `vgchange --lock-start`
|
|
automatically during pool creation (Section 7).
|
|
|
|
This section is included for reference — skip to Section 7 unless you need
|
|
to understand the underlying LVM operations.
|
|
|
|
### 6.1 What Incus does automatically
|
|
|
|
When you create an `lvmcluster` pool with `source=/dev/disk/by-id/scsi-...`,
|
|
Incus runs the equivalent of:
|
|
|
|
```bash
|
|
# 1. Create physical volume on the iSCSI device (one node)
|
|
pvcreate /dev/sdb
|
|
|
|
# 2. Create shared VG with sanlock lock type (one node)
|
|
vgcreate --shared <pool-name> /dev/sdb
|
|
|
|
# 3. Start the lock on all nodes (each node)
|
|
vgchange --lock-start <pool-name>
|
|
```
|
|
|
|
The `--shared` flag creates a VG with `locktype=sanlock`, which uses sanlock
|
|
for distributed locking. `vgchange --lock-start` activates the lock manager
|
|
on each node that needs access to the VG.
|
|
|
|
### 6.2 Verify VG visibility (after pool creation in Section 7)
|
|
|
|
After the pool is created, verify all nodes see the shared VG via the LVM
|
|
service API:
|
|
|
|
```bash
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== $node ==="
|
|
incus query ${node}:/os/1.0/services/lvm | python3 -c "
|
|
import sys, json
|
|
d = json.load(sys.stdin)
|
|
for vg in d.get('state', {}).get('vgs', []):
|
|
print(f\" VG: {vg['vg_name']} Size: {vg['vg_size']} Free: {vg['vg_free']} Attr: {vg['vg_attr']}\")
|
|
"
|
|
done
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
=== oc-node-01 ===
|
|
VG: shared Size: <20.00g Free: <19.75g Attr: wz--ns
|
|
=== oc-node-02 ===
|
|
VG: shared Size: <20.00g Free: <19.75g Attr: wz--ns
|
|
=== oc-node-03 ===
|
|
VG: shared Size: <20.00g Free: <19.75g Attr: wz--ns
|
|
```
|
|
|
|
All 3 nodes see the same VG `shared` (20 GiB, ~256 MiB used for LVM/sanlock
|
|
metadata). The `s` in `vg_attr` confirms sanlock locking is active.
|
|
|
|
---
|
|
|
|
## Section 7: Create lvmcluster Pool in Incus
|
|
|
|
Create the shared storage pool using Incus's `lvmcluster` driver. This
|
|
follows the **two-step cluster pattern**: create a pending entry per member
|
|
with `--target`, then finalize without `--target`.
|
|
|
|
### 7.1 Identify the iSCSI device path
|
|
|
|
Get the stable device path from the storage API (Section 4.4). In our lab:
|
|
|
|
```
|
|
/dev/disk/by-id/scsi-360000000000000000e00000000010001
|
|
```
|
|
|
|
This is a stable SCSI ID path — it won't change across reboots (unlike
|
|
`/dev/sdb` which could shift). You can also use `/dev/sdb` if your setup
|
|
is simple.
|
|
|
|
### 7.2 Create pending pool on each member
|
|
|
|
```bash
|
|
ISCSI_DEVICE="/dev/disk/by-id/scsi-360000000000000000e00000000010001"
|
|
|
|
# Create pending pool entry for each cluster member
|
|
for node in oc-node-01 oc-node-02 oc-node-03; do
|
|
echo "=== Creating pending pool on $node ==="
|
|
incus storage create oc-node-01:shared lvmcluster \
|
|
source=${ISCSI_DEVICE} \
|
|
--target ${node}
|
|
done
|
|
```
|
|
|
|
Each `--target` call creates a **pending** entry for that member. The pool
|
|
does not become active until finalized. Output for each:
|
|
|
|
```
|
|
Storage pool shared pending on member oc-node-01
|
|
Storage pool shared pending on member oc-node-02
|
|
Storage pool shared pending on member oc-node-03
|
|
```
|
|
|
|
### 7.3 Finalize the pool
|
|
|
|
```bash
|
|
incus storage create oc-node-01:shared lvmcluster
|
|
```
|
|
|
|
Output:
|
|
|
|
```
|
|
Storage pool shared created
|
|
```
|
|
|
|
Incus handles all LVM operations automatically:
|
|
1. `pvcreate` on the iSCSI device
|
|
2. `vgcreate --shared` to create the volume group with sanlock
|
|
3. `vgchange --lock-start` on all members
|
|
4. Pool metadata registration in the cluster database
|
|
|
|
### 7.4 Verify the pool
|
|
|
|
```bash
|
|
incus storage list oc-node-01:
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
+--------+------------+--------------------------------------+---------+---------+
|
|
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
|
|
+--------+------------+--------------------------------------+---------+---------+
|
|
| local | zfs | Local storage pool (on system drive) | 17 | CREATED |
|
|
+--------+------------+--------------------------------------+---------+---------+
|
|
| shared | lvmcluster | | 1 | CREATED |
|
|
+--------+------------+--------------------------------------+---------+---------+
|
|
```
|
|
|
|
```bash
|
|
incus storage show oc-node-01:shared
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```yaml
|
|
config: {}
|
|
description: ""
|
|
name: shared
|
|
driver: lvmcluster
|
|
used_by:
|
|
- /1.0/profiles/shared-pool
|
|
status: Created
|
|
locations:
|
|
- oc-node-01
|
|
- oc-node-02
|
|
- oc-node-03
|
|
```
|
|
|
|
All 3 locations listed = pool is active on all cluster members.
|
|
|
|
```bash
|
|
# Per-member config (shows the device source and VG name)
|
|
incus storage show oc-node-01:shared --target oc-node-01
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```yaml
|
|
config:
|
|
lvm.vg_name: shared
|
|
source: shared
|
|
volatile.initial_source: /dev/disk/by-id/scsi-360000000000000000e00000000010001
|
|
description: ""
|
|
name: shared
|
|
driver: lvmcluster
|
|
used_by:
|
|
- /1.0/profiles/shared-pool
|
|
status: Created
|
|
locations:
|
|
- oc-node-01
|
|
- oc-node-02
|
|
- oc-node-03
|
|
```
|
|
|
|
```bash
|
|
# Pool space info
|
|
incus storage info oc-node-01:shared --target oc-node-01
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
info:
|
|
description: ""
|
|
driver: lvmcluster
|
|
name: shared
|
|
space used: 256.00MiB
|
|
total space: 20.00GiB
|
|
used by:
|
|
profiles:
|
|
- shared-pool
|
|
```
|
|
|
|
256 MiB used = LVM metadata + sanlock lease area. 19.75 GiB available for
|
|
instance storage.
|
|
|
|
### 7.5 lvmcluster limitations
|
|
|
|
Be aware of these limitations compared to local ZFS:
|
|
|
|
| Feature | Local ZFS | lvmcluster |
|
|
|---------|-----------|------------|
|
|
| Thin provisioning | Yes (copy-on-write) | **No** (thick LVs only) |
|
|
| Snapshots (custom volumes) | Yes | **No** (sanlock limitation) |
|
|
| Snapshots (VM disks) | Yes | Yes (QEMU internal) |
|
|
| Compression | Yes (LZ4) | **No** |
|
|
| Image caching | Yes | Yes |
|
|
| Container support | Yes | Yes |
|
|
| VM support | Yes | Yes |
|
|
|
|
**Thick provisioning means disk space is allocated up front.** A 4 GiB VM root
|
|
disk uses 4 GiB on the LUN immediately. A 10 GiB root disk uses 10 GiB. Plan
|
|
LUN sizing accordingly — a 20 GiB test LUN fits ~3 small VMs with 4 GiB roots.
|
|
|
|
---
|
|
|
|
## Section 8: Test — Launch and Migrate
|
|
|
|
### 8.1 Create a profile for the shared pool
|
|
|
|
```bash
|
|
incus profile create oc-node-01:shared-pool
|
|
|
|
incus profile device add oc-node-01:shared-pool root disk \
|
|
path=/ pool=shared
|
|
|
|
incus profile device add oc-node-01:shared-pool eth0 nic \
|
|
network=incusbr0 name=eth0
|
|
```
|
|
|
|
Verify the profile:
|
|
|
|
```bash
|
|
incus profile show oc-node-01:shared-pool
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```yaml
|
|
config: {}
|
|
description: ""
|
|
devices:
|
|
eth0:
|
|
name: eth0
|
|
network: incusbr0
|
|
type: nic
|
|
root:
|
|
path: /
|
|
pool: shared
|
|
type: disk
|
|
name: shared-pool
|
|
used_by: []
|
|
project: default
|
|
```
|
|
|
|
### 8.2 Test container migration (stop/move/start)
|
|
|
|
Start with a container — it's fast to launch and validates the shared pool
|
|
before testing VMs.
|
|
|
|
```bash
|
|
# Launch a container on the shared pool
|
|
incus launch images:debian/12 oc-node-01:test-shared-ct \
|
|
--profile shared-pool \
|
|
--target oc-node-01
|
|
```
|
|
|
|
Wait for startup, then create a test file at a **persistent** location
|
|
(`/root/`, not `/tmp/` — tmpfs gets cleared on stop/start):
|
|
|
|
```bash
|
|
sleep 5
|
|
incus exec oc-node-01:test-shared-ct -- bash -c \
|
|
'echo "shared storage works" > /root/test.txt'
|
|
```
|
|
|
|
Stop, move, start:
|
|
|
|
```bash
|
|
incus stop oc-node-01:test-shared-ct
|
|
time incus move oc-node-01:test-shared-ct --target oc-node-02
|
|
incus start oc-node-01:test-shared-ct
|
|
```
|
|
|
|
Actual timing:
|
|
|
|
```
|
|
real 0m0.121s
|
|
```
|
|
|
|
**0.121 seconds** — metadata only, no data copy. Verify data persists:
|
|
|
|
```bash
|
|
incus exec oc-node-01:test-shared-ct -- cat /root/test.txt
|
|
```
|
|
|
|
```
|
|
shared storage works
|
|
```
|
|
|
|
Move again (node-02 → node-03):
|
|
|
|
```bash
|
|
incus stop oc-node-01:test-shared-ct
|
|
time incus move oc-node-01:test-shared-ct --target oc-node-03
|
|
incus start oc-node-01:test-shared-ct
|
|
```
|
|
|
|
```
|
|
real 0m0.152s
|
|
```
|
|
|
|
**0.152 seconds.** Data verified intact. Container migration on shared storage
|
|
is effectively instant — the container's root filesystem LV is already
|
|
accessible from all nodes.
|
|
|
|
### 8.3 Launch a VM on the shared pool
|
|
|
|
```bash
|
|
incus launch images:debian/12 oc-node-01:test-shared-vm --vm \
|
|
--profile shared-pool \
|
|
--target oc-node-01 \
|
|
-c limits.cpu=0-1 \
|
|
-c migration.stateful=true \
|
|
-d root,size=4GiB \
|
|
-d root,size.state=2GiB
|
|
```
|
|
|
|
**Important flags explained:**
|
|
- `limits.cpu=0-1` — CPU range (not integer!) required for live migration
|
|
compatibility across nodes with different core counts
|
|
- `migration.stateful=true` — enables live migration (QEMU state transfer)
|
|
- `size=4GiB` — explicit root disk size (thick provisioned, uses 4 GiB on LUN)
|
|
- `size.state=2GiB` — space for VM RAM state during migration
|
|
|
|
Wait for the VM to boot (~20 seconds for first image download):
|
|
|
|
```bash
|
|
incus list oc-node-01: --format compact | grep test-shared-vm
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
test-shared-vm RUNNING 10.207.217.54 (enp5s0) fd42:...:fe9e:b52a (enp5s0) VIRTUAL-MACHINE 0 oc-node-01
|
|
```
|
|
|
|
### 8.4 Live migrate the VM
|
|
|
|
```bash
|
|
time incus move oc-node-01:test-shared-vm --target oc-node-02
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
Transferring instance: Live migration: 1.05GB remaining (141.85MB/s)
|
|
Transferring instance: Live migration: 879.56MB remaining (141.86MB/s)
|
|
Transferring instance: Live migration: 457.06MB remaining (142.02MB/s)
|
|
Transferring instance: Live migration: 0B remaining (109.96MB/s)
|
|
|
|
real 0m6.011s
|
|
```
|
|
|
|
**6.0 seconds** — only RAM was transferred (~1 GiB at ~140 MB/s). No disk
|
|
data was copied. The progress shows RAM transfer only.
|
|
|
|
```bash
|
|
sleep 4 # Wait for VM agent to reconnect
|
|
incus list oc-node-01: --format compact | grep test-shared-vm
|
|
```
|
|
|
|
```
|
|
test-shared-vm RUNNING 10.207.217.54 (enp5s0) ... VIRTUAL-MACHINE 0 oc-node-02
|
|
```
|
|
|
|
### 8.5 Continue migrating across all nodes
|
|
|
|
```bash
|
|
# node-02 → node-03
|
|
time incus move oc-node-01:test-shared-vm --target oc-node-03
|
|
```
|
|
|
|
```
|
|
real 0m6.083s
|
|
```
|
|
|
|
```bash
|
|
# node-03 → node-01
|
|
sleep 4
|
|
time incus move oc-node-01:test-shared-vm --target oc-node-01
|
|
```
|
|
|
|
```
|
|
real 0m6.132s
|
|
```
|
|
|
|
All live migrations consistent at **~6 seconds** (RAM transfer only).
|
|
|
|
**Known issue: first live migration after stop/start may fail.** If a VM was
|
|
stopped and started (non-live migration), the first live migration attempt
|
|
may fail with `exit status 1` (QEMU on the destination cannot start). This
|
|
appears to be a transient sanlock lease issue. **Workaround:** retry the
|
|
migration — subsequent attempts succeed. Non-live migration (stop/move/start)
|
|
always works reliably.
|
|
|
|
### 8.6 Non-live migration comparison
|
|
|
|
For completeness, test stop/move/start with the VM:
|
|
|
|
```bash
|
|
incus stop oc-node-01:test-shared-vm
|
|
time incus move oc-node-01:test-shared-vm --target oc-node-03
|
|
incus start oc-node-01:test-shared-vm
|
|
```
|
|
|
|
Actual timing:
|
|
|
|
```
|
|
real 0m1.756s
|
|
```
|
|
|
|
**1.8 seconds** — mostly LVM metadata operations. No disk data copied.
|
|
Compare this to local ZFS where the full root disk must be transferred.
|
|
|
|
### 8.7 Cleanup test instances
|
|
|
|
```bash
|
|
incus delete oc-node-01:test-shared-vm --force
|
|
incus delete oc-node-01:test-shared-ct --force
|
|
```
|
|
|
|
---
|
|
|
|
## Section 9: Performance Comparison
|
|
|
|
### 9.1 Migration time comparison (tested)
|
|
|
|
The primary benefit of shared storage is migration speed. Tested results:
|
|
|
|
| Migration type | Local ZFS | Shared lvmcluster | Improvement |
|
|
|---------------|-----------|-------------------|-------------|
|
|
| **VM live migration** | 7.0s | **6.0-6.4s** | ~14% faster (RAM only vs RAM+disk) |
|
|
| **VM non-live (stop/move/start)** | 2.1s | **1.8s** | ~15% faster (metadata only) |
|
|
| **Container non-live** | N/A | **0.12-0.15s** | Near-instant |
|
|
|
|
**Key insight:** with a small 4 GiB VM root disk, the improvement is modest
|
|
because local ZFS with thin provisioning only transfers actual data (not the
|
|
full allocated size). The real win comes with **larger VMs**: a 50 GiB root
|
|
disk on local ZFS must transfer all used data (~minutes); on shared storage
|
|
it's still 6 seconds (RAM only).
|
|
|
|
Tested migration details:
|
|
|
|
```
|
|
Shared pool VM live migration (1 GiB RAM):
|
|
node-01 → node-02: 6.011s (141 MB/s, ~1 GiB RAM transferred)
|
|
node-02 → node-03: 6.083s (141 MB/s)
|
|
node-03 → node-01: 6.132s (141 MB/s)
|
|
|
|
Local ZFS VM live migration (1 GiB RAM, 4 GiB disk):
|
|
node-01 → node-02: 7.027s (141 MB/s, RAM + disk data)
|
|
|
|
Shared pool container (stop/move/start):
|
|
node-01 → node-02: 0.121s (metadata only)
|
|
node-02 → node-03: 0.152s (metadata only)
|
|
|
|
Shared pool VM (stop/move/start):
|
|
node-03 → node-01: 1.756s (LVM metadata update)
|
|
|
|
Local ZFS VM (stop/move/start):
|
|
node-02 → node-03: 2.078s (data copy)
|
|
```
|
|
|
|
### 9.2 I/O benchmarks (optional)
|
|
|
|
To compare I/O performance between the pools, launch VMs and run `fio`:
|
|
|
|
```bash
|
|
# Launch VMs on each pool
|
|
incus launch images:debian/12 oc-node-01:bench-local --vm \
|
|
--target oc-node-01 -c limits.cpu=0-1
|
|
|
|
incus launch images:debian/12 oc-node-01:bench-shared --vm \
|
|
--profile shared-pool --target oc-node-01 \
|
|
-c limits.cpu=0-1 -d root,size=4GiB
|
|
|
|
# Wait for boot, install fio
|
|
sleep 30
|
|
for vm in bench-local bench-shared; do
|
|
incus exec oc-node-01:${vm} -- apt-get update -qq
|
|
incus exec oc-node-01:${vm} -- apt-get install -y -qq fio
|
|
done
|
|
|
|
# Sequential write (1M blocks, 1 GiB)
|
|
for vm in bench-local bench-shared; do
|
|
echo "=== $vm ==="
|
|
incus exec oc-node-01:${vm} -- fio --name=seqwrite \
|
|
--ioengine=libaio --direct=1 --bs=1M --size=1G \
|
|
--numjobs=1 --rw=write --group_reporting
|
|
done
|
|
|
|
# Random 4K read (30 seconds)
|
|
for vm in bench-local bench-shared; do
|
|
echo "=== $vm ==="
|
|
incus exec oc-node-01:${vm} -- fio --name=rand4k \
|
|
--ioengine=libaio --direct=1 --bs=4k --size=256M \
|
|
--numjobs=4 --rw=randread --group_reporting \
|
|
--runtime=30 --time_based
|
|
done
|
|
|
|
# Cleanup
|
|
incus delete oc-node-01:bench-local --force
|
|
incus delete oc-node-01:bench-shared --force
|
|
```
|
|
|
|
### 9.3 Expected I/O performance (1GbE)
|
|
|
|
| Test | Local ZFS | Shared lvmcluster (1GbE) | Notes |
|
|
|------|-----------|--------------------------|-------|
|
|
| Sequential write | ~500-800 MB/s | ~100-110 MB/s | Network-bound on iSCSI |
|
|
| Sequential read | ~500-800 MB/s | ~100-110 MB/s | Network-bound on iSCSI |
|
|
| Random 4K read IOPS | ~10,000-50,000 | ~5,000-15,000 | Latency-sensitive |
|
|
| VM boot time | ~5 seconds | ~8-12 seconds | Acceptable for lab |
|
|
|
|
Local ZFS will always win on raw I/O throughput (local disk vs network).
|
|
The trade-off is clear: **use local ZFS for I/O-heavy workloads, shared
|
|
lvmcluster for HA workloads needing fast migration.**
|
|
|
|
---
|
|
|
|
## Section 10: Hybrid Architecture in Practice
|
|
|
|
Use both pools together. Keep the local ZFS pool as the default for general
|
|
workloads. Use the shared pool only for instances that need HA migration.
|
|
|
|
### 10.1 Profiles
|
|
|
|
Two profiles exist — one for each pool:
|
|
|
|
```bash
|
|
incus profile show oc-node-01:default
|
|
```
|
|
|
|
```yaml
|
|
devices:
|
|
eth0:
|
|
name: eth0
|
|
network: incusbr0
|
|
type: nic
|
|
root:
|
|
path: /
|
|
pool: local # ← local ZFS
|
|
type: disk
|
|
```
|
|
|
|
```bash
|
|
incus profile show oc-node-01:shared-pool
|
|
```
|
|
|
|
```yaml
|
|
devices:
|
|
eth0:
|
|
name: eth0
|
|
network: incusbr0
|
|
type: nic
|
|
root:
|
|
path: /
|
|
pool: shared # ← shared lvmcluster
|
|
type: disk
|
|
```
|
|
|
|
### 10.2 Launching with explicit pool choice
|
|
|
|
```bash
|
|
# Regular container (local ZFS, default profile)
|
|
incus launch images:debian/12 oc-node-01:app-web --target oc-node-01
|
|
|
|
# HA VM (shared lvmcluster, explicit profile + migration settings)
|
|
incus launch images:debian/12 oc-node-01:app-db --vm \
|
|
--profile shared-pool \
|
|
--target oc-node-01 \
|
|
-c limits.cpu=0-1 \
|
|
-c migration.stateful=true \
|
|
-d root,size=4GiB \
|
|
-d root,size.state=2GiB
|
|
```
|
|
|
|
**Remember:** shared pool VMs need explicit `size` because of thick
|
|
provisioning (default 10 GiB may be too large for a small test LUN), and
|
|
`size.state` for stateful migration.
|
|
|
|
### 10.3 Moving instances between pools
|
|
|
|
To move an instance from local to shared (or vice versa), use `incus move`
|
|
with `--storage`:
|
|
|
|
```bash
|
|
incus stop oc-node-01:app-db
|
|
incus move oc-node-01:app-db oc-node-01:app-db --storage shared
|
|
incus start oc-node-01:app-db
|
|
```
|
|
|
|
**Note:** This copies all data from the source pool to the destination pool.
|
|
For large VMs, this can take time depending on disk size and pool speeds.
|
|
|
|
---
|
|
|
|
## Section 11: Future — Multi-Host and Upgrades
|
|
|
|
### Adding a second Proxmox host
|
|
|
|
With shared iSCSI storage, adding a second Proxmox host enables true cross-
|
|
host live migration:
|
|
|
|
```
|
|
┌─ Proxmox Host A ────────┐ ┌─ Proxmox Host B ────────┐
|
|
│ oc-node-01, oc-node-02 │ │ oc-node-03, oc-node-04 │
|
|
│ iSCSI initiators │ │ iSCSI initiators │
|
|
└──────────┬───────────────┘ └──────────┬───────────────┘
|
|
│ │
|
|
└────────── iSCSI ───────────────┘
|
|
│
|
|
┌─────────┴──────────┐
|
|
│ QNAP NAS │
|
|
│ iSCSI target │
|
|
│ Shared LUN │
|
|
└────────────────────┘
|
|
```
|
|
|
|
Requirements:
|
|
- QNAP NAS (or other external iSCSI target) accessible from both hosts
|
|
- Same iSCSI IQN and LUN on all nodes
|
|
- Nodes on the same Incus cluster (already the case)
|
|
|
|
### 2.5GbE upgrade path
|
|
|
|
When the 2.5GbE switch arrives:
|
|
- iSCSI throughput increases from ~110 MB/s to ~275 MB/s
|
|
- Random IOPS may improve slightly (lower network latency)
|
|
- Migration time (already sub-second) is unchanged
|
|
- Consider a dedicated storage VLAN for iSCSI traffic separation
|
|
|
|
### Dedicated storage VLAN
|
|
|
|
For production, isolate iSCSI traffic on its own VLAN:
|
|
|
|
```
|
|
VLAN 69 (192.168.102.x) — management + OVN tunnels
|
|
VLAN 70 (10.69.0.x) — iSCSI storage traffic (dedicated)
|
|
```
|
|
|
|
This prevents storage I/O from competing with management and OVN traffic.
|
|
Requires a second NIC or VLAN trunking on each node.
|
|
|
|
### Ceph (reference only)
|
|
|
|
IncusOS also includes a Ceph client service. Ceph provides distributed
|
|
storage with replication and is the standard for large-scale deployments.
|
|
However, it requires:
|
|
|
|
- Minimum 3 OSD nodes with dedicated disks
|
|
- Significant RAM and CPU overhead
|
|
- More complexity to operate
|
|
|
|
For a home lab with a NAS already available, iSCSI + lvmcluster is simpler
|
|
and more resource-efficient. Ceph is better suited for multi-rack deployments
|
|
where you need storage to survive node failures without a single NAS.
|
|
|
|
---
|
|
|
|
## Section 12: Troubleshooting
|
|
|
|
### Wrong API field name (`iqn` vs `target`)
|
|
|
|
**Symptom:** iSCSI service shows `enabled: true` but no `initiator_name` in
|
|
the state, and no block device appears.
|
|
|
|
**Cause:** Using `"iqn"` instead of `"target"` in the PUT request. The API
|
|
silently accepts the wrong field and stores an empty target string.
|
|
|
|
**How to check:**
|
|
|
|
```bash
|
|
incus query oc-node-01:/os/1.0/services/iscsi
|
|
```
|
|
|
|
If the target field is empty (`"target": ""`), you used the wrong field name.
|
|
|
|
**Fix:** Re-submit the PUT request with `"target"` (not `"iqn"`):
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"enabled": true,
|
|
"targets": [{
|
|
"address": "10.207.217.19",
|
|
"port": 3260,
|
|
"target": "iqn.2026-02.lab.incus:storage.shared"
|
|
}]
|
|
},
|
|
"state": {}
|
|
}
|
|
```
|
|
|
|
### Proxy devices don't work for iSCSI
|
|
|
|
**Symptom:** iSCSI service returns "No records found" (exit status 21) when
|
|
using an Incus proxy device to expose the target's port 3260.
|
|
|
|
**Cause:** IncusOS runs `iscsiadm -m discovery -t sendtargets` before login.
|
|
The iSCSI target (tgt/targetcli) responds with its own IP in the SendTargets
|
|
response. Through a proxy device, the target returns the container's internal
|
|
IP (e.g., `10.207.217.19`) — but the node connected to the proxy's host IP
|
|
(e.g., `192.168.102.140`). The portal mismatch causes the login to fail.
|
|
|
|
**Fix:** Use direct network connectivity instead of proxy devices. The lab
|
|
target container uses dual network paths (bridge + macvlan) as described in
|
|
Section 2.
|
|
|
|
### LVM system_id must be 1-2000
|
|
|
|
**Symptom:** `incus storage create` fails during pool finalization with
|
|
`"Invalid host_id 0, use 1-2000"`.
|
|
|
|
**Cause:** LVM service enabled without specifying `system_id`, or `system_id`
|
|
set to 0. Sanlock requires a unique host ID in the range 1-2000.
|
|
|
|
**Fix:** Set unique `system_id` values on each node:
|
|
|
|
```bash
|
|
incus query oc-node-01:/os/1.0/services/lvm --request PUT --data '{
|
|
"config": { "enabled": true, "system_id": 1 },
|
|
"state": {}
|
|
}'
|
|
# system_id: 2 for node-02, 3 for node-03
|
|
```
|
|
|
|
### First live migration fails after stop/start
|
|
|
|
**Symptom:** Live migration fails with `exit status 1` (QEMU cannot start on
|
|
the destination). The error includes a long QEMU command line ending with
|
|
`-incoming defer ... exit status 1`.
|
|
|
|
**Cause:** Appears to be a transient sanlock lease timing issue. After a VM
|
|
is stopped and restarted (or after a non-live migration), the first live
|
|
migration attempt may fail because the destination node's sanlock lease for
|
|
the LV hasn't fully registered.
|
|
|
|
**Fix:** Retry the live migration — subsequent attempts succeed. Non-live
|
|
migration (stop/move/start) always works as a fallback.
|
|
|
|
### Insufficient space on shared pool
|
|
|
|
**Symptom:** Instance launch fails with space-related errors.
|
|
|
|
**Cause:** lvmcluster uses **thick provisioning**. A 10 GiB VM root disk
|
|
immediately allocates 10 GiB on the LUN. With a 20 GiB test LUN, you can
|
|
fit ~3 VMs with 4 GiB roots (accounting for LVM/sanlock metadata overhead
|
|
of ~256 MiB).
|
|
|
|
**Fix:** Use explicit small sizes for test VMs:
|
|
|
|
```bash
|
|
incus launch ... -d root,size=4GiB -d root,size.state=2GiB
|
|
```
|
|
|
|
Or use a larger LUN (200+ GiB for production).
|
|
|
|
### lvmcluster pool creation errors
|
|
|
|
| Error | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| `device not found` | Wrong device path | Verify via storage API (Section 4.4) |
|
|
| `VG already exists` | Previous attempt left VG | Use existing VG name in `source=` |
|
|
| `lockd not running` | LVM service not enabled | Enable with `system_id` (Section 5) |
|
|
| `Invalid host_id 0` | Missing `system_id` | Set unique `system_id` 1-2000 per node |
|
|
|
|
### Recovery from target container/VM/NAS failure
|
|
|
|
If the iSCSI target goes down:
|
|
|
|
1. **All instances on `shared` pool hang** — I/O operations block until the
|
|
target returns or the iSCSI timeout expires
|
|
2. **Instances on `local` pool are unaffected** — this is why the hybrid
|
|
architecture matters
|
|
3. **When the target recovers**: iSCSI initiators reconnect automatically
|
|
4. **If the target is permanently lost**: shared pool data is gone. Delete
|
|
the pool and recreate from a new target
|
|
|
|
### Target container maintenance
|
|
|
|
```bash
|
|
# 1. List instances on the shared pool
|
|
incus list oc-node-01: --format json | python3 -c "
|
|
import sys, json
|
|
for inst in json.load(sys.stdin):
|
|
for dev in inst.get('expanded_devices', {}).values():
|
|
if dev.get('pool') == 'shared':
|
|
print(f\" {inst['name']} ({inst['status']}) on {inst['location']}\")
|
|
break
|
|
"
|
|
|
|
# 2. Stop all shared-pool instances
|
|
# 3. Stop the target container/VM/NAS
|
|
# 4. Perform maintenance
|
|
# 5. Start the target
|
|
# 6. Wait ~30s for iSCSI reconnection
|
|
# 7. Start instances
|
|
```
|
|
|
|
---
|
|
|
|
## Section 13: 1GbE Performance Assessment
|
|
|
|
### Theoretical limits
|
|
|
|
- 1GbE raw: 125 MB/s
|
|
- TCP/iSCSI overhead: ~10-15%
|
|
- Practical maximum: **106-112 MB/s** sequential
|
|
|
|
### What we actually measured
|
|
|
|
| Operation | Local ZFS | Shared lvmcluster | Observation |
|
|
|-----------|-----------|-------------------|-------------|
|
|
| VM live migration (1 GiB RAM) | 7.0s | **6.0-6.4s** | RAM transfer at ~141 MB/s |
|
|
| VM non-live migration | 2.1s | **1.8s** | Metadata only on shared |
|
|
| Container non-live migration | N/A | **0.12-0.15s** | Near-instant |
|
|
| VM boot time | ~20s | ~20s | No noticeable difference |
|
|
| Migration transfer rate | ~141 MB/s | ~141 MB/s | Same network, same rate |
|
|
|
|
**The migration speed difference grows with disk size.** Our test used a small
|
|
4 GiB VM root. With a 50 GiB root disk on local ZFS, migration must transfer
|
|
all used data — potentially minutes. On shared storage, it's always 6 seconds
|
|
(only RAM).
|
|
|
|
### Expected I/O performance (not yet benchmarked)
|
|
|
|
| Scenario | Throughput | Notes |
|
|
|----------|-----------|-------|
|
|
| Single node sequential I/O | ~100-110 MB/s | Near wire speed |
|
|
| 3 nodes concurrent I/O | ~33-37 MB/s per node | Shared bandwidth |
|
|
| Random 4K IOPS | 5,000-15,000 | Target-dependent, not network-bound |
|
|
| I/O latency | 0.5-1.5ms | vs ~0.1ms for local ZFS |
|
|
|
|
### Verdict
|
|
|
|
**1GbE is viable for a home lab.** The primary goal — eliminating data copy
|
|
during migration — is fully achieved regardless of network speed. Day-to-day
|
|
I/O is slower than local ZFS but acceptable for lab workloads. The hybrid
|
|
architecture (local ZFS for general use, shared for HA) minimizes the impact.
|
|
|
|
2.5GbE will improve throughput ~2.5x when the switch arrives but is not a
|
|
prerequisite for this setup.
|
|
|
|
### sanlock overhead
|
|
|
|
Minimal in testing. sanlock uses ~100 IOPS for lease renewal (every 20
|
|
seconds) and only holds locks during LVM metadata operations (create, delete,
|
|
resize LV). Normal I/O to the logical volumes bypasses sanlock entirely. The
|
|
256 MiB overhead on the 20 GiB LUN is all metadata + lease storage.
|