Add shared storage guide: iSCSI + lvmcluster with tested output

Validated end-to-end on OC-managed 3-node cluster: iSCSI target
container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool,
container migration (0.12s), VM live migration (~6s, RAM only).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-23 14:58:11 +01:00
parent 896731a7ed
commit 4854f15a16
3 changed files with 1724 additions and 0 deletions

View File

@ -29,6 +29,7 @@ incu-contrib/
├── clustering-guide.md # Detailed Incus clustering walkthrough
├── operations-center-guide.md # Operations Center provisioning & management
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested)
├── migration-guide.md # Migration paths into Incus from other hypervisors
└── utm-support.md # UTM support design document (future)
```
@ -893,6 +894,44 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
"Invalid target address".
- See `notes/networking-guide.md` for full tutorial with test results.
### Shared storage (iSCSI + lvmcluster)
- **iSCSI + lvmcluster** is the IncusOS-native path to shared storage.
All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS
and enabled via the REST API — no packages to install.
- **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN
field is `"target"` (NOT `"iqn"` — using the wrong field silently fails).
The config lives under `config.targets[]`, and `state` returns the
auto-generated `initiator_name`.
- **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique
`system_id` (1-2000) per node for sanlock host identification. Using 0
or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool
creation.
- **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on
custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately.
~256 MiB overhead for LVM metadata + sanlock lease area.
- **Pool creation**: two-step cluster pattern (`--target` per member, then
finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`,
and `vgchange --lock-start` automatically.
- **Migration performance** (tested on 1GbE, 2026-02-23):
- Container stop/move/start: **0.12-0.15s** (metadata only)
- VM non-live (stop/move/start): **1.8s** (LVM metadata update)
- VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer)
- Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data)
- **Proxy devices don't work for iSCSI**: SendTargets discovery returns
container IP in TargetAddress, causing portal mismatch. Use direct network
paths (bridge for same-node, macvlan for cross-node).
- **Lab target container**: Debian container with `tgt` (userspace iSCSI
target) on the cluster — no external hardware needed. Uses dual network:
bridge IP for same-node access, macvlan on `mgmt` for cross-node access.
- **First live migration after stop/start may fail**: QEMU on destination
fails to start (transient sanlock lease issue). Retry succeeds. Non-live
migration always works as fallback.
- **Hybrid architecture recommended**: local ZFS (`local` pool) for general
workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant
migration.
- See `notes/shared-storage-guide.md` for the full tested walkthrough.
### Migration into Incus
- **`incus-migrate`**: official tool for importing disk images, running

View File

@ -0,0 +1,72 @@
# lab-iscsi-target.yaml - iSCSI target reference for shared storage
#
# Reference configuration for an iSCSI target serving the IncusOS cluster's
# lvmcluster shared storage pool. All 3 IncusOS cluster nodes connect as
# iSCSI initiators via their built-in iSCSI service.
#
# Two deployment options (see notes/shared-storage-guide.md):
#
# Option A (tested, recommended for lab):
# Deploy as an Incus container with `tgt` on the existing cluster.
# No extra VM needed. See shared-storage-guide.md Section 2 for the
# full tested walkthrough.
#
# Option B (this file):
# Deploy as a standalone Proxmox VM with targetcli-fb. Better for
# production-like setups or when you want the target independent of
# the Incus cluster. Manual Proxmox deployment required.
#
# Architecture:
# iscsi-target (410) - Debian 12 + targetcli-fb, 200G LUN
# oc-node-01 (400) - iSCSI initiator + lvmcluster
# oc-node-02 (401) - iSCSI initiator + lvmcluster
# oc-node-03 (402) - iSCSI initiator + lvmcluster
#
# Option B deployment steps:
# 1. Download Debian 12 cloud image or install from ISO
# 2. Create VM: VMID 410, 1 core, 1 GiB RAM, 200 GiB disk
# 3. Network: vmbr0, VLAN tag 69
# 4. Static IP: 192.168.102.150/22, gateway 192.168.100.1
# 5. After boot: run post-install commands below
# 6. Enable iSCSI on IncusOS nodes (shared-storage-guide.md Section 4)
#
# RAM budget: 1 GiB (adds minimal overhead to the 28 GiB cluster)
#
# See notes/shared-storage-guide.md for the full shared storage guide.
# Reference configuration (for documentation — not used by incusos-proxmox)
target_vm:
name: iscsi-target
vmid: 410
os: debian/12
cores: 1
memory: 1024
disk: 200
ip: 192.168.102.150/22
gateway: 192.168.100.1
dns: 192.168.100.1
vlan: 69
iscsi:
iqn: iqn.2026-02.lab.incus:storage.shared
lun_path: /srv/iscsi/shared-lun.img
lun_size: 200G
port: 3260
auth: none # Lab environment — no CHAP
# Post-install commands (run on the target VM after Debian is installed):
#
# apt-get update && apt-get install -y targetcli-fb
# systemctl enable rtslib-fb-targetctl
# mkdir -p /srv/iscsi
# truncate -s 200G /srv/iscsi/shared-lun.img
#
# targetcli <<'EOF'
# /backstores/fileio create shared-lun /srv/iscsi/shared-lun.img 200G
# /iscsi create iqn.2026-02.lab.incus:storage.shared
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1/luns create /backstores/fileio/shared-lun
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute authentication=0
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute demo_mode_write_protect=0
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute generate_node_acls=1
# saveconfig
# EOF

File diff suppressed because it is too large Load Diff