Add shared storage guide: iSCSI + lvmcluster with tested output
Validated end-to-end on OC-managed 3-node cluster: iSCSI target container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool, container migration (0.12s), VM live migration (~6s, RAM only). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
896731a7ed
commit
4854f15a16
39
CLAUDE.md
39
CLAUDE.md
|
|
@ -29,6 +29,7 @@ incu-contrib/
|
||||||
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
├── clustering-guide.md # Detailed Incus clustering walkthrough
|
||||||
├── operations-center-guide.md # Operations Center provisioning & management
|
├── operations-center-guide.md # Operations Center provisioning & management
|
||||||
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
├── networking-guide.md # OVN overlay networking tutorial (bridge + OVN + LAN)
|
||||||
|
├── shared-storage-guide.md # iSCSI + lvmcluster shared storage (tested)
|
||||||
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
├── migration-guide.md # Migration paths into Incus from other hypervisors
|
||||||
└── utm-support.md # UTM support design document (future)
|
└── utm-support.md # UTM support design document (future)
|
||||||
```
|
```
|
||||||
|
|
@ -893,6 +894,44 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
|
||||||
"Invalid target address".
|
"Invalid target address".
|
||||||
- See `notes/networking-guide.md` for full tutorial with test results.
|
- See `notes/networking-guide.md` for full tutorial with test results.
|
||||||
|
|
||||||
|
### Shared storage (iSCSI + lvmcluster)
|
||||||
|
|
||||||
|
- **iSCSI + lvmcluster** is the IncusOS-native path to shared storage.
|
||||||
|
All services (iSCSI initiator, lvmlockd, sanlock) are built into IncusOS
|
||||||
|
and enabled via the REST API — no packages to install.
|
||||||
|
- **IncusOS iSCSI service API**: `/os/1.0/services/iscsi`. The target IQN
|
||||||
|
field is `"target"` (NOT `"iqn"` — using the wrong field silently fails).
|
||||||
|
The config lives under `config.targets[]`, and `state` returns the
|
||||||
|
auto-generated `initiator_name`.
|
||||||
|
- **IncusOS LVM service API**: `/os/1.0/services/lvm`. Requires unique
|
||||||
|
`system_id` (1-2000) per node for sanlock host identification. Using 0
|
||||||
|
or omitting it causes `"Invalid host_id 0, use 1-2000"` during pool
|
||||||
|
creation.
|
||||||
|
- **lvmcluster driver**: uses thick provisioning (no thin, no snapshots on
|
||||||
|
custom volumes). A 10 GiB VM root = 10 GiB on the LUN immediately.
|
||||||
|
~256 MiB overhead for LVM metadata + sanlock lease area.
|
||||||
|
- **Pool creation**: two-step cluster pattern (`--target` per member, then
|
||||||
|
finalize without `--target`). Incus handles `pvcreate`, `vgcreate --shared`,
|
||||||
|
and `vgchange --lock-start` automatically.
|
||||||
|
- **Migration performance** (tested on 1GbE, 2026-02-23):
|
||||||
|
- Container stop/move/start: **0.12-0.15s** (metadata only)
|
||||||
|
- VM non-live (stop/move/start): **1.8s** (LVM metadata update)
|
||||||
|
- VM live migration: **~6s** (1 GiB RAM at ~141 MB/s, no disk transfer)
|
||||||
|
- Local ZFS comparison: live ~7s, non-live ~2s (transfers disk data)
|
||||||
|
- **Proxy devices don't work for iSCSI**: SendTargets discovery returns
|
||||||
|
container IP in TargetAddress, causing portal mismatch. Use direct network
|
||||||
|
paths (bridge for same-node, macvlan for cross-node).
|
||||||
|
- **Lab target container**: Debian container with `tgt` (userspace iSCSI
|
||||||
|
target) on the cluster — no external hardware needed. Uses dual network:
|
||||||
|
bridge IP for same-node access, macvlan on `mgmt` for cross-node access.
|
||||||
|
- **First live migration after stop/start may fail**: QEMU on destination
|
||||||
|
fails to start (transient sanlock lease issue). Retry succeeds. Non-live
|
||||||
|
migration always works as fallback.
|
||||||
|
- **Hybrid architecture recommended**: local ZFS (`local` pool) for general
|
||||||
|
workloads + shared lvmcluster (`shared` pool) for HA VMs needing instant
|
||||||
|
migration.
|
||||||
|
- See `notes/shared-storage-guide.md` for the full tested walkthrough.
|
||||||
|
|
||||||
### Migration into Incus
|
### Migration into Incus
|
||||||
|
|
||||||
- **`incus-migrate`**: official tool for importing disk images, running
|
- **`incus-migrate`**: official tool for importing disk images, running
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,72 @@
|
||||||
|
# lab-iscsi-target.yaml - iSCSI target reference for shared storage
|
||||||
|
#
|
||||||
|
# Reference configuration for an iSCSI target serving the IncusOS cluster's
|
||||||
|
# lvmcluster shared storage pool. All 3 IncusOS cluster nodes connect as
|
||||||
|
# iSCSI initiators via their built-in iSCSI service.
|
||||||
|
#
|
||||||
|
# Two deployment options (see notes/shared-storage-guide.md):
|
||||||
|
#
|
||||||
|
# Option A (tested, recommended for lab):
|
||||||
|
# Deploy as an Incus container with `tgt` on the existing cluster.
|
||||||
|
# No extra VM needed. See shared-storage-guide.md Section 2 for the
|
||||||
|
# full tested walkthrough.
|
||||||
|
#
|
||||||
|
# Option B (this file):
|
||||||
|
# Deploy as a standalone Proxmox VM with targetcli-fb. Better for
|
||||||
|
# production-like setups or when you want the target independent of
|
||||||
|
# the Incus cluster. Manual Proxmox deployment required.
|
||||||
|
#
|
||||||
|
# Architecture:
|
||||||
|
# iscsi-target (410) - Debian 12 + targetcli-fb, 200G LUN
|
||||||
|
# oc-node-01 (400) - iSCSI initiator + lvmcluster
|
||||||
|
# oc-node-02 (401) - iSCSI initiator + lvmcluster
|
||||||
|
# oc-node-03 (402) - iSCSI initiator + lvmcluster
|
||||||
|
#
|
||||||
|
# Option B deployment steps:
|
||||||
|
# 1. Download Debian 12 cloud image or install from ISO
|
||||||
|
# 2. Create VM: VMID 410, 1 core, 1 GiB RAM, 200 GiB disk
|
||||||
|
# 3. Network: vmbr0, VLAN tag 69
|
||||||
|
# 4. Static IP: 192.168.102.150/22, gateway 192.168.100.1
|
||||||
|
# 5. After boot: run post-install commands below
|
||||||
|
# 6. Enable iSCSI on IncusOS nodes (shared-storage-guide.md Section 4)
|
||||||
|
#
|
||||||
|
# RAM budget: 1 GiB (adds minimal overhead to the 28 GiB cluster)
|
||||||
|
#
|
||||||
|
# See notes/shared-storage-guide.md for the full shared storage guide.
|
||||||
|
|
||||||
|
# Reference configuration (for documentation — not used by incusos-proxmox)
|
||||||
|
target_vm:
|
||||||
|
name: iscsi-target
|
||||||
|
vmid: 410
|
||||||
|
os: debian/12
|
||||||
|
cores: 1
|
||||||
|
memory: 1024
|
||||||
|
disk: 200
|
||||||
|
ip: 192.168.102.150/22
|
||||||
|
gateway: 192.168.100.1
|
||||||
|
dns: 192.168.100.1
|
||||||
|
vlan: 69
|
||||||
|
|
||||||
|
iscsi:
|
||||||
|
iqn: iqn.2026-02.lab.incus:storage.shared
|
||||||
|
lun_path: /srv/iscsi/shared-lun.img
|
||||||
|
lun_size: 200G
|
||||||
|
port: 3260
|
||||||
|
auth: none # Lab environment — no CHAP
|
||||||
|
|
||||||
|
# Post-install commands (run on the target VM after Debian is installed):
|
||||||
|
#
|
||||||
|
# apt-get update && apt-get install -y targetcli-fb
|
||||||
|
# systemctl enable rtslib-fb-targetctl
|
||||||
|
# mkdir -p /srv/iscsi
|
||||||
|
# truncate -s 200G /srv/iscsi/shared-lun.img
|
||||||
|
#
|
||||||
|
# targetcli <<'EOF'
|
||||||
|
# /backstores/fileio create shared-lun /srv/iscsi/shared-lun.img 200G
|
||||||
|
# /iscsi create iqn.2026-02.lab.incus:storage.shared
|
||||||
|
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1/luns create /backstores/fileio/shared-lun
|
||||||
|
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute authentication=0
|
||||||
|
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute demo_mode_write_protect=0
|
||||||
|
# /iscsi/iqn.2026-02.lab.incus:storage.shared/tpg1 set attribute generate_node_acls=1
|
||||||
|
# saveconfig
|
||||||
|
# EOF
|
||||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue