maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/utm-support.md

7.4 KiB
Raw Blame History

UTM Support Design Document

Design notes for a future incusos-utm script (or --backend utm flag on incusos-proxmox) that would deploy IncusOS VMs on macOS using UTM as the hypervisor.

Status: design only. Implementation requires macOS with UTM installed.

Why UTM

UTM is a macOS-native virtualization app built on Apple's Hypervisor.framework (for arm64 VMs) and QEMU (for x86_64 VMs). It provides a GUI and a CLI (utmctl) for VM management.

For macOS users who want to run an IncusOS lab locally (instead of on a remote Proxmox host), UTM is the most accessible option. It supports:

  • UEFI boot (required by IncusOS)
  • TPM 2.0 emulation (required for disk encryption)
  • VirtIO devices (required by IncusOS)
  • Nested virtualization (via Apple Hypervisor.framework)
  • ISO attachment for boot media

Architecture: UTM vs Proxmox

Feature Proxmox UTM
VM creation API (POST /qemu) or SSH (qm create) utmctl CLI or AppleScript
Disk management API-managed, VMID-based naming File-based (.utm bundles)
ISO attach API (ide2, ide3 config keys) GUI or utmctl, possibly AppleScript
Boot monitoring API (blockstat.scsi0.wr_bytes polling) No equivalent -- use timeout + port polling
IP detection ARP after MAC lookup from API ARP after MAC from utmctl or VM config
UEFI/TPM bios=ovmf, tpmstate0 config keys GUI settings per VM
Resource pool Proxmox resource pools No equivalent (single-user)
Remote access SSH or REST API over network Local only (no remote API)
Automation Full API + SSH, headless utmctl CLI, AppleScript, limited headless

UTM CLI: utmctl

UTM provides utmctl for basic VM management:

# List VMs
utmctl list

# Start/stop
utmctl start <vm-name>
utmctl stop <vm-name>

# Get VM info (includes IP if guest agent is available)
utmctl status <vm-name>

# Clone
utmctl clone <vm-name> --name <new-name>

# Delete
utmctl delete <vm-name>

# Attach USB device
utmctl usb connect <vm-name> <device>

Limitations of utmctl

  • No VM creation: VMs must be created via GUI or AppleScript
  • No disk management: cannot attach/detach ISOs programmatically
  • No hardware config: cannot set CPU count, memory, TPM, etc.
  • No boot order control: must use GUI

These limitations mean a fully automated deployment (like incusos-proxmox) would require AppleScript for VM creation and configuration, with utmctl for start/stop/status operations.

Proposed Implementation

Option A: Separate script (incusos-utm)

A dedicated script that handles UTM-specific VM creation and management. Reuses seed generation from incusos-seed.

incusos-utm [OPTIONS] CONFIG_FILE

  --dry-run         Preview actions
  --status          Check VM status
  --cleanup         Delete VMs
  --lab-up          Start stopped VMs
  --lab-down        Stop running VMs

Option B: Backend flag on incusos-proxmox

Add --backend utm to the existing script. This requires abstracting the Proxmox-specific functions behind a backend interface.

incusos-proxmox --backend utm CONFIG_FILE

Recommendation: Option A (separate script) is simpler and avoids complicating the well-tested Proxmox workflow. The scripts can share config format and seed generation.

VM Creation via AppleScript

Since utmctl cannot create VMs, we'd use AppleScript:

tell application "UTM"
    -- Create a new VM
    set newVM to make new virtual machine with properties {
        name: "incus-lab-01",
        backend: QEMU,
        architecture: x86_64
    }

    -- Configure hardware
    set memory of newVM to 4096
    set cpu cores of newVM to 4

    -- Add drives
    -- (AppleScript API for UTM disk management TBD)
end tell

Note: UTM's AppleScript API may not expose all hardware settings. Alternative: use UTM's .utm bundle format (plist + disk files) to create VM bundles programmatically.

Install Detection Without blockstat

Proxmox provides real-time disk I/O statistics via API, which incusos-proxmox uses to detect when IncusOS installation completes. UTM has no equivalent.

Proposed alternative: timeout + port polling

# 1. Start VM with boot ISO + SEED_DATA
utmctl start "$vm_name"

# 2. Wait for expected install duration (2-5 minutes)
sleep 120

# 3. Poll port 8443 until it responds (with timeout)
local elapsed=0
while [[ $elapsed -lt 300 ]]; do
    if nc -z "$vm_ip" 8443 2>/dev/null; then
        echo "IncusOS is up"
        break
    fi
    sleep 10
    elapsed=$((elapsed + 10))
done

# 4. If force_reboot is in the seed, the VM reboots after install.
#    On reboot, stop the VM, remove ISOs, restart from disk.

IP Detection

UTM VMs on the default shared network get IPs from macOS's built-in DHCP. Detection approaches:

  1. utmctl status: may report IP if QEMU guest agent is installed (IncusOS doesn't have guest agent)
  2. ARP lookup: same as Proxmox -- get MAC from VM config, scan ARP table
  3. DHCP lease file: check /var/db/dhcpd_leases on macOS

SEED_DATA Delivery

Two options for attaching the SEED_DATA to a UTM VM:

  1. ISO on virtual CD-ROM: generate ISO with incusos-seed --format iso, attach as secondary CD-ROM in UTM
  2. External drive: generate FAT image with incusos-seed --format fat, attach as secondary disk

ISO is preferred (matches the Proxmox workflow).

Config File Compatibility

The YAML config format should be compatible with incusos-proxmox configs (minus the proxmox: section). UTM-specific settings could go in a utm: section:

# utm: section (optional, UTM-specific overrides)
utm:
  backend: qemu          # qemu or apple (default: qemu)
  network: shared        # shared, bridged, host-only
  display: true          # show VM window (default: true for UTM)

defaults:
  cores: 4
  memory: 4096
  disk: 50

vms:
  - name: incus-lab-01
    app: incus
    apply_defaults: true

Limitations and Trade-offs

Aspect Proxmox UTM
Automation level Full (API/SSH) Partial (utmctl + AppleScript)
Headless operation Yes Limited (UTM needs to be running)
Remote deployment Yes (network API) No (local only)
Multi-node scaling Tested up to 4 VMs Limited by host RAM/CPU
Install monitoring Real-time (blockstat) Timeout-based
Resource isolation Proxmox pools None
Nested virt Intel VT-x passthrough Apple HV.framework (arm64)

When to use which

  • Proxmox: production labs, multi-node clusters, automated CI, remote management, teams
  • UTM: local development, single-developer labs, macOS-only environments, quick experiments

Implementation Roadmap

  1. Research phase (now): document UTM capabilities, utmctl API surface, AppleScript integration
  2. Seed generation: already works (incusos-seed is cross-platform after Phase 3 macOS compatibility)
  3. VM creation: implement AppleScript or .utm bundle generation
  4. Install flow: implement timeout-based install detection
  5. Status/lifecycle: implement --status, --lab-up, --lab-down using utmctl
  6. Testing: test on macOS with Apple Silicon and Intel Macs

References