incus-contrib/notes/utm-support.md

# UTM Support Design Document

Design notes for a future `incusos-utm` script (or `--backend utm` flag on
`incusos-proxmox`) that would deploy IncusOS VMs on macOS using UTM as the
hypervisor.

**Status**: design only. Implementation requires macOS with UTM installed.

---

## Why UTM

UTM is a macOS-native virtualization app built on Apple's Hypervisor.framework
(for arm64 VMs) and QEMU (for x86_64 VMs). It provides a GUI and a CLI
(`utmctl`) for VM management.

For macOS users who want to run an IncusOS lab locally (instead of on a remote
Proxmox host), UTM is the most accessible option. It supports:

- UEFI boot (required by IncusOS)
- TPM 2.0 emulation (required for disk encryption)
- VirtIO devices (required by IncusOS)
- Nested virtualization (via Apple Hypervisor.framework)
- ISO attachment for boot media

---

## Architecture: UTM vs Proxmox

| Feature | Proxmox | UTM |
|---------|---------|-----|
| VM creation | API (`POST /qemu`) or SSH (`qm create`) | `utmctl` CLI or AppleScript |
| Disk management | API-managed, VMID-based naming | File-based (`.utm` bundles) |
| ISO attach | API (`ide2`, `ide3` config keys) | GUI or utmctl, possibly AppleScript |
| Boot monitoring | API (`blockstat.scsi0.wr_bytes` polling) | No equivalent -- use timeout + port polling |
| IP detection | ARP after MAC lookup from API | ARP after MAC from `utmctl` or VM config |
| UEFI/TPM | `bios=ovmf`, `tpmstate0` config keys | GUI settings per VM |
| Resource pool | Proxmox resource pools | No equivalent (single-user) |
| Remote access | SSH or REST API over network | Local only (no remote API) |
| Automation | Full API + SSH, headless | `utmctl` CLI, AppleScript, limited headless |

---

## UTM CLI: utmctl

UTM provides `utmctl` for basic VM management:

```bash
# List VMs
utmctl list

# Start/stop
utmctl start <vm-name>
utmctl stop <vm-name>

# Get VM info (includes IP if guest agent is available)
utmctl status <vm-name>

# Clone
utmctl clone <vm-name> --name <new-name>

# Delete
utmctl delete <vm-name>

# Attach USB device
utmctl usb connect <vm-name> <device>
```

### Limitations of utmctl

- **No VM creation**: VMs must be created via GUI or AppleScript
- **No disk management**: cannot attach/detach ISOs programmatically
- **No hardware config**: cannot set CPU count, memory, TPM, etc.
- **No boot order control**: must use GUI

These limitations mean a fully automated deployment (like `incusos-proxmox`)
would require AppleScript for VM creation and configuration, with `utmctl`
for start/stop/status operations.

---

## Proposed Implementation

### Option A: Separate script (`incusos-utm`)

A dedicated script that handles UTM-specific VM creation and management.
Reuses seed generation from `incusos-seed`.

```
incusos-utm [OPTIONS] CONFIG_FILE

  --dry-run         Preview actions
  --status          Check VM status
  --cleanup         Delete VMs
  --lab-up          Start stopped VMs
  --lab-down        Stop running VMs
```

### Option B: Backend flag on `incusos-proxmox`

Add `--backend utm` to the existing script. This requires abstracting the
Proxmox-specific functions behind a backend interface.

```
incusos-proxmox --backend utm CONFIG_FILE
```

**Recommendation**: Option A (separate script) is simpler and avoids
complicating the well-tested Proxmox workflow. The scripts can share
config format and seed generation.

---

## VM Creation via AppleScript

Since `utmctl` cannot create VMs, we'd use AppleScript:

```applescript
tell application "UTM"
    -- Create a new VM
    set newVM to make new virtual machine with properties {
        name: "incus-lab-01",
        backend: QEMU,
        architecture: x86_64
    }

    -- Configure hardware
    set memory of newVM to 4096
    set cpu cores of newVM to 4

    -- Add drives
    -- (AppleScript API for UTM disk management TBD)
end tell
```

**Note**: UTM's AppleScript API may not expose all hardware settings.
Alternative: use UTM's `.utm` bundle format (plist + disk files) to
create VM bundles programmatically.

---

## Install Detection Without blockstat

Proxmox provides real-time disk I/O statistics via API, which
`incusos-proxmox` uses to detect when IncusOS installation completes.
UTM has no equivalent.

### Proposed alternative: timeout + port polling

```bash
# 1. Start VM with boot ISO + SEED_DATA
utmctl start "$vm_name"

# 2. Wait for expected install duration (2-5 minutes)
sleep 120

# 3. Poll port 8443 until it responds (with timeout)
local elapsed=0
while [[ $elapsed -lt 300 ]]; do
    if nc -z "$vm_ip" 8443 2>/dev/null; then
        echo "IncusOS is up"
        break
    fi
    sleep 10
    elapsed=$((elapsed + 10))
done

# 4. If force_reboot is in the seed, the VM reboots after install.
#    On reboot, stop the VM, remove ISOs, restart from disk.
```

### IP Detection

UTM VMs on the default shared network get IPs from macOS's built-in DHCP.
Detection approaches:

1. **utmctl status**: may report IP if QEMU guest agent is installed
   (IncusOS doesn't have guest agent)
2. **ARP lookup**: same as Proxmox -- get MAC from VM config, scan ARP table
3. **DHCP lease file**: check `/var/db/dhcpd_leases` on macOS

---

## SEED_DATA Delivery

Two options for attaching the SEED_DATA to a UTM VM:

1. **ISO on virtual CD-ROM**: generate ISO with `incusos-seed --format iso`,
   attach as secondary CD-ROM in UTM
2. **External drive**: generate FAT image with `incusos-seed --format fat`,
   attach as secondary disk

ISO is preferred (matches the Proxmox workflow).

---

## Config File Compatibility

The YAML config format should be compatible with `incusos-proxmox` configs
(minus the `proxmox:` section). UTM-specific settings could go in a `utm:`
section:

```yaml
# utm: section (optional, UTM-specific overrides)
utm:
  backend: qemu          # qemu or apple (default: qemu)
  network: shared        # shared, bridged, host-only
  display: true          # show VM window (default: true for UTM)

defaults:
  cores: 4
  memory: 4096
  disk: 50

vms:
  - name: incus-lab-01
    app: incus
    apply_defaults: true
```

---

## Limitations and Trade-offs

| Aspect | Proxmox | UTM |
|--------|---------|-----|
| Automation level | Full (API/SSH) | Partial (utmctl + AppleScript) |
| Headless operation | Yes | Limited (UTM needs to be running) |
| Remote deployment | Yes (network API) | No (local only) |
| Multi-node scaling | Tested up to 4 VMs | Limited by host RAM/CPU |
| Install monitoring | Real-time (blockstat) | Timeout-based |
| Resource isolation | Proxmox pools | None |
| Nested virt | Intel VT-x passthrough | Apple HV.framework (arm64) |

### When to use which

- **Proxmox**: production labs, multi-node clusters, automated CI, remote
  management, teams
- **UTM**: local development, single-developer labs, macOS-only environments,
  quick experiments

---

## Implementation Roadmap

1. **Research phase** (now): document UTM capabilities, `utmctl` API surface,
   AppleScript integration
2. **Seed generation**: already works (`incusos-seed` is cross-platform
   after Phase 3 macOS compatibility)
3. **VM creation**: implement AppleScript or `.utm` bundle generation
4. **Install flow**: implement timeout-based install detection
5. **Status/lifecycle**: implement `--status`, `--lab-up`, `--lab-down`
   using `utmctl`
6. **Testing**: test on macOS with Apple Silicon and Intel Macs

---

## References

- [UTM documentation](https://docs.getutm.app/)
- [utmctl CLI](https://docs.getutm.app/advanced/remote-control/)
- [UTM scripting](https://docs.getutm.app/scripting/scripting/)
- [Apple Hypervisor.framework](https://developer.apple.com/documentation/hypervisor)