maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement end-to-end IncusOS Proxmox deployment via API 

Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.

Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
  ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-19 15:26:58 +01:00
parent dc9814d92d
commit a0a4b48c73
9 changed files with 663 additions and 174 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
CLAUDE.md
View File

@ -18,6 +18,7 @@ incu-contrib/
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
│ ├── incusos-seed # Seed archive generator │ ├── incusos-seed # Seed archive generator
│ ├── incusos-proxmox # Declarative Proxmox VM deployment │ ├── incusos-proxmox # Declarative Proxmox VM deployment
│ ├── TESTING.md # Testing guide for incusos-proxmox
│ └── examples/ # Example seed + Proxmox YAML files │ └── examples/ # Example seed + Proxmox YAML files
└── notes/ # Research notes and reference material └── notes/ # Research notes and reference material
``` ```
@ -51,7 +52,12 @@ incu-contrib/
- Tar archives containing YAML files at the root level. - Tar archives containing YAML files at the root level.
- Written to byte offset 2148532224 (the seed partition) in the image. - Written to byte offset 2148532224 (the seed partition) in the image.
- Alternative: FAT image labeled `SEED_DATA` as external boot media. - Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
- **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
**NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
FAT filesystem labels, so IncusOS cannot find the seed.
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`, - Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
`operations-center.yaml`, `network.yaml`, `update.yaml`. `operations-center.yaml`, `network.yaml`, `update.yaml`.
@ -59,7 +65,9 @@ incu-contrib/
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`. - Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
- Running `incus remote list` triggers auto-generation if no keypair exists. - Running `incus remote list` triggers auto-generation if no keypair exists.
- For Incus seed: injected under `preseed.server.certificates[]`. - For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
`certificates`) are promoted to the top level of the `preseed` object.
- For Operations Center seed: injected under `trusted_client_certificates[]`. - For Operations Center seed: injected under `trusted_client_certificates[]`.
- Operations Center **requires** at least one trusted certificate -- without it, - Operations Center **requires** at least one trusted certificate -- without it,
you are locked out after installation. you are locked out after installation.
@ -67,7 +75,7 @@ incu-contrib/
### Proxmox VE deployment ### Proxmox VE deployment
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
via `incusos-seed --format fat`, uploads the ISO + seeds to Proxmox, creates via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
VMs with IncusOS-correct settings, and boots them through installation. VMs with IncusOS-correct settings, and boots them through installation.
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
@ -85,12 +93,29 @@ incu-contrib/
- `cpu=host` -- needed for x86_64_v3 instruction set requirement - `cpu=host` -- needed for x86_64_v3 instruction set requirement
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS - `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
- `balloon=0` -- IncusOS manages memory internally - `balloon=0` -- IncusOS manages memory internally
- `ide3` -- SEED_DATA FAT image attached as second CD-ROM - `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
- Minimum 50 GiB disk, minimum 4096 MiB RAM - Minimum 50 GiB disk, minimum 4096 MiB RAM
- **Install flow**: boot from ISO (ide2) -> read SEED_DATA (ide3) -> install - **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
to disk (scsi0) -> force_reboot -> boot from disk. Boot order is swapped IncusOS does **literal** string matching (not glob) on disk device IDs.
to `order=scsi0;ide2` immediately after first start so the reboot lands `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
on disk. `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
- **Install flow** (automated by `incusos-proxmox`):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
and refuses to start if found, regardless of boot order
6. Set boot order to `order=scsi0` and start from disk
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
ping broadcast → look up MAC in ARP table. Verify with direct ping before
trusting the result.
- **force_reboot is required**: without `force_reboot: true` in the seed,
IncusOS sits at "please remove installation media" and waits indefinitely.
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
## Coding conventions for scripts ## Coding conventions for scripts

56
incusos/README.md
View File

@ -7,7 +7,7 @@ Scripts for building IncusOS installation media and seed configurations.
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool | | [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar or FAT) for installation customization | | [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE | | [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
All scripts support automatic client certificate injection from the local Incus All scripts support automatic client certificate injection from the local Incus
@ -43,7 +43,10 @@ installation, so the freshly installed system is immediately manageable.
**Optional:** **Optional:**
- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`) - **genisoimage** -- for creating ISO 9660 seed images (`incusos-seed --format iso`).
Required by `incusos-proxmox` (Proxmox CD-ROM devices need ISO 9660, not FAT).
- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`).
Only for USB/block device use; FAT labels are NOT detected on CD-ROM devices.
- **jq** or **python3** -- for parsing the CDN index (cross-architecture builds) - **jq** or **python3** -- for parsing the CDN index (cross-architecture builds)
- **curl** -- for downloading images from the CDN - **curl** -- for downloading images from the CDN
@ -149,7 +152,8 @@ Generates IncusOS seed archives containing the YAML configuration files
that customize an IncusOS installation. Output can be: that customize an IncusOS installation. Output can be:
- **tar archive** (default) -- for use with `flasher-tool --seed` - **tar archive** (default) -- for use with `flasher-tool --seed`
- **FAT image** -- for use as an external SEED_DATA partition/disk - **ISO 9660 image** -- for use as a CD-ROM SEED_DATA device (e.g. Proxmox `ide3`)
- **FAT image** -- for use as a USB/block SEED_DATA device (NOT for CD-ROM)
### Quick Start ### Quick Start
@ -160,8 +164,8 @@ that customize an IncusOS installation. Output can be:
# Operations Center seed # Operations Center seed
./incusos-seed --app operations-center --defaults --cert ~/.config/incus/client.crt ./incusos-seed --app operations-center --defaults --cert ~/.config/incus/client.crt
# Cluster node (no defaults) targeting virtio disks # Cluster node (no defaults) with forced install and reboot
./incusos-seed --disk-target 'virtio-*' --force-install --force-reboot ./incusos-seed --force-install --force-reboot
# FAT image for external boot media # FAT image for external boot media
./incusos-seed --format fat --defaults --output seed.img ./incusos-seed --format fat --defaults --output seed.img
@ -178,7 +182,7 @@ that customize an IncusOS installation. Output can be:
-c, --cert FILE Client certificate PEM file (default: auto-detect) -c, --cert FILE Client certificate PEM file (default: auto-detect)
--no-cert Skip certificate injection --no-cert Skip certificate injection
-o, --output FILE Output file (default: incusos-seed.tar) -o, --output FILE Output file (default: incusos-seed.tar)
-f, --format FORMAT tar or fat (default: tar) -f, --format FORMAT tar, fat, or iso (default: tar)
--fat-size SIZE FAT image size in MiB (default: 10) --fat-size SIZE FAT image size in MiB (default: 10)
--disk-target ID Target disk pattern --disk-target ID Target disk pattern
--force-install Overwrite existing installation --force-install Overwrite existing installation
@ -209,24 +213,40 @@ The following YAML files are generated inside the archive:
| `network.yaml` | when network opts set | Hostname, DNS, static IP | | `network.yaml` | when network opts set | Hostname, DNS, static IP |
| `update.yaml` | when non-default | Update channel, auto-reboot | | `update.yaml` | when non-default | Update channel, auto-reboot |
### Using the FAT Image (SEED_DATA) ### Using ISO 9660 Images (SEED_DATA for CD-ROM)
The ISO output format creates a small ISO 9660 image with volume label `SEED_DATA`.
This is the **required** format when attaching seed data as a CD-ROM device (e.g.
Proxmox `ide3`). FAT filesystem labels are NOT detected by the Linux kernel on
CD-ROM (`/dev/sr*`) devices — only ISO 9660 volume labels work.
```bash
# Generate ISO seed for Proxmox deployment
./incusos-seed --format iso --defaults --output seed-node-1.iso
```
`incusos-proxmox` uses this format automatically.
### Using FAT Images (SEED_DATA for USB/Block Devices)
The FAT output format creates a small disk image labeled `SEED_DATA` that can be The FAT output format creates a small disk image labeled `SEED_DATA` that can be
attached to a VM as a secondary disk. IncusOS detects this label at boot and attached to a VM as a secondary block device or written to USB. IncusOS detects
reads the seed configuration from it. this label at boot and reads the seed configuration from it.
This is useful for the "pristine image" workflow: **Note:** FAT images do NOT work when attached as CD-ROM devices. Use ISO 9660
format instead for CD-ROM use cases.
This is useful for the "pristine image" workflow on bare metal:
1. Download one uncustomized IncusOS ISO 1. Download one uncustomized IncusOS ISO
2. Generate per-node seed images with `incusos-seed --format fat` 2. Generate per-node seed images with `incusos-seed --format fat`
3. Boot each VM with both the ISO and the node-specific seed image 3. Boot each machine with both the ISO and the node-specific seed USB
```bash ```bash
# Generate seeds for three cluster nodes # Generate seeds for three cluster nodes (USB/block device use)
for i in 1 2 3; do for i in 1 2 3; do
./incusos-seed --format fat \ ./incusos-seed --format fat \
--hostname "node-${i}" \ --hostname "node-${i}" \
--disk-target 'virtio-*' \
--force-install \ --force-install \
--output "seed-node-${i}.img" --output "seed-node-${i}.img"
done done
@ -287,7 +307,7 @@ This script ensures all settings are correct, preventing install failures.
| `prepare` | Download IncusOS ISO from CDN, generate per-VM SEED_DATA images | | `prepare` | Download IncusOS ISO from CDN, generate per-VM SEED_DATA images |
| `upload` | Upload ISO and seed images to Proxmox ISO storage | | `upload` | Upload ISO and seed images to Proxmox ISO storage |
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) | | `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
| `install` | Start VMs, swap boot order, wait for install, clean up CD-ROMs | | `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
### Config File Format ### Config File Format
@ -357,11 +377,12 @@ Every VM is created with these IncusOS-specific settings:
| CPU | host | Needs x86_64_v3 instruction set | | CPU | host | Needs x86_64_v3 instruction set |
| SCSI controller | virtio-scsi-pci | VirtIO-blk is broken with IncusOS | | SCSI controller | virtio-scsi-pci | VirtIO-blk is broken with IncusOS |
| Balloon | 0 (disabled) | IncusOS manages memory internally | | Balloon | 0 (disabled) | IncusOS manages memory internally |
| SEED_DATA | Attached as ide3 CD-ROM | Per-VM seed configuration | | SEED_DATA | ISO 9660 image on ide3 CD-ROM | Per-VM seed configuration |
### Prerequisites ### Prerequisites
- **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing - **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing
- **genisoimage** -- for creating ISO 9660 seed images
- **ssh** + **scp** (for SSH method) or **curl** (for API method) - **ssh** + **scp** (for SSH method) or **curl** (for API method)
- **incusos-seed** -- sibling script (auto-detected in same directory) - **incusos-seed** -- sibling script (auto-detected in same directory)
- **Bash 4+** -- associative arrays - **Bash 4+** -- associative arrays
@ -406,10 +427,11 @@ For full installation instructions per platform, see
### After Installation ### After Installation
With an injected certificate, the remote is already trusted: With an injected certificate, the remote is already trusted — no authentication
prompt appears:
```bash ```bash
incus remote add my-server <ip-address> incus remote add my-server <ip-address> --accept-certificate
incus remote list incus remote list
``` ```

28
incusos/TESTING.md
View File

@ -11,12 +11,13 @@ the full lifecycle:
- **Config parsing** -- YAML with PyYAML or built-in fallback parser - **Config parsing** -- YAML with PyYAML or built-in fallback parser
- **ISO download** -- fetches latest IncusOS from the CDN by channel - **ISO download** -- fetches latest IncusOS from the CDN by channel
(stable/testing) (stable/testing)
- **Per-VM seed generation** -- calls `incusos-seed --format fat` for each VM - **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method) - **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM, - **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
VirtIO-SCSI, etc.) VirtIO-SCSI, etc.)
- **Automated install monitoring** -- state-machine polling with 10-minute - **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
timeout to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
detaches install media (ide2 + ide3), and boots from disk
- **Cleanup** -- safe teardown with managed-VM marker checks - **Cleanup** -- safe teardown with managed-VM marker checks
### Cluster Formation (Manual Step) ### Cluster Formation (Manual Step)
@ -109,7 +110,7 @@ For a home lab, `/` (root) is simpler and perfectly acceptable.
bash --version # Need 4+ bash --version # Need 4+
python3 --version # For YAML parsing python3 --version # For YAML parsing
which jq curl ssh scp # All needed for SSH method which jq curl ssh scp # All needed for SSH method
which mcopy mkfs.fat # Needed for seed FAT images (mtools + dosfstools) which genisoimage # Needed for seed ISO images (used by incusos-proxmox)
# Verify the Incus client has generated a keypair # Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key ls ~/.config/incus/client.crt ~/.config/incus/client.key
@ -184,7 +185,7 @@ Using VMID 900+ avoids clashing with existing VMs.
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed 3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
4. **Upload** -- SCPs ISO + seed to Proxmox 4. **Upload** -- SCPs ISO + seed to Proxmox
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings 5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
6. **Install** -- starts VM, monitors install completion (~3-5 min) 6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)
**Simultaneously on the Proxmox web UI:** **Simultaneously on the Proxmox web UI:**
@ -200,8 +201,8 @@ Using VMID 900+ avoids clashing with existing VMs.
Once the script reports success with an IP address: Once the script reports success with an IP address:
```bash ```bash
# The cert was injected via seed, so it's already trusted # The cert was injected via seed, so no auth prompt appears
incus remote add incus-test https://<REPORTED_IP>:8443 --accept-certificate incus remote add incus-test <REPORTED_IP> --accept-certificate
# Test connectivity # Test connectivity
incus remote list incus remote list
@ -212,6 +213,10 @@ incus storage list incus-test:
incus network list incus-test: incus network list incus-test:
``` ```
**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading.
### Step 6: Clean Up the Test VM ### Step 6: Clean Up the Test VM
```bash ```bash
@ -292,7 +297,10 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` | | ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` | | VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI | | Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
| No IP reported | Guest agent not ready | Wait a minute, check Proxmox UI for IP | | No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one | | Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
| `mcopy` not found | mtools not installed | Install `mtools` package | | `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
| `mkfs.fat` not found | dosfstools not installed | Install `dosfstools` package |

3
incusos/examples/incus-cluster-node.yaml
View File

@ -13,11 +13,10 @@
# --- incus.yaml --- # --- incus.yaml ---
apply_defaults: false apply_defaults: false
preseed: preseed:
server:
certificates: certificates:
- name: "my-workstation" - name: "my-workstation"
type: "client" type: "client"
certificate: | certificate: |-
-----BEGIN CERTIFICATE----- -----BEGIN CERTIFICATE-----
<paste your certificate from: incus remote get-client-certificate> <paste your certificate from: incus remote get-client-certificate>
-----END CERTIFICATE----- -----END CERTIFICATE-----

3
incusos/examples/incus-standalone.yaml
View File

@ -28,11 +28,10 @@
# --- incus.yaml --- # --- incus.yaml ---
apply_defaults: true apply_defaults: true
preseed: preseed:
server:
certificates: certificates:
- name: "my-workstation" - name: "my-workstation"
type: "client" type: "client"
certificate: | certificate: |-
-----BEGIN CERTIFICATE----- -----BEGIN CERTIFICATE-----
<paste your certificate from: incus remote get-client-certificate> <paste your certificate from: incus remote get-client-certificate>
-----END CERTIFICATE----- -----END CERTIFICATE-----

17
incusos/examples/my-test.yaml Normal file
View File

@ -0,0 +1,17 @@
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs # VM disk storage (default: local-lvm)
defaults:
start_vmid: 900 # High range to avoid collisions
vms:
- name: incus-test
app: incus
apply_defaults: true
cores: 4
memory: 8192
disk: 64

452
incusos/incusos-proxmox
View File

@ -423,9 +423,20 @@ check_dependencies() {
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
WORKDIR="" WORKDIR=""
CACHEDIR=""
setup_workdir() { setup_workdir() {
WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-XXXXXX") # Use persistent storage for the work directory -- the ISO download +
# decompression can exceed 4 GiB, which overflows tmpfs on most systems.
# Honour TMPDIR if explicitly set; otherwise use ~/.cache on real disk.
local base="${TMPDIR:-${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}}"
mkdir -p "$base"
# Persistent cache for ISO downloads (survives across runs)
CACHEDIR="${base}/iso-cache"
mkdir -p "$CACHEDIR"
WORKDIR=$(mktemp -d "${base}/${SCRIPT_NAME}-XXXXXX")
trap 'rm -rf "$WORKDIR"' EXIT trap 'rm -rf "$WORKDIR"' EXIT
} }
@ -1017,10 +1028,13 @@ pve_run() {
} }
# Execute a Proxmox API call (API method) # Execute a Proxmox API call (API method)
# Usage: pve_api METHOD ENDPOINT [DATA] [json]
# Pass "json" as 4th arg to send data as application/json instead of form-encoded.
pve_api() { pve_api() {
local method="$1" local method="$1"
local endpoint="$2" local endpoint="$2"
local data="${3:-}" local data="${3:-}"
local content_type="${4:-form}"
local url="https://${PVE_HOST}:8006/api2/json${endpoint}" local url="https://${PVE_HOST}:8006/api2/json${endpoint}"
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
@ -1034,6 +1048,9 @@ pve_api() {
fi fi
local curl_args=(-fsSk -H "$auth") local curl_args=(-fsSk -H "$auth")
if [[ "$content_type" == "json" ]]; then
curl_args+=(-H "Content-Type: application/json")
fi
case "$method" in case "$method" in
GET) GET)
@ -1082,13 +1099,48 @@ pve_upload() {
"$local_path" "${PVE_SSH_USER}@${PVE_HOST}:${iso_dir}/${remote_filename}" "$local_path" "${PVE_SSH_USER}@${PVE_HOST}:${iso_dir}/${remote_filename}"
;; ;;
api) api)
curl -fsSk \ local upload_response upload_rc
upload_response=$(curl -sSk \
-H "Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" \ -H "Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" \
-X POST \ -X POST \
-F "content=iso" \ -F "content=iso" \
-F "filename=${remote_filename}" \ -F "filename=@${local_path};filename=${remote_filename}" \
-F "file=@${local_path}" \ "https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" \
"https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" -w "\n%{http_code}" 2>&1) || true
local http_code
http_code=$(echo "$upload_response" | tail -1)
local body
body=$(echo "$upload_response" | sed '$d')
if [[ "$http_code" -ge 200 ]] && [[ "$http_code" -lt 300 ]] 2>/dev/null; then
return 0
else
error "Upload failed (HTTP ${http_code})"
# Extract error message from JSON response
local api_error
api_error=$(echo "$body" | python3 -c "
import json, sys
try:
data = json.load(sys.stdin)
errors = data.get('errors', {})
if errors:
for k, v in errors.items():
print(f' {k}: {v}')
elif data.get('message'):
print(f' {data[\"message\"]}')
else:
print(json.dumps(data, indent=2))
except:
print(sys.stdin.read() if hasattr(sys.stdin, 'read') else '')
" 2>/dev/null) || api_error="$body"
if [[ -n "$api_error" ]]; then
error "Proxmox API response:"
echo "$api_error" | while IFS= read -r line; do
error " ${line}"
done
fi
return 1
fi
;; ;;
esac esac
} }
@ -1178,33 +1230,34 @@ pve_create_vm() {
pve_run "$cmd" pve_run "$cmd"
;; ;;
api) api)
# URL-encode the description for the API # Build JSON payload -- the Proxmox API requires JSON for VM
local url_encoded_desc # creation because comma-separated disk/net specs are misinterpreted
url_encoded_desc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${description}'))" 2>/dev/null) # when sent as form-encoded data.
local json_payload
# Build API parameters json_payload=$(python3 -c "
local params="" import json
params+="vmid=${vmid}" print(json.dumps({
params+="&name=${name}" 'vmid': ${vmid},
params+="&description=${url_encoded_desc}" 'name': '${name}',
params+="&ostype=l26" 'description': '''${description}''',
params+="&bios=ovmf" 'ostype': 'l26',
params+="&machine=q35" 'bios': 'ovmf',
params+="&efidisk0=${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0" 'machine': 'q35',
params+="&tpmstate0=${PVE_STORAGE}:1,version=v2.0" 'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0',
params+="&cpu=host" 'tpmstate0': '${PVE_STORAGE}:1,version=v2.0',
params+="&cores=${cores}" 'cpu': 'host',
params+="&memory=${memory}" 'cores': ${cores},
params+="&balloon=0" 'memory': ${memory},
params+="&scsihw=virtio-scsi-pci" 'balloon': 0,
params+="&scsi0=${PVE_STORAGE}:${disk_gib},iothread=1" 'scsihw': 'virtio-scsi-pci',
params+="&ide2=${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom" 'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1',
params+="&ide3=${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom" 'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom',
params+="&net0=virtio,bridge=${PVE_BRIDGE}" 'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom',
params+="&boot=order%3Dide2%3Bscsi0" 'net0': 'virtio,bridge=${PVE_BRIDGE}',
params+="&agent=1" 'boot': 'order=ide2;scsi0',
'agent': 1,
pve_api POST "/nodes/${PVE_NODE}/qemu" "$params" }))")
pve_api POST "/nodes/${PVE_NODE}/qemu" "$json_payload" json >/dev/null
;; ;;
esac esac
} }
@ -1215,7 +1268,7 @@ pve_start_vm() {
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) pve_run "qm start ${vmid}" ;; ssh) pve_run "qm start ${vmid}" ;;
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" ;; api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" >/dev/null ;;
esac esac
} }
@ -1225,7 +1278,7 @@ pve_stop_vm() {
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) pve_run "qm stop ${vmid}" ;; ssh) pve_run "qm stop ${vmid}" ;;
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" ;; api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" >/dev/null ;;
esac esac
} }
@ -1235,7 +1288,7 @@ pve_destroy_vm() {
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) pve_run "qm destroy ${vmid} --purge" ;; ssh) pve_run "qm destroy ${vmid} --purge" ;;
api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" ;; api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;;
esac esac
} }
@ -1248,10 +1301,35 @@ pve_set_vm() {
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) pve_run "qm set ${vmid} ${opts}" ;; ssh) pve_run "qm set ${vmid} ${opts}" ;;
api) api)
# Convert CLI-style opts to API params # Convert CLI-style opts (--key value) to JSON.
local params # JSON avoids form-encoding ambiguity with values containing = and ;
params=$(echo "$opts" | sed 's/--//g; s/ /\&/g; s/\&\([a-z]\)/\&\1/g') # Duplicate keys (e.g. --delete ide2 --delete ide3) are merged with commas.
pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$params" # Shell escapes (e.g. \;) are stripped since the API doesn't need them.
local json_payload
json_payload=$(echo "$opts" | python3 -c "
import json, sys
tokens = sys.stdin.read().split()
obj = {}
i = 0
while i < len(tokens):
t = tokens[i]
if t.startswith('--'):
key = t.lstrip('-')
if i + 1 < len(tokens) and not tokens[i+1].startswith('--'):
val = tokens[i+1].replace(chr(92), '')
i += 2
else:
val = ''
i += 1
if key in obj and obj[key]:
obj[key] = obj[key] + ',' + val
else:
obj[key] = val
else:
i += 1
print(json.dumps(obj))
" 2>/dev/null)
pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$json_payload" json >/dev/null
;; ;;
esac esac
} }
@ -1271,6 +1349,42 @@ pve_vm_status() {
esac esac
} }
# Get VM uptime (seconds); returns 0 if stopped or unknown
pve_vm_uptime() {
local vmid="$1"
case "$PVE_METHOD" in
ssh)
pve_run "qm status ${vmid} --verbose" | awk '/^uptime:/{print $2}'
;;
api)
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('uptime',0))" 2>/dev/null || echo "0"
;;
esac
}
# Get cumulative write bytes for scsi0 (from QEMU blockstat).
# Returns 0 if unavailable. Only works with api method.
pve_vm_disk_writes() {
local vmid="$1"
case "$PVE_METHOD" in
ssh)
echo "0"
;;
api)
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \
python3 -c "
import json,sys
d = json.load(sys.stdin).get('data',{})
bs = d.get('blockstat',{}).get('scsi0',{})
print(bs.get('wr_bytes',0))
" 2>/dev/null || echo "0"
;;
esac
}
# Check if a VMID exists # Check if a VMID exists
pve_vm_exists() { pve_vm_exists() {
local vmid="$1" local vmid="$1"
@ -1293,9 +1407,11 @@ pve_vm_exists() {
pve_vm_ip() { pve_vm_ip() {
local vmid="$1" local vmid="$1"
# Method 1: Try QEMU guest agent (works if agent is installed)
local ip=""
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) ssh)
pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \ ip=$(pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \
python3 -c " python3 -c "
import json, sys import json, sys
try: try:
@ -1309,10 +1425,10 @@ try:
sys.exit(0) sys.exit(0)
except: except:
pass pass
" 2>/dev/null || true " 2>/dev/null) || true
;; ;;
api) api)
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \ ip=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \
python3 -c " python3 -c "
import json, sys import json, sys
try: try:
@ -1326,9 +1442,69 @@ try:
sys.exit(0) sys.exit(0)
except: except:
pass pass
" 2>/dev/null || true " 2>/dev/null) || true
;; ;;
esac esac
if [[ -n "$ip" ]]; then
echo "$ip"
return 0
fi
# Method 2: ARP-based lookup using VM's MAC address.
# Works without guest agent (e.g. IncusOS which is immutable).
# Get MAC from VM config, flush stale ARP, ping sweep, look up.
local mac=""
case "$PVE_METHOD" in
ssh)
mac=$(pve_run "qm config ${vmid}" 2>/dev/null | \
awk -F'=' '/^net0:/{print $1}' | grep -oiE '[0-9a-f:]{17}') || true
;;
api)
mac=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null | \
python3 -c "
import json, sys, re
try:
cfg = json.load(sys.stdin).get('data', {})
net0 = cfg.get('net0', '')
m = re.search(r'([0-9A-Fa-f:]{17})', net0)
if m:
print(m.group(1).lower())
except:
pass
" 2>/dev/null) || true
;;
esac
if [[ -z "$mac" ]]; then
return 1
fi
# Flush stale ARP entries for this MAC, ping broadcast to populate, then look up
ip neigh flush nud stale 2>/dev/null || true
local bridge_subnet
bridge_subnet=$(ip -4 addr show | awk '/inet /{print $2}' | head -1)
if [[ -n "$bridge_subnet" ]]; then
local broadcast
broadcast=$(python3 -c "import ipaddress; print(ipaddress.ip_interface('${bridge_subnet}').network.broadcast_address)" 2>/dev/null) || true
if [[ -n "$broadcast" ]]; then
ping -c 2 -W 1 -b "$broadcast" &>/dev/null || true
sleep 1
fi
fi
# Look up MAC in ARP table (only REACHABLE or DELAY, not STALE)
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/ || /STALE/) {print $1; exit}') || true
if [[ -n "$ip" ]]; then
# Verify it's actually reachable
if ping -c 1 -W 1 "$ip" &>/dev/null; then
echo "$ip"
return 0
fi
fi
return 1
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1416,9 +1592,10 @@ preflight_checks() {
fi fi
;; ;;
api) api)
local storage_json
storage_json=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null) || storage_json=""
local storage_list local storage_list
storage_list=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null | \ storage_list=$(echo "$storage_json" | python3 -c "
python3 -c "
import json, sys import json, sys
data = json.load(sys.stdin) data = json.load(sys.stdin)
for s in data.get('data', []): for s in data.get('data', []):
@ -1435,6 +1612,22 @@ for s in data.get('data', []):
error "Available pools: $(echo "$storage_list" | tr '\n' ' ')" error "Available pools: $(echo "$storage_list" | tr '\n' ' ')"
error "Fix: update 'proxmox.iso_storage' in your config file" error "Fix: update 'proxmox.iso_storage' in your config file"
errors=$((errors + 1)) errors=$((errors + 1))
else
# Check that ISO storage accepts 'iso' content type
local iso_content
iso_content=$(echo "$storage_json" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for s in data.get('data', []):
if s.get('storage') == '${PVE_ISO_STORAGE}':
print(s.get('content', ''))
break
" 2>/dev/null) || iso_content=""
if [[ -n "$iso_content" ]] && ! echo "$iso_content" | grep -q "iso"; then
error "Storage '${PVE_ISO_STORAGE}' does not accept ISO content (content types: ${iso_content})"
error "Fix: enable 'iso' content on '${PVE_ISO_STORAGE}' in Proxmox, or use a different iso_storage"
errors=$((errors + 1))
fi
fi fi
;; ;;
esac esac
@ -1475,15 +1668,10 @@ for s in data.get('data', []):
fi fi
fi fi
# 4. FAT tools for SEED_DATA generation # 4. genisoimage for SEED_DATA ISO 9660 generation
if ! command -v mcopy &>/dev/null; then if ! command -v genisoimage &>/dev/null; then
error "mcopy (mtools) is required for SEED_DATA generation but not found" error "genisoimage is required for SEED_DATA generation but not found"
error "Install mtools for your platform (e.g. mtools package)" error "Install with: sudo apt install genisoimage"
errors=$((errors + 1))
fi
if ! command -v mkfs.fat &>/dev/null; then
error "mkfs.fat (dosfstools) is required for SEED_DATA generation but not found"
error "Install dosfstools for your platform (e.g. dosfstools package)"
errors=$((errors + 1)) errors=$((errors + 1))
fi fi
@ -1506,12 +1694,6 @@ for s in data.get('data', []):
check_connectivity() { check_connectivity() {
step "Checking Proxmox connectivity (${PVE_METHOD})" step "Checking Proxmox connectivity (${PVE_METHOD})"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would check connectivity to ${PVE_HOST}"
success "Connectivity check skipped (dry-run)"
return 0
fi
case "$PVE_METHOD" in case "$PVE_METHOD" in
ssh) ssh)
if ! ssh -o BatchMode=yes -o ConnectTimeout=10 \ if ! ssh -o BatchMode=yes -o ConnectTimeout=10 \
@ -1529,8 +1711,10 @@ check_connectivity() {
success "Connected to ${PVE_HOST} (Proxmox VE ${pve_version})" success "Connected to ${PVE_HOST} (Proxmox VE ${pve_version})"
;; ;;
api) api)
local url="https://${PVE_HOST}:8006/api2/json/version"
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
local response local response
if ! response=$(pve_api GET "/version" 2>/dev/null); then if ! response=$(curl -fsSk -H "$auth" "$url" 2>/dev/null); then
error "Cannot connect to Proxmox API at https://${PVE_HOST}:8006" error "Cannot connect to Proxmox API at https://${PVE_HOST}:8006"
error "Check that:" error "Check that:"
error " 1. The host is reachable" error " 1. The host is reachable"
@ -1750,12 +1934,23 @@ if versions:
download_iso() { download_iso() {
local version="$1" local version="$1"
local output_path="$2" local output_path="$2"
local iso_filename
iso_filename=$(basename "$output_path")
# Check persistent cache first
local cached_iso="${CACHEDIR}/${iso_filename}"
if [[ -f "$cached_iso" ]]; then
success "ISO found in local cache: ${cached_iso}"
cp "$cached_iso" "$output_path"
return 0
fi
local url="${CDN_BASE}/${version}/x86_64/IncusOS_${version}.iso.gz" local url="${CDN_BASE}/${version}/x86_64/IncusOS_${version}.iso.gz"
local gz_path="${output_path}.gz" local gz_path="${cached_iso}.gz"
step "Downloading IncusOS ISO" step "Downloading IncusOS ISO"
detail "URL: ${url}" detail "URL: ${url}"
detail "Cache directory: ${CACHEDIR}"
if ! curl -fL --progress-bar "$url" -o "$gz_path"; then if ! curl -fL --progress-bar "$url" -o "$gz_path"; then
error "Failed to download ISO from ${url}" error "Failed to download ISO from ${url}"
@ -1765,12 +1960,19 @@ download_iso() {
step "Decompressing ISO" step "Decompressing ISO"
if ! gzip -d "$gz_path"; then if ! gzip -d "$gz_path"; then
error "Failed to decompress ISO" local avail
avail=$(df -h --output=avail "$(dirname "$gz_path")" 2>/dev/null | tail -1 | tr -d ' ') || avail="unknown"
error "Failed to decompress ISO (${avail} available on disk)"
error "The decompressed ISO requires several GiB of disk space"
error "Free up space, or set TMPDIR to a larger filesystem:"
error " TMPDIR=/path/with/space ${SCRIPT_NAME} ${CONFIG_FILE}"
rm -f "$gz_path" rm -f "$gz_path"
exit 1 exit 1
fi fi
success "ISO ready: ${output_path}" # Copy from cache to workdir
cp "$cached_iso" "$output_path"
success "ISO ready: ${output_path} (cached for future runs)"
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1904,16 +2106,15 @@ phase_prepare() {
local app="${VM_APPS[$i]}" local app="${VM_APPS[$i]}"
local apply_defaults="${VM_APPLY_DEFAULTS[$i]}" local apply_defaults="${VM_APPLY_DEFAULTS[$i]}"
local seed_output="${WORKDIR}/seed-${name}.img" local seed_output="${WORKDIR}/seed-${name}.iso"
detail "Generating seed for '${name}' (${app}, defaults=${apply_defaults})" detail "Generating seed for '${name}' (${app}, defaults=${apply_defaults})"
# Build incusos-seed command # Build incusos-seed command
local seed_cmd=("$SEED_TOOL") local seed_cmd=("$SEED_TOOL")
seed_cmd+=(--format fat) seed_cmd+=(--format iso)
seed_cmd+=(--app "$app") seed_cmd+=(--app "$app")
seed_cmd+=(--output "$seed_output") seed_cmd+=(--output "$seed_output")
seed_cmd+=(--disk-target "scsi-*")
seed_cmd+=(--force-install) seed_cmd+=(--force-install)
seed_cmd+=(--force-reboot) seed_cmd+=(--force-reboot)
seed_cmd+=(--hostname "$name") seed_cmd+=(--hostname "$name")
@ -1980,7 +2181,7 @@ phase_upload() {
continue continue
fi fi
local seed_file="seed-${name}.img" local seed_file="seed-${name}.iso"
local seed_path="${WORKDIR}/${seed_file}" local seed_path="${WORKDIR}/${seed_file}"
if [[ -f "$seed_path" ]] || [[ "$DRY_RUN" == true ]]; then if [[ -f "$seed_path" ]] || [[ "$DRY_RUN" == true ]]; then
@ -2020,7 +2221,7 @@ phase_create() {
local cores="${VM_CORES[$i]}" local cores="${VM_CORES[$i]}"
local memory="${VM_MEMORY[$i]}" local memory="${VM_MEMORY[$i]}"
local disk="${VM_DISK[$i]}" local disk="${VM_DISK[$i]}"
local seed_file="seed-${name}.img" local seed_file="seed-${name}.iso"
step "Creating VM '${name}' (VMID ${vmid})" step "Creating VM '${name}' (VMID ${vmid})"
@ -2093,27 +2294,30 @@ phase_install() {
fi fi
success "VM '${name}' started" success "VM '${name}' started"
# 2. Immediately swap boot order (takes effect on next reboot) # 2. Wait for installation to complete by monitoring disk writes
detail "Swapping boot order to disk-first" #
if ! pve_set_vm "$vmid" "--boot order=scsi0\\;ide2"; then # Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot
warn "Failed to swap boot order for '${name}' -- manual intervention may be needed" # (force_reboot in seed). A guest-level reboot does NOT reset the QEMU
fi # VM uptime, so we cannot detect reboot that way. Instead we monitor
# scsi0 write bytes via blockstat:
# 3. Wait for installation + reboot cycle # - During install: wr_bytes increases steadily
# - After install + reboot: wr_bytes stops (VM is idle at post-reboot
# screen or "install media detected" error)
#
# When wr_bytes has been stable for STABLE_COUNT consecutive polls,
# installation is complete. We then stop the VM, remove install media,
# and start cleanly from disk.
step "Waiting for '${name}' to complete installation" step "Waiting for '${name}' to complete installation"
detail "IncusOS installs from ISO, then reboots to disk automatically" detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
detail "Polling every 15s (timeout: 10 minutes)"
local timeout=600 local timeout=600
local interval=15 local interval=5
local elapsed=0 local elapsed=0
local booted=false local install_done=false
local prev_wr_bytes=0
# Wait for the VM to go through: running -> (install) -> stopped -> running local writes_started=false
# The force_reboot in seed causes an automatic reboot after install. local stable_count=0
# We detect the reboot by looking for the VM to be "running" after local STABLE_THRESHOLD=3 # 3 consecutive polls with no change = 15s idle
# having been "stopped" (or we just wait for it to stabilize).
local saw_stop=false
while [[ $elapsed -lt $timeout ]]; do while [[ $elapsed -lt $timeout ]]; do
sleep "$interval" sleep "$interval"
@ -2124,28 +2328,35 @@ phase_install() {
case "$status" in case "$status" in
stopped) stopped)
if [[ "$saw_stop" == false ]]; then detail "VM stopped [${elapsed}s]"
detail "VM stopped (installation complete, awaiting reboot) [${elapsed}s]" install_done=true
saw_stop=true break
# The VM should auto-reboot via force_reboot, but if it
# doesn't after 30s, start it manually
sleep 30
elapsed=$((elapsed + 30))
local recheck
recheck=$(pve_vm_status "$vmid") || recheck="unknown"
if [[ "$recheck" == "stopped" ]]; then
detail "VM did not auto-reboot, starting manually"
pve_start_vm "$vmid" || true
fi
fi
;; ;;
running) running)
if [[ "$saw_stop" == true ]] || [[ $elapsed -ge 120 ]]; then local wr_bytes
# VM has been running long enough post-reboot -- likely booted from disk wr_bytes=$(pve_vm_disk_writes "$vmid") || wr_bytes=0
booted=true
if [[ $wr_bytes -gt $prev_wr_bytes ]]; then
# Disk writes are happening -- install in progress
writes_started=true
stable_count=0
local wr_mb=$(( wr_bytes / 1048576 ))
# Show progress every 15s
if [[ $((elapsed % 15)) -eq 0 ]]; then
detail "Installing... (${wr_mb} MiB written) [${elapsed}s]"
fi
elif [[ "$writes_started" == true ]]; then
# Writes have stopped after being active
stable_count=$((stable_count + 1))
if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then
local wr_mb=$(( wr_bytes / 1048576 ))
detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
install_done=true
break break
fi fi
detail "VM running (installing...) [${elapsed}s]" fi
prev_wr_bytes=$wr_bytes
;; ;;
*) *)
detail "VM status: ${status} [${elapsed}s]" detail "VM status: ${status} [${elapsed}s]"
@ -2153,20 +2364,39 @@ phase_install() {
esac esac
done done
if [[ "$booted" != true ]]; then if [[ "$install_done" != true ]]; then
warn "VM '${name}' did not complete installation within ${timeout}s" warn "VM '${name}' did not complete installation within ${timeout}s"
warn "Check the VM console on Proxmox for details" warn "Check the VM console on Proxmox for details"
i=$((i + 1)) i=$((i + 1))
continue continue
fi fi
# 4. Give it a moment to fully initialize # 3. Stop the VM (it's either at "remove install media" prompt or
detail "Waiting for services to start..." # post-reboot "install media detected" error -- either way idle).
sleep 15 local cur_status
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
if [[ "$cur_status" == "running" ]]; then
detail "Stopping VM to remove install media..."
pve_stop_vm "$vmid" || true
sleep 5
fi
# 5. Remove CD-ROM drives # 4. Remove install media and set clean boot order
step "Cleaning up '${name}'" step "Removing install media from '${name}'"
pve_set_vm "$vmid" "--delete ide2 --delete ide3" || true pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true
success "Install media detached, boot order set to disk-only"
# 5. Start VM from disk
step "Starting '${name}' from disk"
if ! pve_start_vm "$vmid"; then
error "Failed to start VM '${name}' from disk"
i=$((i + 1))
continue
fi
# 5. Wait for services to come up
detail "Waiting for services to start..."
sleep 30
# 6. Try to detect IP # 6. Try to detect IP
local ip local ip

55
incusos/incusos-seed
View File

@ -76,8 +76,9 @@ ${BOLD}OPTIONS${RESET}
${BOLD}--auto-reboot${RESET} Enable automatic reboot for updates ${BOLD}--auto-reboot${RESET} Enable automatic reboot for updates
${BOLD}Output options:${RESET} ${BOLD}Output options:${RESET}
${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}fat${RESET} (default: tar) ${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}iso${RESET}, ${CYAN}fat${RESET} (default: tar)
tar: for use with flasher-tool --seed tar: for use with flasher-tool --seed
iso: ISO 9660 image with SEED_DATA label for CD-ROM
fat: FAT image with SEED_DATA label for external media fat: FAT image with SEED_DATA label for external media
${BOLD}--fat-size${RESET} SIZE FAT image size in MiB (default: 10) ${BOLD}--fat-size${RESET} SIZE FAT image size in MiB (default: 10)
@ -97,6 +98,9 @@ ${BOLD}EXAMPLES${RESET}
# Cluster node (no defaults) with forced install on virtio disk # Cluster node (no defaults) with forced install on virtio disk
${SCRIPT_NAME} --disk-target 'virtio-*' --force-install --force-reboot ${SCRIPT_NAME} --disk-target 'virtio-*' --force-install --force-reboot
# Create an ISO 9660 image for CD-ROM seed delivery (e.g. Proxmox VMs)
${SCRIPT_NAME} --format iso --defaults --output seed.iso
# Create a FAT image for SEED_DATA partition # Create a FAT image for SEED_DATA partition
${SCRIPT_NAME} --format fat --defaults --output seed.img ${SCRIPT_NAME} --format fat --defaults --output seed.img
@ -257,10 +261,10 @@ validate_args() {
esac esac
case "$FORMAT" in case "$FORMAT" in
tar|fat) ;; tar|iso|fat) ;;
*) *)
error "Invalid output format: ${FORMAT}" error "Invalid output format: ${FORMAT}"
error "Supported: tar, fat" error "Supported: tar, iso, fat"
exit 1 exit 1
;; ;;
esac esac
@ -284,8 +288,21 @@ validate_args() {
exit 1 exit 1
fi fi
if [[ "$FORMAT" == "iso" ]]; then
if ! command -v genisoimage &>/dev/null; then
error "ISO 9660 image creation requires genisoimage but it was not found"
error "Install with: sudo apt install genisoimage"
exit 1
fi
fi
if [[ "$FORMAT" == "fat" ]]; then if [[ "$FORMAT" == "fat" ]]; then
if ! command -v mcopy &>/dev/null && ! command -v mkfs.fat &>/dev/null; then # mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users
local has_mkfs_fat=false
if command -v mkfs.fat &>/dev/null || [[ -x /usr/sbin/mkfs.fat ]] || [[ -x /sbin/mkfs.fat ]]; then
has_mkfs_fat=true
fi
if ! command -v mcopy &>/dev/null && [[ "$has_mkfs_fat" != true ]]; then
error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)" error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)"
error "Install with: sudo apt install mtools dosfstools" error "Install with: sudo apt install mtools dosfstools"
exit 1 exit 1
@ -296,6 +313,7 @@ validate_args() {
if [[ -z "$OUTPUT" ]]; then if [[ -z "$OUTPUT" ]]; then
case "$FORMAT" in case "$FORMAT" in
tar) OUTPUT="incusos-seed.tar" ;; tar) OUTPUT="incusos-seed.tar" ;;
iso) OUTPUT="incusos-seed.iso" ;;
fat) OUTPUT="incusos-seed.img" ;; fat) OUTPUT="incusos-seed.img" ;;
esac esac
fi fi
@ -464,11 +482,10 @@ generate_incus_yaml() {
if [[ -n "$CERT_CONTENT" ]]; then if [[ -n "$CERT_CONTENT" ]]; then
echo "preseed:" echo "preseed:"
echo " server:"
echo " certificates:" echo " certificates:"
echo " - name: \"auto-injected-client\"" echo " - name: \"auto-injected-client\""
echo " type: \"client\"" echo " type: \"client\""
echo " certificate: |" echo " certificate: |-"
# Indent each line of the certificate # Indent each line of the certificate
while IFS= read -r line; do while IFS= read -r line; do
echo " ${line}" echo " ${line}"
@ -592,14 +609,34 @@ create_tar_output() {
success "Seed archive created: ${OUTPUT} (${size})" success "Seed archive created: ${OUTPUT} (${size})"
} }
create_iso_output() {
step "Creating ISO 9660 image: ${OUTPUT}"
genisoimage -o "$OUTPUT" -V SEED_DATA -J -r "$WORKDIR"/ 2>/dev/null
local size
size=$(du -h "$OUTPUT" | cut -f1)
success "ISO 9660 seed image created: ${OUTPUT} (${size})"
}
create_fat_output() { create_fat_output() {
step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)" step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)"
# Create empty image # Create empty image
dd if=/dev/zero of="$OUTPUT" bs=1M count="$FAT_SIZE" status=none dd if=/dev/zero of="$OUTPUT" bs=1M count="$FAT_SIZE" status=none
# mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users
local mkfs_fat=""
if command -v mkfs.fat &>/dev/null; then if command -v mkfs.fat &>/dev/null; then
mkfs.fat -n SEED_DATA "$OUTPUT" >/dev/null mkfs_fat="mkfs.fat"
elif [[ -x /usr/sbin/mkfs.fat ]]; then
mkfs_fat="/usr/sbin/mkfs.fat"
elif [[ -x /sbin/mkfs.fat ]]; then
mkfs_fat="/sbin/mkfs.fat"
fi
if [[ -n "$mkfs_fat" ]]; then
"$mkfs_fat" -n SEED_DATA "$OUTPUT" >/dev/null
if command -v mcopy &>/dev/null; then if command -v mcopy &>/dev/null; then
# Use mtools to copy files without mounting # Use mtools to copy files without mounting
@ -663,6 +700,9 @@ print_summary() {
if [[ "$FORMAT" == "tar" ]]; then if [[ "$FORMAT" == "tar" ]]; then
echo -e "${DIM} Use with flasher-tool:${RESET}" echo -e "${DIM} Use with flasher-tool:${RESET}"
echo -e " flasher-tool --seed ${OUTPUT}" echo -e " flasher-tool --seed ${OUTPUT}"
elif [[ "$FORMAT" == "iso" ]]; then
echo -e "${DIM} Attach as CD-ROM to the VM.${RESET}"
echo -e "${DIM} IncusOS will detect the SEED_DATA volume label at boot.${RESET}"
elif [[ "$FORMAT" == "fat" ]]; then elif [[ "$FORMAT" == "fat" ]]; then
echo -e "${DIM} Attach as secondary disk/ISO to the VM.${RESET}" echo -e "${DIM} Attach as secondary disk/ISO to the VM.${RESET}"
echo -e "${DIM} IncusOS will detect the SEED_DATA label at boot.${RESET}" echo -e "${DIM} IncusOS will detect the SEED_DATA label at boot.${RESET}"
@ -718,6 +758,7 @@ main() {
else else
case "$FORMAT" in case "$FORMAT" in
tar) create_tar_output ;; tar) create_tar_output ;;
iso) create_iso_output ;;
fat) create_fat_output ;; fat) create_fat_output ;;
esac esac

148
notes/proxmox-deploy-progress.md Normal file
View File

@ -0,0 +1,148 @@
# IncusOS Proxmox Deploy — Session Progress
_Last updated: 2026-02-19_
## Status: Complete — end-to-end automated deployment working
Full automated deployment of IncusOS to Proxmox VE via API is working.
Tested with a single standalone Incus VM (VMID 900) — seed generated,
ISO uploaded, VM created with correct settings, installation detected via
disk I/O monitoring, media ejected, booted from disk, Incus API accessible,
client certificate pre-trusted.
## Proxmox environment
| Item | Value |
|----------------------|-------------------------------------------|
| Host | 192.168.1.29 |
| Node name | pve |
| Proxmox VE version | 9.1.5 |
| API token ID | `automation@pve!deploy` |
| Token secret env var | `PROXMOX_TOKEN_SECRET` |
| ISO storage | `local` (directory, supports iso content) |
| VM disk storage | `local-zfs` (ZFS pool, images/rootdir) |
| Test config | `incusos/examples/my-test.yaml` |
| Test VM ID | 900 |
## Bugs found and fixed
### 1. FAT labels not detected on CD-ROM devices
**Symptom:** IncusOS reports "unable to begin install from read-only device
/dev/mapper/sr0 without seed configuration"
**Root cause:** Linux kernel's `sr_mod` driver does NOT expose FAT filesystem
labels for `/dev/sr*` devices. The `SEED_DATA` label is never created in
`/dev/disk/by-label/`.
**Fix:** Switched from FAT to ISO 9660 format. Added `--format iso` to
`incusos-seed` using `genisoimage -V SEED_DATA -J -r`. Changed
`incusos-proxmox` to use `--format iso` and `.iso` extension.
### 2. Disk target literal matching (not glob)
**Symptom:** "no target device matched 'scsi-*' (detected devices:
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)"
**Root cause:** IncusOS does literal string matching on disk target IDs,
not glob matching. `scsi-*` is not a valid device ID.
**Fix:** Removed `--disk-target "scsi-*"` from seed generation in
`incusos-proxmox`. IncusOS auto-detects when there's a single disk.
### 3. Preseed YAML structure (Go `yaml:",inline"`)
**Symptom:** "yaml: unmarshal errors: line 4 field server not found in
type api.initpreseed"
**Root cause:** The Incus `InitPreseed` struct has
`Server InitLocalPreseed` with `yaml:",inline"`, which promotes the
Server's fields (including `certificates`) to the parent level. The
correct path is `preseed.certificates[]`, NOT `preseed.server.certificates[]`.
**Fix:** Updated `generate_incus_yaml()` in `incusos-seed` and all
example files.
### 4. Install media detection blocking boot
**Symptom:** After installation and reboot, IncusOS refuses to boot with
"install media detected" error, regardless of boot order.
**Root cause:** IncusOS checks for the _presence_ of install media at
every boot. If IDE CD-ROMs are still attached, it blocks boot even when
disk is first in boot order.
**Fix:** After detecting install completion, the script now: stops the VM,
deletes ide2 and ide3 devices completely, sets boot order to `order=scsi0`,
then starts from disk.
### 5. Install completion detection
**Symptom:** Cannot reliably detect when IncusOS finishes installing.
Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start
does), making uptime-based detection impossible.
**Root cause:** `force_reboot: true` in the seed triggers a guest OS reboot,
but QEMU treats this as a guest-initiated reset — the VM process continues,
uptime keeps incrementing.
**Fix:** Poll `blockstat.scsi0.wr_bytes` via Proxmox API every 5 seconds.
During installation, disk writes increase (~876 MiB total). After install +
reboot, writes stop. Three consecutive stable readings (15 seconds of no
writes) = install done. Works reliably.
### 6. force_reboot is required for automation
**Symptom:** Without `force_reboot: true`, IncusOS sits at "please remove
installation media to complete the installation" and waits indefinitely.
**Root cause:** IncusOS does not auto-halt after installation. It expects
the user to remove media and manually reboot.
**Fix:** Always include `force_reboot: true` in the seed for automated
deployments.
### 7. IP detection without guest agent
**Symptom:** IncusOS is immutable with no QEMU guest agent. Standard
Proxmox guest agent IP detection returns nothing.
**Fix:** ARP-based detection: extract MAC from Proxmox VM config (net0
field), flush stale ARP entries, ping broadcast to populate ARP table,
look up MAC in ARP output. Verify with direct ping before returning.
### 8. Various API-related fixes (earlier session)
- **ISO upload HTTP 400**: Upload field must be named `filename` (not `file`)
- **ISO caching**: Added persistent cache at `~/.cache/incusos-proxmox/iso-cache/`
- **VM creation HTTP 400** ("duplicate key"): Use JSON Content-Type, not form-encoded
- **Boot order swap HTTP 400**: Convert CLI flags to JSON with Python helper
- **API response leak**: Suppress JSON responses from mutating API calls
## Verified deployment flow
1. Seed generated as ISO 9660 (372K) with `genisoimage`
2. IncusOS ISO reused from Proxmox cache (already uploaded)
3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.)
4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3
5. blockstat detects install completion at ~876 MiB written, ~50s elapsed
6. VM stopped, ide2 + ide3 deleted, boot order set to `order=scsi0`
7. VM started from disk — IncusOS boots successfully
8. Port 8443 open, Incus API responding
9. `incus remote add` works without auth prompt (certificate pre-trusted)
10. Storage pool (`local`, ZFS) and network bridge (`incusbr0`) confirmed
## Existing VMs on Proxmox
- 100: dev-vm-beelink
- 201: IncusOS-01
- 202: IncusOS-02
- 203: IncusOS-03
- 300: IncusOS-OC
- 900: incus-test (deployed and running)
## Next steps
- Multi-VM deployment testing (full lab with ops-center + cluster)
- Cluster formation automation (`incus cluster add` / join tokens)
- SSH method testing (currently only API method verified)