Implement end-to-end IncusOS Proxmox deployment via API
Major changes to incusos-proxmox and incusos-seed to achieve fully automated IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML structure (Go yaml:",inline"), disk target auto-detection, install completion monitoring via blockstat disk I/O, install media ejection, and ARP-based IP detection for the agent-less IncusOS. Key changes: - incusos-seed: add --format iso using genisoimage, fix preseed structure - incusos-proxmox: ISO seeds, blockstat install detection, media ejection, ARP-based IP lookup, force_reboot requirement - Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md - Examples: fix preseed.certificates path in all YAML examples - Notes: full session progress with all bugs found and fixed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
dc9814d92d
commit
a0a4b48c73
41
CLAUDE.md
41
CLAUDE.md
|
|
@ -18,6 +18,7 @@ incu-contrib/
|
|||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||
│ ├── incusos-seed # Seed archive generator
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
||||
│ ├── TESTING.md # Testing guide for incusos-proxmox
|
||||
│ └── examples/ # Example seed + Proxmox YAML files
|
||||
└── notes/ # Research notes and reference material
|
||||
```
|
||||
|
|
@ -51,7 +52,12 @@ incu-contrib/
|
|||
|
||||
- Tar archives containing YAML files at the root level.
|
||||
- Written to byte offset 2148532224 (the seed partition) in the image.
|
||||
- Alternative: FAT image labeled `SEED_DATA` as external boot media.
|
||||
- Alternative: external boot media labeled `SEED_DATA` attached as CD-ROM.
|
||||
- **ISO 9660** (`genisoimage -V SEED_DATA -J -r`): preferred for CD-ROM
|
||||
devices. Volume labels are correctly detected by the kernel on `/dev/sr*`.
|
||||
- **FAT image** (`mkfs.fat -n SEED_DATA`): works for USB/block devices but
|
||||
**NOT for CD-ROM** -- the Linux kernel's `sr_mod` driver does not expose
|
||||
FAT filesystem labels, so IncusOS cannot find the seed.
|
||||
- Key files: `install.yaml`, `applications.yaml`, `incus.yaml`,
|
||||
`operations-center.yaml`, `network.yaml`, `update.yaml`.
|
||||
|
||||
|
|
@ -59,7 +65,9 @@ incu-contrib/
|
|||
|
||||
- Stored at `~/.config/incus/client.crt` and `~/.config/incus/client.key`.
|
||||
- Running `incus remote list` triggers auto-generation if no keypair exists.
|
||||
- For Incus seed: injected under `preseed.server.certificates[]`.
|
||||
- For Incus seed: injected under `preseed.certificates[]` (NOT `preseed.server.certificates[]`).
|
||||
The `InitPreseed.Server` field uses `yaml:",inline"` so its fields (including
|
||||
`certificates`) are promoted to the top level of the `preseed` object.
|
||||
- For Operations Center seed: injected under `trusted_client_certificates[]`.
|
||||
- Operations Center **requires** at least one trusted certificate -- without it,
|
||||
you are locked out after installation.
|
||||
|
|
@ -67,7 +75,7 @@ incu-contrib/
|
|||
### Proxmox VE deployment
|
||||
|
||||
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
||||
via `incusos-seed --format fat`, uploads the ISO + seeds to Proxmox, creates
|
||||
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
||||
VMs with IncusOS-correct settings, and boots them through installation.
|
||||
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
||||
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
||||
|
|
@ -85,12 +93,29 @@ incu-contrib/
|
|||
- `cpu=host` -- needed for x86_64_v3 instruction set requirement
|
||||
- `scsihw=virtio-scsi-pci` + `scsi0` -- VirtIO-blk is broken with IncusOS
|
||||
- `balloon=0` -- IncusOS manages memory internally
|
||||
- `ide3` -- SEED_DATA FAT image attached as second CD-ROM
|
||||
- `ide3` -- SEED_DATA **ISO 9660** image attached as second CD-ROM
|
||||
- Minimum 50 GiB disk, minimum 4096 MiB RAM
|
||||
- **Install flow**: boot from ISO (ide2) -> read SEED_DATA (ide3) -> install
|
||||
to disk (scsi0) -> force_reboot -> boot from disk. Boot order is swapped
|
||||
to `order=scsi0;ide2` immediately after first start so the reboot lands
|
||||
on disk.
|
||||
- **Disk target**: do NOT specify `disk-target` in the seed for Proxmox VMs.
|
||||
IncusOS does **literal** string matching (not glob) on disk device IDs.
|
||||
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
|
||||
`disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
|
||||
- **Install flow** (automated by `incusos-proxmox`):
|
||||
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
|
||||
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
|
||||
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
|
||||
when disk writes start then stop for 15s (3 stable polls), install is done
|
||||
4. Stop the VM (Proxmox stop, not guest shutdown)
|
||||
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
|
||||
and refuses to start if found, regardless of boot order
|
||||
6. Set boot order to `order=scsi0` and start from disk
|
||||
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
|
||||
ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
|
||||
ping broadcast → look up MAC in ARP table. Verify with direct ping before
|
||||
trusting the result.
|
||||
- **force_reboot is required**: without `force_reboot: true` in the seed,
|
||||
IncusOS sits at "please remove installation media" and waits indefinitely.
|
||||
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
|
||||
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
|
||||
|
||||
## Coding conventions for scripts
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Scripts for building IncusOS installation media and seed configurations.
|
|||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
|
||||
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar or FAT) for installation customization |
|
||||
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
|
||||
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
|
||||
|
||||
All scripts support automatic client certificate injection from the local Incus
|
||||
|
|
@ -43,7 +43,10 @@ installation, so the freshly installed system is immediately manageable.
|
|||
|
||||
**Optional:**
|
||||
|
||||
- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`)
|
||||
- **genisoimage** -- for creating ISO 9660 seed images (`incusos-seed --format iso`).
|
||||
Required by `incusos-proxmox` (Proxmox CD-ROM devices need ISO 9660, not FAT).
|
||||
- **mtools** / **dosfstools** -- for creating FAT seed images (`incusos-seed --format fat`).
|
||||
Only for USB/block device use; FAT labels are NOT detected on CD-ROM devices.
|
||||
- **jq** or **python3** -- for parsing the CDN index (cross-architecture builds)
|
||||
- **curl** -- for downloading images from the CDN
|
||||
|
||||
|
|
@ -149,7 +152,8 @@ Generates IncusOS seed archives containing the YAML configuration files
|
|||
that customize an IncusOS installation. Output can be:
|
||||
|
||||
- **tar archive** (default) -- for use with `flasher-tool --seed`
|
||||
- **FAT image** -- for use as an external SEED_DATA partition/disk
|
||||
- **ISO 9660 image** -- for use as a CD-ROM SEED_DATA device (e.g. Proxmox `ide3`)
|
||||
- **FAT image** -- for use as a USB/block SEED_DATA device (NOT for CD-ROM)
|
||||
|
||||
### Quick Start
|
||||
|
||||
|
|
@ -160,8 +164,8 @@ that customize an IncusOS installation. Output can be:
|
|||
# Operations Center seed
|
||||
./incusos-seed --app operations-center --defaults --cert ~/.config/incus/client.crt
|
||||
|
||||
# Cluster node (no defaults) targeting virtio disks
|
||||
./incusos-seed --disk-target 'virtio-*' --force-install --force-reboot
|
||||
# Cluster node (no defaults) with forced install and reboot
|
||||
./incusos-seed --force-install --force-reboot
|
||||
|
||||
# FAT image for external boot media
|
||||
./incusos-seed --format fat --defaults --output seed.img
|
||||
|
|
@ -178,7 +182,7 @@ that customize an IncusOS installation. Output can be:
|
|||
-c, --cert FILE Client certificate PEM file (default: auto-detect)
|
||||
--no-cert Skip certificate injection
|
||||
-o, --output FILE Output file (default: incusos-seed.tar)
|
||||
-f, --format FORMAT tar or fat (default: tar)
|
||||
-f, --format FORMAT tar, fat, or iso (default: tar)
|
||||
--fat-size SIZE FAT image size in MiB (default: 10)
|
||||
--disk-target ID Target disk pattern
|
||||
--force-install Overwrite existing installation
|
||||
|
|
@ -209,24 +213,40 @@ The following YAML files are generated inside the archive:
|
|||
| `network.yaml` | when network opts set | Hostname, DNS, static IP |
|
||||
| `update.yaml` | when non-default | Update channel, auto-reboot |
|
||||
|
||||
### Using the FAT Image (SEED_DATA)
|
||||
### Using ISO 9660 Images (SEED_DATA for CD-ROM)
|
||||
|
||||
The ISO output format creates a small ISO 9660 image with volume label `SEED_DATA`.
|
||||
This is the **required** format when attaching seed data as a CD-ROM device (e.g.
|
||||
Proxmox `ide3`). FAT filesystem labels are NOT detected by the Linux kernel on
|
||||
CD-ROM (`/dev/sr*`) devices — only ISO 9660 volume labels work.
|
||||
|
||||
```bash
|
||||
# Generate ISO seed for Proxmox deployment
|
||||
./incusos-seed --format iso --defaults --output seed-node-1.iso
|
||||
```
|
||||
|
||||
`incusos-proxmox` uses this format automatically.
|
||||
|
||||
### Using FAT Images (SEED_DATA for USB/Block Devices)
|
||||
|
||||
The FAT output format creates a small disk image labeled `SEED_DATA` that can be
|
||||
attached to a VM as a secondary disk. IncusOS detects this label at boot and
|
||||
reads the seed configuration from it.
|
||||
attached to a VM as a secondary block device or written to USB. IncusOS detects
|
||||
this label at boot and reads the seed configuration from it.
|
||||
|
||||
This is useful for the "pristine image" workflow:
|
||||
**Note:** FAT images do NOT work when attached as CD-ROM devices. Use ISO 9660
|
||||
format instead for CD-ROM use cases.
|
||||
|
||||
This is useful for the "pristine image" workflow on bare metal:
|
||||
|
||||
1. Download one uncustomized IncusOS ISO
|
||||
2. Generate per-node seed images with `incusos-seed --format fat`
|
||||
3. Boot each VM with both the ISO and the node-specific seed image
|
||||
3. Boot each machine with both the ISO and the node-specific seed USB
|
||||
|
||||
```bash
|
||||
# Generate seeds for three cluster nodes
|
||||
# Generate seeds for three cluster nodes (USB/block device use)
|
||||
for i in 1 2 3; do
|
||||
./incusos-seed --format fat \
|
||||
--hostname "node-${i}" \
|
||||
--disk-target 'virtio-*' \
|
||||
--force-install \
|
||||
--output "seed-node-${i}.img"
|
||||
done
|
||||
|
|
@ -287,7 +307,7 @@ This script ensures all settings are correct, preventing install failures.
|
|||
| `prepare` | Download IncusOS ISO from CDN, generate per-VM SEED_DATA images |
|
||||
| `upload` | Upload ISO and seed images to Proxmox ISO storage |
|
||||
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
|
||||
| `install` | Start VMs, swap boot order, wait for install, clean up CD-ROMs |
|
||||
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
||||
|
||||
### Config File Format
|
||||
|
||||
|
|
@ -357,11 +377,12 @@ Every VM is created with these IncusOS-specific settings:
|
|||
| CPU | host | Needs x86_64_v3 instruction set |
|
||||
| SCSI controller | virtio-scsi-pci | VirtIO-blk is broken with IncusOS |
|
||||
| Balloon | 0 (disabled) | IncusOS manages memory internally |
|
||||
| SEED_DATA | Attached as ide3 CD-ROM | Per-VM seed configuration |
|
||||
| SEED_DATA | ISO 9660 image on ide3 CD-ROM | Per-VM seed configuration |
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing
|
||||
- **genisoimage** -- for creating ISO 9660 seed images
|
||||
- **ssh** + **scp** (for SSH method) or **curl** (for API method)
|
||||
- **incusos-seed** -- sibling script (auto-detected in same directory)
|
||||
- **Bash 4+** -- associative arrays
|
||||
|
|
@ -406,10 +427,11 @@ For full installation instructions per platform, see
|
|||
|
||||
### After Installation
|
||||
|
||||
With an injected certificate, the remote is already trusted:
|
||||
With an injected certificate, the remote is already trusted — no authentication
|
||||
prompt appears:
|
||||
|
||||
```bash
|
||||
incus remote add my-server <ip-address>
|
||||
incus remote add my-server <ip-address> --accept-certificate
|
||||
incus remote list
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -11,12 +11,13 @@ the full lifecycle:
|
|||
- **Config parsing** -- YAML with PyYAML or built-in fallback parser
|
||||
- **ISO download** -- fetches latest IncusOS from the CDN by channel
|
||||
(stable/testing)
|
||||
- **Per-VM seed generation** -- calls `incusos-seed --format fat` for each VM
|
||||
- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
|
||||
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
|
||||
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
|
||||
VirtIO-SCSI, etc.)
|
||||
- **Automated install monitoring** -- state-machine polling with 10-minute
|
||||
timeout
|
||||
- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
|
||||
to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
|
||||
detaches install media (ide2 + ide3), and boots from disk
|
||||
- **Cleanup** -- safe teardown with managed-VM marker checks
|
||||
|
||||
### Cluster Formation (Manual Step)
|
||||
|
|
@ -109,7 +110,7 @@ For a home lab, `/` (root) is simpler and perfectly acceptable.
|
|||
bash --version # Need 4+
|
||||
python3 --version # For YAML parsing
|
||||
which jq curl ssh scp # All needed for SSH method
|
||||
which mcopy mkfs.fat # Needed for seed FAT images (mtools + dosfstools)
|
||||
which genisoimage # Needed for seed ISO images (used by incusos-proxmox)
|
||||
|
||||
# Verify the Incus client has generated a keypair
|
||||
ls ~/.config/incus/client.crt ~/.config/incus/client.key
|
||||
|
|
@ -184,7 +185,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
|||
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
|
||||
4. **Upload** -- SCPs ISO + seed to Proxmox
|
||||
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
|
||||
6. **Install** -- starts VM, monitors install completion (~3-5 min)
|
||||
6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)
|
||||
|
||||
**Simultaneously on the Proxmox web UI:**
|
||||
|
||||
|
|
@ -200,8 +201,8 @@ Using VMID 900+ avoids clashing with existing VMs.
|
|||
Once the script reports success with an IP address:
|
||||
|
||||
```bash
|
||||
# The cert was injected via seed, so it's already trusted
|
||||
incus remote add incus-test https://<REPORTED_IP>:8443 --accept-certificate
|
||||
# The cert was injected via seed, so no auth prompt appears
|
||||
incus remote add incus-test <REPORTED_IP> --accept-certificate
|
||||
|
||||
# Test connectivity
|
||||
incus remote list
|
||||
|
|
@ -212,6 +213,10 @@ incus storage list incus-test:
|
|||
incus network list incus-test:
|
||||
```
|
||||
|
||||
**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
|
||||
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
||||
check the Proxmox console — stale ARP entries can be misleading.
|
||||
|
||||
### Step 6: Clean Up the Test VM
|
||||
|
||||
```bash
|
||||
|
|
@ -292,7 +297,10 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
|||
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
|
||||
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
|
||||
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
|
||||
| No IP reported | Guest agent not ready | Wait a minute, check Proxmox UI for IP |
|
||||
| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
|
||||
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
|
||||
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
|
||||
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
|
||||
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
|
||||
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
|
||||
| `mcopy` not found | mtools not installed | Install `mtools` package |
|
||||
| `mkfs.fat` not found | dosfstools not installed | Install `dosfstools` package |
|
||||
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
|
||||
|
|
|
|||
|
|
@ -13,11 +13,10 @@
|
|||
# --- incus.yaml ---
|
||||
apply_defaults: false
|
||||
preseed:
|
||||
server:
|
||||
certificates:
|
||||
- name: "my-workstation"
|
||||
type: "client"
|
||||
certificate: |
|
||||
certificate: |-
|
||||
-----BEGIN CERTIFICATE-----
|
||||
<paste your certificate from: incus remote get-client-certificate>
|
||||
-----END CERTIFICATE-----
|
||||
|
|
|
|||
|
|
@ -28,11 +28,10 @@
|
|||
# --- incus.yaml ---
|
||||
apply_defaults: true
|
||||
preseed:
|
||||
server:
|
||||
certificates:
|
||||
- name: "my-workstation"
|
||||
type: "client"
|
||||
certificate: |
|
||||
certificate: |-
|
||||
-----BEGIN CERTIFICATE-----
|
||||
<paste your certificate from: incus remote get-client-certificate>
|
||||
-----END CERTIFICATE-----
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs # VM disk storage (default: local-lvm)
|
||||
|
||||
defaults:
|
||||
start_vmid: 900 # High range to avoid collisions
|
||||
|
||||
vms:
|
||||
- name: incus-test
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 64
|
||||
|
||||
|
|
@ -423,9 +423,20 @@ check_dependencies() {
|
|||
# ---------------------------------------------------------------------------
|
||||
|
||||
WORKDIR=""
|
||||
CACHEDIR=""
|
||||
|
||||
setup_workdir() {
|
||||
WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-XXXXXX")
|
||||
# Use persistent storage for the work directory -- the ISO download +
|
||||
# decompression can exceed 4 GiB, which overflows tmpfs on most systems.
|
||||
# Honour TMPDIR if explicitly set; otherwise use ~/.cache on real disk.
|
||||
local base="${TMPDIR:-${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}}"
|
||||
mkdir -p "$base"
|
||||
|
||||
# Persistent cache for ISO downloads (survives across runs)
|
||||
CACHEDIR="${base}/iso-cache"
|
||||
mkdir -p "$CACHEDIR"
|
||||
|
||||
WORKDIR=$(mktemp -d "${base}/${SCRIPT_NAME}-XXXXXX")
|
||||
trap 'rm -rf "$WORKDIR"' EXIT
|
||||
}
|
||||
|
||||
|
|
@ -1017,10 +1028,13 @@ pve_run() {
|
|||
}
|
||||
|
||||
# Execute a Proxmox API call (API method)
|
||||
# Usage: pve_api METHOD ENDPOINT [DATA] [json]
|
||||
# Pass "json" as 4th arg to send data as application/json instead of form-encoded.
|
||||
pve_api() {
|
||||
local method="$1"
|
||||
local endpoint="$2"
|
||||
local data="${3:-}"
|
||||
local content_type="${4:-form}"
|
||||
|
||||
local url="https://${PVE_HOST}:8006/api2/json${endpoint}"
|
||||
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
|
||||
|
|
@ -1034,6 +1048,9 @@ pve_api() {
|
|||
fi
|
||||
|
||||
local curl_args=(-fsSk -H "$auth")
|
||||
if [[ "$content_type" == "json" ]]; then
|
||||
curl_args+=(-H "Content-Type: application/json")
|
||||
fi
|
||||
|
||||
case "$method" in
|
||||
GET)
|
||||
|
|
@ -1082,13 +1099,48 @@ pve_upload() {
|
|||
"$local_path" "${PVE_SSH_USER}@${PVE_HOST}:${iso_dir}/${remote_filename}"
|
||||
;;
|
||||
api)
|
||||
curl -fsSk \
|
||||
local upload_response upload_rc
|
||||
upload_response=$(curl -sSk \
|
||||
-H "Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" \
|
||||
-X POST \
|
||||
-F "content=iso" \
|
||||
-F "filename=${remote_filename}" \
|
||||
-F "file=@${local_path}" \
|
||||
"https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload"
|
||||
-F "filename=@${local_path};filename=${remote_filename}" \
|
||||
"https://${PVE_HOST}:8006/api2/json/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/upload" \
|
||||
-w "\n%{http_code}" 2>&1) || true
|
||||
local http_code
|
||||
http_code=$(echo "$upload_response" | tail -1)
|
||||
local body
|
||||
body=$(echo "$upload_response" | sed '$d')
|
||||
|
||||
if [[ "$http_code" -ge 200 ]] && [[ "$http_code" -lt 300 ]] 2>/dev/null; then
|
||||
return 0
|
||||
else
|
||||
error "Upload failed (HTTP ${http_code})"
|
||||
# Extract error message from JSON response
|
||||
local api_error
|
||||
api_error=$(echo "$body" | python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
data = json.load(sys.stdin)
|
||||
errors = data.get('errors', {})
|
||||
if errors:
|
||||
for k, v in errors.items():
|
||||
print(f' {k}: {v}')
|
||||
elif data.get('message'):
|
||||
print(f' {data[\"message\"]}')
|
||||
else:
|
||||
print(json.dumps(data, indent=2))
|
||||
except:
|
||||
print(sys.stdin.read() if hasattr(sys.stdin, 'read') else '')
|
||||
" 2>/dev/null) || api_error="$body"
|
||||
if [[ -n "$api_error" ]]; then
|
||||
error "Proxmox API response:"
|
||||
echo "$api_error" | while IFS= read -r line; do
|
||||
error " ${line}"
|
||||
done
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
|
@ -1178,33 +1230,34 @@ pve_create_vm() {
|
|||
pve_run "$cmd"
|
||||
;;
|
||||
api)
|
||||
# URL-encode the description for the API
|
||||
local url_encoded_desc
|
||||
url_encoded_desc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${description}'))" 2>/dev/null)
|
||||
|
||||
# Build API parameters
|
||||
local params=""
|
||||
params+="vmid=${vmid}"
|
||||
params+="&name=${name}"
|
||||
params+="&description=${url_encoded_desc}"
|
||||
params+="&ostype=l26"
|
||||
params+="&bios=ovmf"
|
||||
params+="&machine=q35"
|
||||
params+="&efidisk0=${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0"
|
||||
params+="&tpmstate0=${PVE_STORAGE}:1,version=v2.0"
|
||||
params+="&cpu=host"
|
||||
params+="&cores=${cores}"
|
||||
params+="&memory=${memory}"
|
||||
params+="&balloon=0"
|
||||
params+="&scsihw=virtio-scsi-pci"
|
||||
params+="&scsi0=${PVE_STORAGE}:${disk_gib},iothread=1"
|
||||
params+="&ide2=${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom"
|
||||
params+="&ide3=${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom"
|
||||
params+="&net0=virtio,bridge=${PVE_BRIDGE}"
|
||||
params+="&boot=order%3Dide2%3Bscsi0"
|
||||
params+="&agent=1"
|
||||
|
||||
pve_api POST "/nodes/${PVE_NODE}/qemu" "$params"
|
||||
# Build JSON payload -- the Proxmox API requires JSON for VM
|
||||
# creation because comma-separated disk/net specs are misinterpreted
|
||||
# when sent as form-encoded data.
|
||||
local json_payload
|
||||
json_payload=$(python3 -c "
|
||||
import json
|
||||
print(json.dumps({
|
||||
'vmid': ${vmid},
|
||||
'name': '${name}',
|
||||
'description': '''${description}''',
|
||||
'ostype': 'l26',
|
||||
'bios': 'ovmf',
|
||||
'machine': 'q35',
|
||||
'efidisk0': '${PVE_STORAGE}:0,efitype=4m,pre-enrolled-keys=0',
|
||||
'tpmstate0': '${PVE_STORAGE}:1,version=v2.0',
|
||||
'cpu': 'host',
|
||||
'cores': ${cores},
|
||||
'memory': ${memory},
|
||||
'balloon': 0,
|
||||
'scsihw': 'virtio-scsi-pci',
|
||||
'scsi0': '${PVE_STORAGE}:${disk_gib},iothread=1',
|
||||
'ide2': '${PVE_ISO_STORAGE}:iso/${iso_filename},media=cdrom',
|
||||
'ide3': '${PVE_ISO_STORAGE}:iso/${seed_filename},media=cdrom',
|
||||
'net0': 'virtio,bridge=${PVE_BRIDGE}',
|
||||
'boot': 'order=ide2;scsi0',
|
||||
'agent': 1,
|
||||
}))")
|
||||
pve_api POST "/nodes/${PVE_NODE}/qemu" "$json_payload" json >/dev/null
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
|
@ -1215,7 +1268,7 @@ pve_start_vm() {
|
|||
|
||||
case "$PVE_METHOD" in
|
||||
ssh) pve_run "qm start ${vmid}" ;;
|
||||
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" ;;
|
||||
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/start" >/dev/null ;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
|
@ -1225,7 +1278,7 @@ pve_stop_vm() {
|
|||
|
||||
case "$PVE_METHOD" in
|
||||
ssh) pve_run "qm stop ${vmid}" ;;
|
||||
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" ;;
|
||||
api) pve_api POST "/nodes/${PVE_NODE}/qemu/${vmid}/status/stop" >/dev/null ;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
|
@ -1235,7 +1288,7 @@ pve_destroy_vm() {
|
|||
|
||||
case "$PVE_METHOD" in
|
||||
ssh) pve_run "qm destroy ${vmid} --purge" ;;
|
||||
api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" ;;
|
||||
api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
|
@ -1248,10 +1301,35 @@ pve_set_vm() {
|
|||
case "$PVE_METHOD" in
|
||||
ssh) pve_run "qm set ${vmid} ${opts}" ;;
|
||||
api)
|
||||
# Convert CLI-style opts to API params
|
||||
local params
|
||||
params=$(echo "$opts" | sed 's/--//g; s/ /\&/g; s/\&\([a-z]\)/\&\1/g')
|
||||
pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$params"
|
||||
# Convert CLI-style opts (--key value) to JSON.
|
||||
# JSON avoids form-encoding ambiguity with values containing = and ;
|
||||
# Duplicate keys (e.g. --delete ide2 --delete ide3) are merged with commas.
|
||||
# Shell escapes (e.g. \;) are stripped since the API doesn't need them.
|
||||
local json_payload
|
||||
json_payload=$(echo "$opts" | python3 -c "
|
||||
import json, sys
|
||||
tokens = sys.stdin.read().split()
|
||||
obj = {}
|
||||
i = 0
|
||||
while i < len(tokens):
|
||||
t = tokens[i]
|
||||
if t.startswith('--'):
|
||||
key = t.lstrip('-')
|
||||
if i + 1 < len(tokens) and not tokens[i+1].startswith('--'):
|
||||
val = tokens[i+1].replace(chr(92), '')
|
||||
i += 2
|
||||
else:
|
||||
val = ''
|
||||
i += 1
|
||||
if key in obj and obj[key]:
|
||||
obj[key] = obj[key] + ',' + val
|
||||
else:
|
||||
obj[key] = val
|
||||
else:
|
||||
i += 1
|
||||
print(json.dumps(obj))
|
||||
" 2>/dev/null)
|
||||
pve_api PUT "/nodes/${PVE_NODE}/qemu/${vmid}/config" "$json_payload" json >/dev/null
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
|
@ -1271,6 +1349,42 @@ pve_vm_status() {
|
|||
esac
|
||||
}
|
||||
|
||||
# Get VM uptime (seconds); returns 0 if stopped or unknown
|
||||
pve_vm_uptime() {
|
||||
local vmid="$1"
|
||||
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
pve_run "qm status ${vmid} --verbose" | awk '/^uptime:/{print $2}'
|
||||
;;
|
||||
api)
|
||||
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \
|
||||
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('uptime',0))" 2>/dev/null || echo "0"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Get cumulative write bytes for scsi0 (from QEMU blockstat).
|
||||
# Returns 0 if unavailable. Only works with api method.
|
||||
pve_vm_disk_writes() {
|
||||
local vmid="$1"
|
||||
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
echo "0"
|
||||
;;
|
||||
api)
|
||||
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/status/current" | \
|
||||
python3 -c "
|
||||
import json,sys
|
||||
d = json.load(sys.stdin).get('data',{})
|
||||
bs = d.get('blockstat',{}).get('scsi0',{})
|
||||
print(bs.get('wr_bytes',0))
|
||||
" 2>/dev/null || echo "0"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Check if a VMID exists
|
||||
pve_vm_exists() {
|
||||
local vmid="$1"
|
||||
|
|
@ -1293,9 +1407,11 @@ pve_vm_exists() {
|
|||
pve_vm_ip() {
|
||||
local vmid="$1"
|
||||
|
||||
# Method 1: Try QEMU guest agent (works if agent is installed)
|
||||
local ip=""
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \
|
||||
ip=$(pve_run "qm guest cmd ${vmid} network-get-interfaces 2>/dev/null" | \
|
||||
python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
|
|
@ -1309,10 +1425,10 @@ try:
|
|||
sys.exit(0)
|
||||
except:
|
||||
pass
|
||||
" 2>/dev/null || true
|
||||
" 2>/dev/null) || true
|
||||
;;
|
||||
api)
|
||||
pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \
|
||||
ip=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/agent/network-get-interfaces" 2>/dev/null | \
|
||||
python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
|
|
@ -1326,9 +1442,69 @@ try:
|
|||
sys.exit(0)
|
||||
except:
|
||||
pass
|
||||
" 2>/dev/null || true
|
||||
" 2>/dev/null) || true
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ -n "$ip" ]]; then
|
||||
echo "$ip"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Method 2: ARP-based lookup using VM's MAC address.
|
||||
# Works without guest agent (e.g. IncusOS which is immutable).
|
||||
# Get MAC from VM config, flush stale ARP, ping sweep, look up.
|
||||
local mac=""
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
mac=$(pve_run "qm config ${vmid}" 2>/dev/null | \
|
||||
awk -F'=' '/^net0:/{print $1}' | grep -oiE '[0-9a-f:]{17}') || true
|
||||
;;
|
||||
api)
|
||||
mac=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null | \
|
||||
python3 -c "
|
||||
import json, sys, re
|
||||
try:
|
||||
cfg = json.load(sys.stdin).get('data', {})
|
||||
net0 = cfg.get('net0', '')
|
||||
m = re.search(r'([0-9A-Fa-f:]{17})', net0)
|
||||
if m:
|
||||
print(m.group(1).lower())
|
||||
except:
|
||||
pass
|
||||
" 2>/dev/null) || true
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ -z "$mac" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Flush stale ARP entries for this MAC, ping broadcast to populate, then look up
|
||||
ip neigh flush nud stale 2>/dev/null || true
|
||||
local bridge_subnet
|
||||
bridge_subnet=$(ip -4 addr show | awk '/inet /{print $2}' | head -1)
|
||||
if [[ -n "$bridge_subnet" ]]; then
|
||||
local broadcast
|
||||
broadcast=$(python3 -c "import ipaddress; print(ipaddress.ip_interface('${bridge_subnet}').network.broadcast_address)" 2>/dev/null) || true
|
||||
if [[ -n "$broadcast" ]]; then
|
||||
ping -c 2 -W 1 -b "$broadcast" &>/dev/null || true
|
||||
sleep 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Look up MAC in ARP table (only REACHABLE or DELAY, not STALE)
|
||||
ip=$(ip neigh show | awk -v mac="$mac" 'tolower($0) ~ tolower(mac) && (/REACHABLE/ || /DELAY/ || /STALE/) {print $1; exit}') || true
|
||||
|
||||
if [[ -n "$ip" ]]; then
|
||||
# Verify it's actually reachable
|
||||
if ping -c 1 -W 1 "$ip" &>/dev/null; then
|
||||
echo "$ip"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1416,9 +1592,10 @@ preflight_checks() {
|
|||
fi
|
||||
;;
|
||||
api)
|
||||
local storage_json
|
||||
storage_json=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null) || storage_json=""
|
||||
local storage_list
|
||||
storage_list=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null | \
|
||||
python3 -c "
|
||||
storage_list=$(echo "$storage_json" | python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
for s in data.get('data', []):
|
||||
|
|
@ -1435,6 +1612,22 @@ for s in data.get('data', []):
|
|||
error "Available pools: $(echo "$storage_list" | tr '\n' ' ')"
|
||||
error "Fix: update 'proxmox.iso_storage' in your config file"
|
||||
errors=$((errors + 1))
|
||||
else
|
||||
# Check that ISO storage accepts 'iso' content type
|
||||
local iso_content
|
||||
iso_content=$(echo "$storage_json" | python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
for s in data.get('data', []):
|
||||
if s.get('storage') == '${PVE_ISO_STORAGE}':
|
||||
print(s.get('content', ''))
|
||||
break
|
||||
" 2>/dev/null) || iso_content=""
|
||||
if [[ -n "$iso_content" ]] && ! echo "$iso_content" | grep -q "iso"; then
|
||||
error "Storage '${PVE_ISO_STORAGE}' does not accept ISO content (content types: ${iso_content})"
|
||||
error "Fix: enable 'iso' content on '${PVE_ISO_STORAGE}' in Proxmox, or use a different iso_storage"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
|
@ -1475,15 +1668,10 @@ for s in data.get('data', []):
|
|||
fi
|
||||
fi
|
||||
|
||||
# 4. FAT tools for SEED_DATA generation
|
||||
if ! command -v mcopy &>/dev/null; then
|
||||
error "mcopy (mtools) is required for SEED_DATA generation but not found"
|
||||
error "Install mtools for your platform (e.g. mtools package)"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
if ! command -v mkfs.fat &>/dev/null; then
|
||||
error "mkfs.fat (dosfstools) is required for SEED_DATA generation but not found"
|
||||
error "Install dosfstools for your platform (e.g. dosfstools package)"
|
||||
# 4. genisoimage for SEED_DATA ISO 9660 generation
|
||||
if ! command -v genisoimage &>/dev/null; then
|
||||
error "genisoimage is required for SEED_DATA generation but not found"
|
||||
error "Install with: sudo apt install genisoimage"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
|
||||
|
|
@ -1506,12 +1694,6 @@ for s in data.get('data', []):
|
|||
check_connectivity() {
|
||||
step "Checking Proxmox connectivity (${PVE_METHOD})"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
detail "[dry-run] Would check connectivity to ${PVE_HOST}"
|
||||
success "Connectivity check skipped (dry-run)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
case "$PVE_METHOD" in
|
||||
ssh)
|
||||
if ! ssh -o BatchMode=yes -o ConnectTimeout=10 \
|
||||
|
|
@ -1529,8 +1711,10 @@ check_connectivity() {
|
|||
success "Connected to ${PVE_HOST} (Proxmox VE ${pve_version})"
|
||||
;;
|
||||
api)
|
||||
local url="https://${PVE_HOST}:8006/api2/json/version"
|
||||
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
|
||||
local response
|
||||
if ! response=$(pve_api GET "/version" 2>/dev/null); then
|
||||
if ! response=$(curl -fsSk -H "$auth" "$url" 2>/dev/null); then
|
||||
error "Cannot connect to Proxmox API at https://${PVE_HOST}:8006"
|
||||
error "Check that:"
|
||||
error " 1. The host is reachable"
|
||||
|
|
@ -1750,12 +1934,23 @@ if versions:
|
|||
download_iso() {
|
||||
local version="$1"
|
||||
local output_path="$2"
|
||||
local iso_filename
|
||||
iso_filename=$(basename "$output_path")
|
||||
|
||||
# Check persistent cache first
|
||||
local cached_iso="${CACHEDIR}/${iso_filename}"
|
||||
if [[ -f "$cached_iso" ]]; then
|
||||
success "ISO found in local cache: ${cached_iso}"
|
||||
cp "$cached_iso" "$output_path"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local url="${CDN_BASE}/${version}/x86_64/IncusOS_${version}.iso.gz"
|
||||
local gz_path="${output_path}.gz"
|
||||
local gz_path="${cached_iso}.gz"
|
||||
|
||||
step "Downloading IncusOS ISO"
|
||||
detail "URL: ${url}"
|
||||
detail "Cache directory: ${CACHEDIR}"
|
||||
|
||||
if ! curl -fL --progress-bar "$url" -o "$gz_path"; then
|
||||
error "Failed to download ISO from ${url}"
|
||||
|
|
@ -1765,12 +1960,19 @@ download_iso() {
|
|||
|
||||
step "Decompressing ISO"
|
||||
if ! gzip -d "$gz_path"; then
|
||||
error "Failed to decompress ISO"
|
||||
local avail
|
||||
avail=$(df -h --output=avail "$(dirname "$gz_path")" 2>/dev/null | tail -1 | tr -d ' ') || avail="unknown"
|
||||
error "Failed to decompress ISO (${avail} available on disk)"
|
||||
error "The decompressed ISO requires several GiB of disk space"
|
||||
error "Free up space, or set TMPDIR to a larger filesystem:"
|
||||
error " TMPDIR=/path/with/space ${SCRIPT_NAME} ${CONFIG_FILE}"
|
||||
rm -f "$gz_path"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "ISO ready: ${output_path}"
|
||||
# Copy from cache to workdir
|
||||
cp "$cached_iso" "$output_path"
|
||||
success "ISO ready: ${output_path} (cached for future runs)"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1904,16 +2106,15 @@ phase_prepare() {
|
|||
|
||||
local app="${VM_APPS[$i]}"
|
||||
local apply_defaults="${VM_APPLY_DEFAULTS[$i]}"
|
||||
local seed_output="${WORKDIR}/seed-${name}.img"
|
||||
local seed_output="${WORKDIR}/seed-${name}.iso"
|
||||
|
||||
detail "Generating seed for '${name}' (${app}, defaults=${apply_defaults})"
|
||||
|
||||
# Build incusos-seed command
|
||||
local seed_cmd=("$SEED_TOOL")
|
||||
seed_cmd+=(--format fat)
|
||||
seed_cmd+=(--format iso)
|
||||
seed_cmd+=(--app "$app")
|
||||
seed_cmd+=(--output "$seed_output")
|
||||
seed_cmd+=(--disk-target "scsi-*")
|
||||
seed_cmd+=(--force-install)
|
||||
seed_cmd+=(--force-reboot)
|
||||
seed_cmd+=(--hostname "$name")
|
||||
|
|
@ -1980,7 +2181,7 @@ phase_upload() {
|
|||
continue
|
||||
fi
|
||||
|
||||
local seed_file="seed-${name}.img"
|
||||
local seed_file="seed-${name}.iso"
|
||||
local seed_path="${WORKDIR}/${seed_file}"
|
||||
|
||||
if [[ -f "$seed_path" ]] || [[ "$DRY_RUN" == true ]]; then
|
||||
|
|
@ -2020,7 +2221,7 @@ phase_create() {
|
|||
local cores="${VM_CORES[$i]}"
|
||||
local memory="${VM_MEMORY[$i]}"
|
||||
local disk="${VM_DISK[$i]}"
|
||||
local seed_file="seed-${name}.img"
|
||||
local seed_file="seed-${name}.iso"
|
||||
|
||||
step "Creating VM '${name}' (VMID ${vmid})"
|
||||
|
||||
|
|
@ -2093,27 +2294,30 @@ phase_install() {
|
|||
fi
|
||||
success "VM '${name}' started"
|
||||
|
||||
# 2. Immediately swap boot order (takes effect on next reboot)
|
||||
detail "Swapping boot order to disk-first"
|
||||
if ! pve_set_vm "$vmid" "--boot order=scsi0\\;ide2"; then
|
||||
warn "Failed to swap boot order for '${name}' -- manual intervention may be needed"
|
||||
fi
|
||||
|
||||
# 3. Wait for installation + reboot cycle
|
||||
# 2. Wait for installation to complete by monitoring disk writes
|
||||
#
|
||||
# Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot
|
||||
# (force_reboot in seed). A guest-level reboot does NOT reset the QEMU
|
||||
# VM uptime, so we cannot detect reboot that way. Instead we monitor
|
||||
# scsi0 write bytes via blockstat:
|
||||
# - During install: wr_bytes increases steadily
|
||||
# - After install + reboot: wr_bytes stops (VM is idle at post-reboot
|
||||
# screen or "install media detected" error)
|
||||
#
|
||||
# When wr_bytes has been stable for STABLE_COUNT consecutive polls,
|
||||
# installation is complete. We then stop the VM, remove install media,
|
||||
# and start cleanly from disk.
|
||||
step "Waiting for '${name}' to complete installation"
|
||||
detail "IncusOS installs from ISO, then reboots to disk automatically"
|
||||
detail "Polling every 15s (timeout: 10 minutes)"
|
||||
detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
|
||||
|
||||
local timeout=600
|
||||
local interval=15
|
||||
local interval=5
|
||||
local elapsed=0
|
||||
local booted=false
|
||||
|
||||
# Wait for the VM to go through: running -> (install) -> stopped -> running
|
||||
# The force_reboot in seed causes an automatic reboot after install.
|
||||
# We detect the reboot by looking for the VM to be "running" after
|
||||
# having been "stopped" (or we just wait for it to stabilize).
|
||||
local saw_stop=false
|
||||
local install_done=false
|
||||
local prev_wr_bytes=0
|
||||
local writes_started=false
|
||||
local stable_count=0
|
||||
local STABLE_THRESHOLD=3 # 3 consecutive polls with no change = 15s idle
|
||||
|
||||
while [[ $elapsed -lt $timeout ]]; do
|
||||
sleep "$interval"
|
||||
|
|
@ -2124,28 +2328,35 @@ phase_install() {
|
|||
|
||||
case "$status" in
|
||||
stopped)
|
||||
if [[ "$saw_stop" == false ]]; then
|
||||
detail "VM stopped (installation complete, awaiting reboot) [${elapsed}s]"
|
||||
saw_stop=true
|
||||
# The VM should auto-reboot via force_reboot, but if it
|
||||
# doesn't after 30s, start it manually
|
||||
sleep 30
|
||||
elapsed=$((elapsed + 30))
|
||||
local recheck
|
||||
recheck=$(pve_vm_status "$vmid") || recheck="unknown"
|
||||
if [[ "$recheck" == "stopped" ]]; then
|
||||
detail "VM did not auto-reboot, starting manually"
|
||||
pve_start_vm "$vmid" || true
|
||||
fi
|
||||
fi
|
||||
detail "VM stopped [${elapsed}s]"
|
||||
install_done=true
|
||||
break
|
||||
;;
|
||||
running)
|
||||
if [[ "$saw_stop" == true ]] || [[ $elapsed -ge 120 ]]; then
|
||||
# VM has been running long enough post-reboot -- likely booted from disk
|
||||
booted=true
|
||||
local wr_bytes
|
||||
wr_bytes=$(pve_vm_disk_writes "$vmid") || wr_bytes=0
|
||||
|
||||
if [[ $wr_bytes -gt $prev_wr_bytes ]]; then
|
||||
# Disk writes are happening -- install in progress
|
||||
writes_started=true
|
||||
stable_count=0
|
||||
local wr_mb=$(( wr_bytes / 1048576 ))
|
||||
# Show progress every 15s
|
||||
if [[ $((elapsed % 15)) -eq 0 ]]; then
|
||||
detail "Installing... (${wr_mb} MiB written) [${elapsed}s]"
|
||||
fi
|
||||
elif [[ "$writes_started" == true ]]; then
|
||||
# Writes have stopped after being active
|
||||
stable_count=$((stable_count + 1))
|
||||
if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then
|
||||
local wr_mb=$(( wr_bytes / 1048576 ))
|
||||
detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
|
||||
install_done=true
|
||||
break
|
||||
fi
|
||||
detail "VM running (installing...) [${elapsed}s]"
|
||||
fi
|
||||
|
||||
prev_wr_bytes=$wr_bytes
|
||||
;;
|
||||
*)
|
||||
detail "VM status: ${status} [${elapsed}s]"
|
||||
|
|
@ -2153,20 +2364,39 @@ phase_install() {
|
|||
esac
|
||||
done
|
||||
|
||||
if [[ "$booted" != true ]]; then
|
||||
if [[ "$install_done" != true ]]; then
|
||||
warn "VM '${name}' did not complete installation within ${timeout}s"
|
||||
warn "Check the VM console on Proxmox for details"
|
||||
i=$((i + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
# 4. Give it a moment to fully initialize
|
||||
detail "Waiting for services to start..."
|
||||
sleep 15
|
||||
# 3. Stop the VM (it's either at "remove install media" prompt or
|
||||
# post-reboot "install media detected" error -- either way idle).
|
||||
local cur_status
|
||||
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
|
||||
if [[ "$cur_status" == "running" ]]; then
|
||||
detail "Stopping VM to remove install media..."
|
||||
pve_stop_vm "$vmid" || true
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
# 5. Remove CD-ROM drives
|
||||
step "Cleaning up '${name}'"
|
||||
pve_set_vm "$vmid" "--delete ide2 --delete ide3" || true
|
||||
# 4. Remove install media and set clean boot order
|
||||
step "Removing install media from '${name}'"
|
||||
pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true
|
||||
success "Install media detached, boot order set to disk-only"
|
||||
|
||||
# 5. Start VM from disk
|
||||
step "Starting '${name}' from disk"
|
||||
if ! pve_start_vm "$vmid"; then
|
||||
error "Failed to start VM '${name}' from disk"
|
||||
i=$((i + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
# 5. Wait for services to come up
|
||||
detail "Waiting for services to start..."
|
||||
sleep 30
|
||||
|
||||
# 6. Try to detect IP
|
||||
local ip
|
||||
|
|
|
|||
|
|
@ -76,8 +76,9 @@ ${BOLD}OPTIONS${RESET}
|
|||
${BOLD}--auto-reboot${RESET} Enable automatic reboot for updates
|
||||
|
||||
${BOLD}Output options:${RESET}
|
||||
${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}fat${RESET} (default: tar)
|
||||
${BOLD}-f, --format${RESET} FORMAT Output format: ${CYAN}tar${RESET}, ${CYAN}iso${RESET}, ${CYAN}fat${RESET} (default: tar)
|
||||
tar: for use with flasher-tool --seed
|
||||
iso: ISO 9660 image with SEED_DATA label for CD-ROM
|
||||
fat: FAT image with SEED_DATA label for external media
|
||||
${BOLD}--fat-size${RESET} SIZE FAT image size in MiB (default: 10)
|
||||
|
||||
|
|
@ -97,6 +98,9 @@ ${BOLD}EXAMPLES${RESET}
|
|||
# Cluster node (no defaults) with forced install on virtio disk
|
||||
${SCRIPT_NAME} --disk-target 'virtio-*' --force-install --force-reboot
|
||||
|
||||
# Create an ISO 9660 image for CD-ROM seed delivery (e.g. Proxmox VMs)
|
||||
${SCRIPT_NAME} --format iso --defaults --output seed.iso
|
||||
|
||||
# Create a FAT image for SEED_DATA partition
|
||||
${SCRIPT_NAME} --format fat --defaults --output seed.img
|
||||
|
||||
|
|
@ -257,10 +261,10 @@ validate_args() {
|
|||
esac
|
||||
|
||||
case "$FORMAT" in
|
||||
tar|fat) ;;
|
||||
tar|iso|fat) ;;
|
||||
*)
|
||||
error "Invalid output format: ${FORMAT}"
|
||||
error "Supported: tar, fat"
|
||||
error "Supported: tar, iso, fat"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
|
@ -284,8 +288,21 @@ validate_args() {
|
|||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$FORMAT" == "iso" ]]; then
|
||||
if ! command -v genisoimage &>/dev/null; then
|
||||
error "ISO 9660 image creation requires genisoimage but it was not found"
|
||||
error "Install with: sudo apt install genisoimage"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$FORMAT" == "fat" ]]; then
|
||||
if ! command -v mcopy &>/dev/null && ! command -v mkfs.fat &>/dev/null; then
|
||||
# mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users
|
||||
local has_mkfs_fat=false
|
||||
if command -v mkfs.fat &>/dev/null || [[ -x /usr/sbin/mkfs.fat ]] || [[ -x /sbin/mkfs.fat ]]; then
|
||||
has_mkfs_fat=true
|
||||
fi
|
||||
if ! command -v mcopy &>/dev/null && [[ "$has_mkfs_fat" != true ]]; then
|
||||
error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)"
|
||||
error "Install with: sudo apt install mtools dosfstools"
|
||||
exit 1
|
||||
|
|
@ -296,6 +313,7 @@ validate_args() {
|
|||
if [[ -z "$OUTPUT" ]]; then
|
||||
case "$FORMAT" in
|
||||
tar) OUTPUT="incusos-seed.tar" ;;
|
||||
iso) OUTPUT="incusos-seed.iso" ;;
|
||||
fat) OUTPUT="incusos-seed.img" ;;
|
||||
esac
|
||||
fi
|
||||
|
|
@ -464,11 +482,10 @@ generate_incus_yaml() {
|
|||
|
||||
if [[ -n "$CERT_CONTENT" ]]; then
|
||||
echo "preseed:"
|
||||
echo " server:"
|
||||
echo " certificates:"
|
||||
echo " - name: \"auto-injected-client\""
|
||||
echo " type: \"client\""
|
||||
echo " certificate: |"
|
||||
echo " certificate: |-"
|
||||
# Indent each line of the certificate
|
||||
while IFS= read -r line; do
|
||||
echo " ${line}"
|
||||
|
|
@ -592,14 +609,34 @@ create_tar_output() {
|
|||
success "Seed archive created: ${OUTPUT} (${size})"
|
||||
}
|
||||
|
||||
create_iso_output() {
|
||||
step "Creating ISO 9660 image: ${OUTPUT}"
|
||||
|
||||
genisoimage -o "$OUTPUT" -V SEED_DATA -J -r "$WORKDIR"/ 2>/dev/null
|
||||
|
||||
local size
|
||||
size=$(du -h "$OUTPUT" | cut -f1)
|
||||
success "ISO 9660 seed image created: ${OUTPUT} (${size})"
|
||||
}
|
||||
|
||||
create_fat_output() {
|
||||
step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)"
|
||||
|
||||
# Create empty image
|
||||
dd if=/dev/zero of="$OUTPUT" bs=1M count="$FAT_SIZE" status=none
|
||||
|
||||
# mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users
|
||||
local mkfs_fat=""
|
||||
if command -v mkfs.fat &>/dev/null; then
|
||||
mkfs.fat -n SEED_DATA "$OUTPUT" >/dev/null
|
||||
mkfs_fat="mkfs.fat"
|
||||
elif [[ -x /usr/sbin/mkfs.fat ]]; then
|
||||
mkfs_fat="/usr/sbin/mkfs.fat"
|
||||
elif [[ -x /sbin/mkfs.fat ]]; then
|
||||
mkfs_fat="/sbin/mkfs.fat"
|
||||
fi
|
||||
|
||||
if [[ -n "$mkfs_fat" ]]; then
|
||||
"$mkfs_fat" -n SEED_DATA "$OUTPUT" >/dev/null
|
||||
|
||||
if command -v mcopy &>/dev/null; then
|
||||
# Use mtools to copy files without mounting
|
||||
|
|
@ -663,6 +700,9 @@ print_summary() {
|
|||
if [[ "$FORMAT" == "tar" ]]; then
|
||||
echo -e "${DIM} Use with flasher-tool:${RESET}"
|
||||
echo -e " flasher-tool --seed ${OUTPUT}"
|
||||
elif [[ "$FORMAT" == "iso" ]]; then
|
||||
echo -e "${DIM} Attach as CD-ROM to the VM.${RESET}"
|
||||
echo -e "${DIM} IncusOS will detect the SEED_DATA volume label at boot.${RESET}"
|
||||
elif [[ "$FORMAT" == "fat" ]]; then
|
||||
echo -e "${DIM} Attach as secondary disk/ISO to the VM.${RESET}"
|
||||
echo -e "${DIM} IncusOS will detect the SEED_DATA label at boot.${RESET}"
|
||||
|
|
@ -718,6 +758,7 @@ main() {
|
|||
else
|
||||
case "$FORMAT" in
|
||||
tar) create_tar_output ;;
|
||||
iso) create_iso_output ;;
|
||||
fat) create_fat_output ;;
|
||||
esac
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,148 @@
|
|||
# IncusOS Proxmox Deploy — Session Progress
|
||||
|
||||
_Last updated: 2026-02-19_
|
||||
|
||||
## Status: Complete — end-to-end automated deployment working
|
||||
|
||||
Full automated deployment of IncusOS to Proxmox VE via API is working.
|
||||
Tested with a single standalone Incus VM (VMID 900) — seed generated,
|
||||
ISO uploaded, VM created with correct settings, installation detected via
|
||||
disk I/O monitoring, media ejected, booted from disk, Incus API accessible,
|
||||
client certificate pre-trusted.
|
||||
|
||||
## Proxmox environment
|
||||
|
||||
| Item | Value |
|
||||
|----------------------|-------------------------------------------|
|
||||
| Host | 192.168.1.29 |
|
||||
| Node name | pve |
|
||||
| Proxmox VE version | 9.1.5 |
|
||||
| API token ID | `automation@pve!deploy` |
|
||||
| Token secret env var | `PROXMOX_TOKEN_SECRET` |
|
||||
| ISO storage | `local` (directory, supports iso content) |
|
||||
| VM disk storage | `local-zfs` (ZFS pool, images/rootdir) |
|
||||
| Test config | `incusos/examples/my-test.yaml` |
|
||||
| Test VM ID | 900 |
|
||||
|
||||
## Bugs found and fixed
|
||||
|
||||
### 1. FAT labels not detected on CD-ROM devices
|
||||
|
||||
**Symptom:** IncusOS reports "unable to begin install from read-only device
|
||||
/dev/mapper/sr0 without seed configuration"
|
||||
|
||||
**Root cause:** Linux kernel's `sr_mod` driver does NOT expose FAT filesystem
|
||||
labels for `/dev/sr*` devices. The `SEED_DATA` label is never created in
|
||||
`/dev/disk/by-label/`.
|
||||
|
||||
**Fix:** Switched from FAT to ISO 9660 format. Added `--format iso` to
|
||||
`incusos-seed` using `genisoimage -V SEED_DATA -J -r`. Changed
|
||||
`incusos-proxmox` to use `--format iso` and `.iso` extension.
|
||||
|
||||
### 2. Disk target literal matching (not glob)
|
||||
|
||||
**Symptom:** "no target device matched 'scsi-*' (detected devices:
|
||||
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0)"
|
||||
|
||||
**Root cause:** IncusOS does literal string matching on disk target IDs,
|
||||
not glob matching. `scsi-*` is not a valid device ID.
|
||||
|
||||
**Fix:** Removed `--disk-target "scsi-*"` from seed generation in
|
||||
`incusos-proxmox`. IncusOS auto-detects when there's a single disk.
|
||||
|
||||
### 3. Preseed YAML structure (Go `yaml:",inline"`)
|
||||
|
||||
**Symptom:** "yaml: unmarshal errors: line 4 field server not found in
|
||||
type api.initpreseed"
|
||||
|
||||
**Root cause:** The Incus `InitPreseed` struct has
|
||||
`Server InitLocalPreseed` with `yaml:",inline"`, which promotes the
|
||||
Server's fields (including `certificates`) to the parent level. The
|
||||
correct path is `preseed.certificates[]`, NOT `preseed.server.certificates[]`.
|
||||
|
||||
**Fix:** Updated `generate_incus_yaml()` in `incusos-seed` and all
|
||||
example files.
|
||||
|
||||
### 4. Install media detection blocking boot
|
||||
|
||||
**Symptom:** After installation and reboot, IncusOS refuses to boot with
|
||||
"install media detected" error, regardless of boot order.
|
||||
|
||||
**Root cause:** IncusOS checks for the _presence_ of install media at
|
||||
every boot. If IDE CD-ROMs are still attached, it blocks boot even when
|
||||
disk is first in boot order.
|
||||
|
||||
**Fix:** After detecting install completion, the script now: stops the VM,
|
||||
deletes ide2 and ide3 devices completely, sets boot order to `order=scsi0`,
|
||||
then starts from disk.
|
||||
|
||||
### 5. Install completion detection
|
||||
|
||||
**Symptom:** Cannot reliably detect when IncusOS finishes installing.
|
||||
Guest-level reboot does NOT reset QEMU VM uptime (only Proxmox stop/start
|
||||
does), making uptime-based detection impossible.
|
||||
|
||||
**Root cause:** `force_reboot: true` in the seed triggers a guest OS reboot,
|
||||
but QEMU treats this as a guest-initiated reset — the VM process continues,
|
||||
uptime keeps incrementing.
|
||||
|
||||
**Fix:** Poll `blockstat.scsi0.wr_bytes` via Proxmox API every 5 seconds.
|
||||
During installation, disk writes increase (~876 MiB total). After install +
|
||||
reboot, writes stop. Three consecutive stable readings (15 seconds of no
|
||||
writes) = install done. Works reliably.
|
||||
|
||||
### 6. force_reboot is required for automation
|
||||
|
||||
**Symptom:** Without `force_reboot: true`, IncusOS sits at "please remove
|
||||
installation media to complete the installation" and waits indefinitely.
|
||||
|
||||
**Root cause:** IncusOS does not auto-halt after installation. It expects
|
||||
the user to remove media and manually reboot.
|
||||
|
||||
**Fix:** Always include `force_reboot: true` in the seed for automated
|
||||
deployments.
|
||||
|
||||
### 7. IP detection without guest agent
|
||||
|
||||
**Symptom:** IncusOS is immutable with no QEMU guest agent. Standard
|
||||
Proxmox guest agent IP detection returns nothing.
|
||||
|
||||
**Fix:** ARP-based detection: extract MAC from Proxmox VM config (net0
|
||||
field), flush stale ARP entries, ping broadcast to populate ARP table,
|
||||
look up MAC in ARP output. Verify with direct ping before returning.
|
||||
|
||||
### 8. Various API-related fixes (earlier session)
|
||||
|
||||
- **ISO upload HTTP 400**: Upload field must be named `filename` (not `file`)
|
||||
- **ISO caching**: Added persistent cache at `~/.cache/incusos-proxmox/iso-cache/`
|
||||
- **VM creation HTTP 400** ("duplicate key"): Use JSON Content-Type, not form-encoded
|
||||
- **Boot order swap HTTP 400**: Convert CLI flags to JSON with Python helper
|
||||
- **API response leak**: Suppress JSON responses from mutating API calls
|
||||
|
||||
## Verified deployment flow
|
||||
|
||||
1. Seed generated as ISO 9660 (372K) with `genisoimage`
|
||||
2. IncusOS ISO reused from Proxmox cache (already uploaded)
|
||||
3. VM 900 created with all correct settings (UEFI, TPM, VirtIO-SCSI, etc.)
|
||||
4. VM started — IncusOS boots from ISO, reads SEED_DATA from ide3
|
||||
5. blockstat detects install completion at ~876 MiB written, ~50s elapsed
|
||||
6. VM stopped, ide2 + ide3 deleted, boot order set to `order=scsi0`
|
||||
7. VM started from disk — IncusOS boots successfully
|
||||
8. Port 8443 open, Incus API responding
|
||||
9. `incus remote add` works without auth prompt (certificate pre-trusted)
|
||||
10. Storage pool (`local`, ZFS) and network bridge (`incusbr0`) confirmed
|
||||
|
||||
## Existing VMs on Proxmox
|
||||
|
||||
- 100: dev-vm-beelink
|
||||
- 201: IncusOS-01
|
||||
- 202: IncusOS-02
|
||||
- 203: IncusOS-03
|
||||
- 300: IncusOS-OC
|
||||
- 900: incus-test (deployed and running)
|
||||
|
||||
## Next steps
|
||||
|
||||
- Multi-VM deployment testing (full lab with ops-center + cluster)
|
||||
- Cluster formation automation (`incus cluster add` / join tokens)
|
||||
- SSH method testing (currently only API method verified)
|
||||
Loading…
Reference in New Issue