maarten
/
ce-infra
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add Incus research, implementation plan, and comparison document 

Research and planning for an Incus-based alternative to our Docker Compose
infrastructure. Includes:
- docs/incus/research.md — Incus 6.22 capabilities, OCI support, networking,
  storage, Ansible integration, backup features
- docs/incus/implementation-plan.md — full plan to replicate current stack
  using Incus OCI containers on a separate git branch
- docs/incus/comparison-docker-vs-incus.md — detailed feature, security,
  operations, and resource comparison

Implementation deferred to a future session on a machine with Proxmox
testlab access.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
M. van der Woord 2026-03-19 22:41:13 +01:00
parent b0e0f64439
commit d2baed8cea
3 changed files with 585 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

166
docs/incus/comparison-docker-vs-incus.md Normal file
View File

@ -0,0 +1,166 @@
# Docker vs Incus — Comparison for Cloud Elves Infrastructure
## Executive Summary
Docker Compose is simpler for day-to-day operations and has a massive ecosystem. Incus offers stronger isolation, first-class backup/snapshot capabilities, and a unified platform for containers and VMs. For our current 4-person team on a single server, Docker is the pragmatic choice. Incus becomes compelling when we need VMs alongside containers, stronger tenant isolation for customer workloads, or a platform to offer as a product.
## Architecture Comparison
### How They Work
```mermaid
sequenceDiagram
participant Admin
participant Docker as Docker Compose
participant Containers as App Containers
Admin->>Docker: docker compose up
Docker->>Containers: Create all containers from YAML
Note over Docker: Declarative — single file defines entire stack
Docker-->>Admin: All services running
```
```mermaid
sequenceDiagram
participant Admin
participant Ansible
participant Incus
participant Containers as OCI Containers
Admin->>Ansible: ansible-playbook deploy.yml
Ansible->>Incus: incus launch (per service)
Incus->>Containers: Create containers individually
Ansible->>Incus: incus config device add (volumes, proxies)
Note over Ansible: Imperative — Ansible orchestrates step by step
Incus-->>Admin: All services running
```
### Feature Comparison
| Feature | Docker Compose | Incus |
|---|---|---|
| **Orchestration** | Declarative YAML — one file defines everything | Imperative CLI — each container managed individually |
| **Learning curve** | Low — widely known, massive community | Medium — less common, different mental model |
| **OCI image support** | Native (this is Docker) | Supported since v6.3 (July 2024) |
| **System containers** | Not supported | First-class — full OS with systemd |
| **Virtual machines** | Not supported | First-class — run VMs alongside containers |
| **DNS discovery** | Compose service names on network | Container names on managed bridge (dnsmasq) |
| **Port forwarding** | `ports:` in compose file | Proxy devices |
| **Volumes** | Named volumes | Storage pool volumes |
| **Config mounts** | Bind mounts | Disk devices |
| **Network isolation** | Multiple compose networks | Multiple bridges (but single bridge is simpler) |
| **Restart policy** | `restart: unless-stopped` | `boot.autostart=true` (no crash restart) |
| **Compose-like tooling** | Docker Compose (mature) | `incus-compose` (incomplete/WIP) |
| **Ecosystem** | Enormous — Docker Hub, GitHub Actions, etc. | Growing — Linux Containers community |
### Resource Overhead
| Scenario | Docker | Incus OCI | Incus System Container |
|---|---|---|---|
| Base container overhead | ~5-10MB | ~5-10MB (comparable) | ~50-100MB (full OS) |
| 9 containers (our stack) | ~100MB overhead | ~100MB overhead | ~500-900MB overhead |
| CPU overhead | Negligible | Negligible | Negligible |
| Disk per container | Layered images (efficient) | Converted images | Full rootfs (150-500MB) |
**Bottom line**: OCI containers in Incus have comparable overhead to Docker. System containers are heavier due to systemd and OS services.
## Security & Isolation
| Aspect | Docker | Incus |
|---|---|---|
| **Kernel sharing** | Shared with host (same as Incus) | Shared with host (same as Docker) |
| **Namespace isolation** | Process, network, mount, IPC, UTS, user | Same namespaces + optional AppArmor/seccomp profiles |
| **Root in container** | Often runs as root (configurable) | System containers: full user space. OCI: same as Docker |
| **Container escape risk** | Kernel vulnerability = escape | Same risk for both |
| **Multi-tenant isolation** | Not designed for it | System containers provide stronger tenant boundaries |
| **Docker socket exposure** | Common pattern (e.g., Gitea runner) | Not needed — Incus has its own API |
**Key insight**: For our use case (single team, all trusted), the security difference is negligible. Incus's stronger isolation matters when hosting untrusted workloads or providing infrastructure to customers.
## Operational Comparison
### Day-to-Day Operations
| Task | Docker | Incus |
|---|---|---|
| Deploy all services | `docker compose up -d` | `ansible-playbook deploy.yml` |
| View running services | `docker compose ps` | `incus list` |
| View logs | `docker compose logs gitea` | `incus exec gitea -- cat /var/log/...` or `incus console gitea --type=log` |
| Restart service | `docker compose restart gitea` | `incus restart gitea` |
| Update service image | Edit compose, `docker compose up -d` | `incus stop gitea && incus delete gitea` + relaunch |
| Enter container shell | `docker compose exec gitea bash` | `incus exec gitea -- bash` |
| Deploy static site | `rsync` to host (unchanged) | `rsync` to host (unchanged) |
### Backup & Restore
| Capability | Docker | Incus |
|---|---|---|
| Database dump | `docker compose exec db pg_dump` | `incus exec db -- pg_dump` |
| Volume backup | Manual: `docker run --rm -v ... tar czf` | Built-in: `incus storage volume export` |
| Full instance snapshot | Not built-in | `incus snapshot create` (instant with ZFS) |
| Portable export | Manual tarball assembly | `incus export` (single command, includes volumes) |
| Remote backup | Needs external tool (Restic, etc.) | Same — snapshots are local only |
| Restore to different host | Manual — recreate containers, import data | `incus import` (single command) |
**Incus advantage**: Backup is a first-class feature. `incus export` creates a single portable file containing the container and all its data. With ZFS, snapshots are instant and space-efficient.
### Service Updates
| Scenario | Docker | Incus |
|---|---|---|
| Bump image version | Edit compose file, `up -d` | Delete container, relaunch with new image (data in volumes survives) |
| Config change | Edit template, `up -d` (smart restart) | Edit template on host, `incus restart` |
| Rollback | Re-pull old image, `up -d` | `incus snapshot restore` (if snapshot taken pre-upgrade) |
**Docker advantage**: Image updates are more ergonomic. Incus requires deleting and recreating the container (since OCI containers can't change their image in-place).
## When to Choose Which
### Choose Docker When
- Team is familiar with Docker (faster onboarding)
- Using many third-party services distributed as Docker images
- Want declarative stack definition (single compose file)
- Simple single-node deployment
- Ecosystem matters (GitHub Actions, CI/CD, monitoring tools all expect Docker)
### Choose Incus When
- Need VMs alongside containers (e.g., testing, customer environments)
- Want first-class backup/snapshot capabilities
- Building infrastructure as a product for customers
- Need stronger multi-tenant isolation
- Want a single platform for containers + VMs + storage
- Comfortable with Ansible-driven infrastructure (no compose equivalent)
### For Cloud Elves Specifically
**Current recommendation: Stay with Docker for production.** The stack is working, tested, and everyone knows it.
**Build the Incus branch as a parallel capability** for:
- Learning and R&D
- Potential product offering (Incus-managed customer environments)
- Testing whether Incus + ZFS backup model is simpler than Docker + Restic
- Evaluating system containers for future services that need full OS
## Migration Path
Moving from Docker to Incus (if we decide to switch):
```mermaid
sequenceDiagram
participant Docker as Docker Server
participant Backup as Backup Storage
participant Incus as Incus Server
Docker->>Backup: Run backup.sh (pg_dumps + volume tarballs)
Note over Incus: Fresh Debian 13, Incus installed
Incus->>Incus: Run bootstrap.yml + deploy.yml
Backup->>Incus: Run restore.sh (import pg_dumps + volume data)
Note over Incus: All services running with restored data
```
The data formats are identical (Postgres dumps, file tarballs). The migration is just:
1. Backup from Docker server
2. Deploy empty Incus server
3. Restore data into Incus containers
No data format conversion needed.

184
docs/incus/implementation-plan.md Normal file
View File

@ -0,0 +1,184 @@
# Incus Implementation Plan
## Goal
Build an Incus-based replica of our current Docker Compose infrastructure on a separate git branch (`incus`), managed by Ansible, and test it on a Proxmox VM before considering production use.
## Approach: OCI Containers in Incus
Use Incus's native OCI container support (stable since v6.3, July 2024) to run the **exact same Docker images** we use today. The orchestration layer changes from `docker compose` to `incus` CLI commands driven by Ansible.
**Why OCI over system containers?** Services like Outline, Dex, and LLDAP are only distributed as Docker images. OCI containers give us the same images with Incus's networking and storage management. No need to install anything from source.
**Gitea Runner**: Deferred. It requires Docker socket access — a separate problem that needs its own solution (likely a system container with Docker inside).
## Architecture
### Container Mapping
Every service maps 1:1 from Docker to Incus. Container names are preserved so config files (Caddyfile, dex-config.yml) work unchanged.
| Container | Image | Volumes | Exposed Ports |
|---|---|---|---|
| `caddy` | `caddy:2` | caddy-data, caddy-config, Caddyfile mount, /opt/www mount | 80, 443, 443/udp |
| `lldap` | `lldap/lldap:stable` | lldap-data | (internal: 17170, 3890) |
| `dex` | `dexidp/dex:latest` | dex-data, dex-config mount | (internal: 5556) |
| `gitea` | `gitea/gitea:1.21` | gitea-data | 2222→22 |
| `gitea-db` | `postgres:16-alpine` | gitea-db-data | (internal: 5432) |
| `outline` | `outlinewiki/outline:1.6.0` | outline-data | (internal: 3000) |
| `outline-db` | `postgres:16-alpine` | outline-db-data | (internal: 5432) |
| `outline-redis` | `redis:7-alpine` | (none) | (internal: 6379) |
### Networking
Single managed bridge (`incusbr0`) with dnsmasq providing:
- DHCP for all containers
- DNS resolution by container name (e.g., `gitea-db` resolves from any container on the bridge)
- IPv4 NAT for outbound internet access
Port forwarding via Incus proxy devices (replaces Docker's `ports:` directive):
```
caddy: 0.0.0.0:80 → 127.0.0.1:80
0.0.0.0:443 → 127.0.0.1:443
gitea: 0.0.0.0:2222 → 127.0.0.1:22
```
### Storage
Storage pool `default` with `dir` driver for testing (switch to `zfs` for production). Named volumes created with `incus storage volume create` and attached to containers as disk devices.
### Config File Mounting
Config files templated to `/opt/ce-config/` on host, mounted into containers via disk devices:
```
/opt/ce-config/Caddyfile → caddy:/etc/caddy/Caddyfile
/opt/ce-config/dex-config.yml → dex:/etc/dex/config.yaml
/opt/www/ → caddy:/srv/www
```
## Ansible Role Structure (on `incus` branch)
```
roles/
common/ ← UNCHANGED from master
incus/ ← NEW (replaces roles/docker/)
tasks/main.yml ← Install Incus, init preseed, add OCI remote
templates/preseed.yml.j2 ← Incus init config
services/ ← REWRITTEN for Incus
tasks/
main.yml ← Orchestration: config templating, calls service files
lldap.yml ← LLDAP container + volume
dex.yml ← Dex container + volume + config mount
gitea.yml ← Gitea + Gitea-DB + volumes + proxy device
outline.yml ← Outline + Outline-DB + Redis + volumes
caddy.yml ← Caddy + volumes + config mounts + proxy devices
templates/
Caddyfile.j2 ← UNCHANGED (container names match)
dex-config.yml.j2 ← UNCHANGED (container names match)
```
## Per-Service Task Pattern
Each service file follows this idempotent pattern:
```yaml
# 1. Check if container exists
- name: Check if <service> exists
command: incus info <service>
register: exists
failed_when: false
changed_when: false
# 2. Create storage volume (idempotent — incus ignores if exists)
- name: Create <service> volume
command: incus storage volume create {{ incus_storage_pool }} <volume-name>
when: exists.rc != 0
# 3. Launch OCI container with environment variables
- name: Create <service> container
command: >
incus launch docker:<image>:<tag> <service>
--config environment.KEY1=value1
--config environment.KEY2=value2
when: exists.rc != 0
# 4. Attach storage volume
- name: Attach <volume> to <service>
command: >
incus config device add <service> <device-name> disk
pool={{ incus_storage_pool }} source=<volume-name> path=<mount-path>
when: exists.rc != 0
# 5. Attach proxy device (if externally exposed)
- name: Add <service> proxy device
command: >
incus config device add <service> <name> proxy
listen=tcp:0.0.0.0:<host-port> connect=tcp:127.0.0.1:<container-port>
when: exists.rc != 0
# 6. Wait for readiness
- name: Wait for <service>
command: incus exec <service> -- <health-check-command>
retries: 10
delay: 3
until: result.rc == 0
```
## Implementation Phases
### Phase 0 — Branch + Test Environment
1. `git checkout -b incus` from master
2. Create Debian 13 VM on Proxmox testlab
3. Update `inventory/hosts.yml` to target test VM
4. Verify: `ansible all -m ping`
### Phase 1 — Install Incus (`roles/incus/`)
1. Install `incus` package from Debian 13 repos
2. Initialize with preseed (storage pool + bridge)
3. Add Docker Hub OCI remote
4. Verify: `incus info` succeeds
### Phase 2 — Deploy Services (one at a time)
Deploy in dependency order, verifying each before moving on:
1. LLDAP → verify web UI responds
2. Dex → verify OIDC discovery endpoint
3. Gitea-DB → verify `pg_isready`
4. Gitea → verify web UI, add OIDC auth source
5. Outline-DB → verify `pg_isready`
6. Outline-Redis → verify `redis-cli ping`
7. Outline → verify web UI
8. Caddy → verify reverse proxy to all services
### Phase 3 — Playbook Integration
- Wire up `bootstrap.yml` (common + incus)
- Wire up `deploy.yml` (services)
- Full end-to-end test from clean VM
### Phase 4 — Backup/Restore
- Rewrite scripts using `incus exec` for pg_dump and `incus storage volume export` for volumes
- Test full backup + destroy + restore cycle
### Phase 5 — Documentation
- Complete comparison document
- Update CLAUDE.md on the incus branch
## Variables to Add
```yaml
# Incus configuration (add to inventory/group_vars/all.yml)
incus_storage_pool: "default"
incus_storage_driver: "dir" # "zfs" for production
incus_bridge: "incusbr0"
```
All existing variables (domain, passwords, secrets, image versions) remain unchanged.
## Key Gotchas
1. **Container names must match DNS references** — Caddyfile uses `gitea:3000`, dex-config uses `lldap:3890`. Container names must be exactly `gitea`, `lldap`, etc.
2. **No depends_on** — Ansible task ordering handles startup sequence. Use `retries/delay/until` for readiness checks.
3. **OCI image pull syntax** — `incus launch docker:lldap/lldap:stable lldap` (note: `docker:` prefix)
4. **Config file updates** — Template to host, then `incus restart <container>`. Register changed state in Ansible.
5. **Restart policy** — Set `boot.autostart=true` on all containers. No automatic restart on crash (add watchdog if needed).
6. **UFW still applies** — Host firewall from `common` role still controls which ports are reachable. Current rules (22, 80, 443, 2222) are correct.
7. **First image pull is slow** — Incus downloads and converts OCI images on first launch. Subsequent launches use cache.

235
docs/incus/research.md Normal file
View File

@ -0,0 +1,235 @@
# Incus Research Notes
Research conducted 2026-03-17. Incus development moves fast — verify against latest docs before implementing.
## Incus Version & Status
- **Latest**: Incus 6.22 (February 28, 2026)
- **LTS**: Incus 6.0 (5 years support)
- **Release cadence**: Monthly feature releases
- **Debian 13 (Trixie)**: Native packages available (`apt install incus`)
## OCI Container Support
- **Added in**: Incus 6.3 (July 2024) — 18+ months mature
- **How it works**: Pulls Docker/OCI images from registries, converts them, runs natively
- **Setup**: `incus remote add docker https://docker.io --protocol=oci`
- **Usage**: `incus launch docker:postgres:16-alpine my-postgres`
- **Dependencies**: `skopeo` and `umoci` (included in Debian packages)
- **Maturity**: Stable for production use, actively improved each release
## System Containers vs OCI Containers
| Aspect | System Containers | OCI Containers |
|---|---|---|
| Purpose | Full OS simulation | Single application |
| Init | systemd (runs until stopped) | Entrypoint process only |
| Image size | 150-500MB per container | Smaller, layered |
| RAM overhead | ~50-100MB more than OCI | Comparable to Docker |
| Use case | Legacy apps, multi-process, full sysadmin | Microservices, pre-built images |
## Networking
### Managed Bridges
- Incus creates bridges with dnsmasq for DHCP + DNS
- Containers get DNS entries for their instance name automatically
- Container-to-container by name works out of the box
- Default bridge: `incusbr0` (created during `incus admin init`)
### Port Forwarding — Proxy Devices
```bash
# Single port
incus config device add container http proxy \
listen=tcp:0.0.0.0:80 connect=tcp:127.0.0.1:80
# Port range
incus config device add container ports proxy \
listen=tcp:0.0.0.0:8080-9000 connect=tcp:127.0.0.1:8080-9000
# UDP (for HTTP/3)
incus config device add container h3 proxy \
listen=udp:0.0.0.0:443 connect=udp:127.0.0.1:443
```
### Network Forwards (alternative to proxy devices)
```bash
incus network forward create <network_name> <listen_address>
```
## Storage
### Pool Backends
| Backend | Pros | Cons |
|---|---|---|
| `dir` | Simple, no setup | Slower, no snapshots |
| `zfs` | Fast snapshots, dedup, compression | Complex setup |
| `btrfs` | CoW, snapshots | Less tooling than ZFS |
| `lvm` | Reliable, mature | Block-level only |
### Volume Operations
```bash
# Create volume
incus storage volume create <pool> <name>
# Attach to container
incus config device add <container> <device-name> disk \
pool=<pool> source=<volume-name> path=<mount-path>
# Mount host directory into container
incus config device add <container> <device-name> disk \
source=/host/path path=/container/path
```
## Ansible Integration
### Connection Plugin (community.general.incus)
- Exec commands inside containers without SSH
- Install: `ansible-galaxy collection install community.general`
- Usage: `ansible_connection: community.general.incus`
- Variables: `ansible_incus_remote`, `ansible_incus_project`, `ansible_host`
### Inventory Plugin (community.general.incus_inventory)
- Auto-discovers running Incus instances
- Dynamic inventory source
### Third-Party: kmpm.incus Collection
- Dedicated modules: `incus_instance`, `incus_network`, `incus_profile`
- Status: Beta/WIP, actively maintained
- Install: `ansible-galaxy collection install git+https://github.com/kmpm/ansible-collection-incus.git`
### Practical Approach for This Project
Use `ansible.builtin.command` with `incus` CLI directly. Simpler, more transparent, easier to debug than collections. Connection plugin useful for exec-inside-container tasks.
## Incus Initialization (Preseed)
Non-interactive setup via `incus admin init --preseed`:
```yaml
config: {}
networks:
- config:
ipv4.address: auto
ipv4.nat: "true"
ipv6.address: none
name: incusbr0
type: bridge
storage_pools:
- config:
source: /var/lib/incus/storage-pools/default
name: default
driver: dir
profiles:
- config: {}
devices:
eth0:
name: eth0
network: incusbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
```
## Environment Variables in OCI Containers
Pass via `--config` flags at launch time:
```bash
incus launch docker:postgres:16-alpine gitea-db \
--config environment.POSTGRES_DB=gitea \
--config environment.POSTGRES_USER=gitea \
--config environment.POSTGRES_PASSWORD=secretpass
```
Update existing container:
```bash
incus config set gitea-db environment.POSTGRES_PASSWORD=newpass
incus restart gitea-db
```
## Backup & Restore
### Snapshots (fast, local only)
```bash
incus snapshot create <instance> <snapshot-name>
incus snapshot restore <instance> <snapshot-name>
incus snapshot list <instance>
```
### Exports (portable, for off-site backup)
```bash
incus export <instance> backup.tar.gz --optimized-storage
incus import backup.tar.gz <new-instance-name>
```
### Volume Exports
```bash
incus storage volume snapshot <pool> <volume> <snapshot-name>
incus storage volume export <pool> <volume> volume-backup.tar.gz
incus storage volume import <pool> volume-backup.tar.gz <new-volume-name>
```
### Database Dumps (same pattern as Docker)
```bash
incus exec gitea-db -- pg_dump -U gitea gitea > gitea-db.sql
incus exec outline-db -- pg_dump -U outline outline > outline-db.sql
```
### 6.22 Enhancements
- Direct backup streaming (no temporary disk files)
- Disk-only snapshot restore (revert storage without config)
## Auto-Start / Restart Policy
```bash
# Start container on host boot
incus config set <container> boot.autostart true
# Set start order (lower = earlier)
incus config set <container> boot.autostart.priority 10
# Start delay (seconds, for dependency ordering)
incus config set <container> boot.autostart.delay 5
```
No automatic restart on crash (unlike Docker's `restart: unless-stopped`). Options:
- Accept manual restart for crashes (rare)
- Add a systemd watchdog timer on the host
- Use `incus monitor` to watch for state changes
## Running Docker Inside Incus (for Gitea Runner)
If needed for the runner:
```bash
incus launch images:debian/13/cloud gitea-runner-host
incus config set gitea-runner-host security.nesting true
incus exec gitea-runner-host -- apt install docker.io
# Then run act_runner inside this system container
```
## Cloud-init Support
System containers with `/cloud` images support cloud-init:
```bash
incus launch images:debian/13/cloud my-container \
--config cloud-init.user-data="#cloud-config
packages:
- curl
- git
"
```
Note: Cloud-init runs only on first boot. Not applicable to OCI containers.
## Sources
- [Incus Documentation](https://linuxcontainers.org/incus/docs/main/)
- [Incus 6.22 Release](https://linuxcontainers.org/incus/news/)
- [Incus OCI Support (FOSDEM 2025)](https://archive.fosdem.org/2025/schedule/event/fosdem-2025-5742-bringing-application-containers-to-incus/)
- [Incus Ansible Connection Plugin](https://docs.ansible.com/projects/ansible/latest/collections/community/general/incus_connection.html)
- [kmpm/ansible-collection-incus](https://github.com/kmpm/ansible-collection-incus)
- [lxc/incus-deploy](https://github.com/lxc/incus-deploy)
- [Incus Backup Strategy (Forum)](https://discuss.linuxcontainers.org/t/incus-backup-strategy-incus-export-or-zfs-snapshot/25188)
- [Running OCI Images in Incus (blog.simos.info)](https://blog.simos.info/running-oci-images-i-e-docker-directly-in-incus/)
- [Incus Debian Wiki](https://wiki.debian.org/Incus)