ce-infra/docs/incus/comparison-docker-vs-incus.md

Docker vs Incus — Comparison for Cloud Elves Infrastructure

Executive Summary

Docker Compose is simpler for day-to-day operations and has a massive ecosystem. Incus offers stronger isolation, first-class backup/snapshot capabilities, and a unified platform for containers and VMs. For our current 4-person team on a single server, Docker is the pragmatic choice. Incus becomes compelling when we need VMs alongside containers, stronger tenant isolation for customer workloads, or a platform to offer as a product.

Architecture Comparison

How They Work

sequenceDiagram
    participant Admin
    participant Docker as Docker Compose
    participant Containers as App Containers

    Admin->>Docker: docker compose up
    Docker->>Containers: Create all containers from YAML
    Note over Docker: Declarative — single file defines entire stack
    Docker-->>Admin: All services running

sequenceDiagram
    participant Admin
    participant Ansible
    participant Incus
    participant Containers as OCI Containers

    Admin->>Ansible: ansible-playbook deploy.yml
    Ansible->>Incus: incus launch (per service)
    Incus->>Containers: Create containers individually
    Ansible->>Incus: incus config device add (volumes, proxies)
    Note over Ansible: Imperative — Ansible orchestrates step by step
    Incus-->>Admin: All services running

Feature Comparison

Feature	Docker Compose	Incus
Orchestration	Declarative YAML — one file defines everything	Imperative CLI — each container managed individually
Learning curve	Low — widely known, massive community	Medium — less common, different mental model
OCI image support	Native (this is Docker)	Supported since v6.3 (July 2024)
System containers	Not supported	First-class — full OS with systemd
Virtual machines	Not supported	First-class — run VMs alongside containers
DNS discovery	Compose service names on network	Container names on managed bridge (dnsmasq)
Port forwarding	ports: in compose file	Proxy devices
Volumes	Named volumes	Storage pool volumes
Config mounts	Bind mounts	Disk devices
Network isolation	Multiple compose networks	Multiple bridges (but single bridge is simpler)
Restart policy	restart: unless-stopped	boot.autostart=true (no crash restart)
Compose-like tooling	Docker Compose (mature)	incus-compose (incomplete/WIP)
Ecosystem	Enormous — Docker Hub, GitHub Actions, etc.	Growing — Linux Containers community

Resource Overhead

Scenario	Docker	Incus OCI	Incus System Container
Base container overhead	~5-10MB	~5-10MB (comparable)	~50-100MB (full OS)
9 containers (our stack)	~100MB overhead	~100MB overhead	~500-900MB overhead
CPU overhead	Negligible	Negligible	Negligible
Disk per container	Layered images (efficient)	Converted images	Full rootfs (150-500MB)

Bottom line: OCI containers in Incus have comparable overhead to Docker. System containers are heavier due to systemd and OS services.

Security & Isolation

Aspect	Docker	Incus
Kernel sharing	Shared with host (same as Incus)	Shared with host (same as Docker)
Namespace isolation	Process, network, mount, IPC, UTS, user	Same namespaces + optional AppArmor/seccomp profiles
Root in container	Often runs as root (configurable)	System containers: full user space. OCI: same as Docker
Container escape risk	Kernel vulnerability = escape	Same risk for both
Multi-tenant isolation	Not designed for it	System containers provide stronger tenant boundaries
Docker socket exposure	Common pattern (e.g., Gitea runner)	Not needed — Incus has its own API

Key insight: For our use case (single team, all trusted), the security difference is negligible. Incus's stronger isolation matters when hosting untrusted workloads or providing infrastructure to customers.

Operational Comparison

Day-to-Day Operations

Task	Docker	Incus
Deploy all services	docker compose up -d	ansible-playbook deploy.yml
View running services	docker compose ps	incus list
View logs	docker compose logs gitea	incus exec gitea -- cat /var/log/... or incus console gitea --type=log
Restart service	docker compose restart gitea	incus restart gitea
Update service image	Edit compose, docker compose up -d	incus stop gitea && incus delete gitea + relaunch
Enter container shell	docker compose exec gitea bash	incus exec gitea -- bash
Deploy static site	rsync to host (unchanged)	rsync to host (unchanged)

Backup & Restore

Capability	Docker	Incus
Database dump	docker compose exec db pg_dump	incus exec db -- pg_dump
Volume backup	Manual: docker run --rm -v ... tar czf	Built-in: incus storage volume export
Full instance snapshot	Not built-in	incus snapshot create (instant with ZFS)
Portable export	Manual tarball assembly	incus export (single command, includes volumes)
Remote backup	Needs external tool (Restic, etc.)	Same — snapshots are local only
Restore to different host	Manual — recreate containers, import data	incus import (single command)

Incus advantage: Backup is a first-class feature. incus export creates a single portable file containing the container and all its data. With ZFS, snapshots are instant and space-efficient.

Service Updates

Scenario	Docker	Incus
Bump image version	Edit compose file, up -d	Delete container, relaunch with new image (data in volumes survives)
Config change	Edit template, up -d (smart restart)	Edit template on host, incus restart
Rollback	Re-pull old image, up -d	incus snapshot restore (if snapshot taken pre-upgrade)

Docker advantage: Image updates are more ergonomic. Incus requires deleting and recreating the container (since OCI containers can't change their image in-place).

When to Choose Which

Choose Docker When

• Team is familiar with Docker (faster onboarding)
• Using many third-party services distributed as Docker images
• Want declarative stack definition (single compose file)
• Simple single-node deployment
• Ecosystem matters (GitHub Actions, CI/CD, monitoring tools all expect Docker)

Choose Incus When

• Need VMs alongside containers (e.g., testing, customer environments)
• Want first-class backup/snapshot capabilities
• Building infrastructure as a product for customers
• Need stronger multi-tenant isolation
• Want a single platform for containers + VMs + storage
• Comfortable with Ansible-driven infrastructure (no compose equivalent)

For Cloud Elves Specifically

Current recommendation: Stay with Docker for production. The stack is working, tested, and everyone knows it.

Build the Incus branch as a parallel capability for:

• Learning and R&D
• Potential product offering (Incus-managed customer environments)
• Testing whether Incus + ZFS backup model is simpler than Docker + Restic
• Evaluating system containers for future services that need full OS

Migration Path

Moving from Docker to Incus (if we decide to switch):

sequenceDiagram
    participant Docker as Docker Server
    participant Backup as Backup Storage
    participant Incus as Incus Server

    Docker->>Backup: Run backup.sh (pg_dumps + volume tarballs)
    Note over Incus: Fresh Debian 13, Incus installed
    Incus->>Incus: Run bootstrap.yml + deploy.yml
    Backup->>Incus: Run restore.sh (import pg_dumps + volume data)
    Note over Incus: All services running with restored data

The data formats are identical (Postgres dumps, file tarballs). The migration is just:

1. Backup from Docker server
2. Deploy empty Incus server
3. Restore data into Incus containers

No data format conversion needed.