ce-infra/docs/incus/comparison-docker-vs-incus.md

# Docker vs Incus — Comparison for Cloud Elves Infrastructure

## Executive Summary

Docker Compose is simpler for day-to-day operations and has a massive ecosystem. Incus offers stronger isolation, first-class backup/snapshot capabilities, and a unified platform for containers and VMs. For our current 4-person team on a single server, Docker is the pragmatic choice. Incus becomes compelling when we need VMs alongside containers, stronger tenant isolation for customer workloads, or a platform to offer as a product.

## Architecture Comparison

### How They Work

```mermaid
sequenceDiagram
    participant Admin
    participant Docker as Docker Compose
    participant Containers as App Containers

    Admin->>Docker: docker compose up
    Docker->>Containers: Create all containers from YAML
    Note over Docker: Declarative — single file defines entire stack
    Docker-->>Admin: All services running
```

```mermaid
sequenceDiagram
    participant Admin
    participant Ansible
    participant Incus
    participant Containers as OCI Containers

    Admin->>Ansible: ansible-playbook deploy.yml
    Ansible->>Incus: incus launch (per service)
    Incus->>Containers: Create containers individually
    Ansible->>Incus: incus config device add (volumes, proxies)
    Note over Ansible: Imperative — Ansible orchestrates step by step
    Incus-->>Admin: All services running
```

### Feature Comparison

| Feature | Docker Compose | Incus |
|---|---|---|
| **Orchestration** | Declarative YAML — one file defines everything | Imperative CLI — each container managed individually |
| **Learning curve** | Low — widely known, massive community | Medium — less common, different mental model |
| **OCI image support** | Native (this is Docker) | Supported since v6.3 (July 2024) |
| **System containers** | Not supported | First-class — full OS with systemd |
| **Virtual machines** | Not supported | First-class — run VMs alongside containers |
| **DNS discovery** | Compose service names on network | Container names on managed bridge (dnsmasq) |
| **Port forwarding** | `ports:` in compose file | Proxy devices |
| **Volumes** | Named volumes | Storage pool volumes |
| **Config mounts** | Bind mounts | Disk devices |
| **Network isolation** | Multiple compose networks | Multiple bridges (but single bridge is simpler) |
| **Restart policy** | `restart: unless-stopped` | `boot.autostart=true` (no crash restart) |
| **Compose-like tooling** | Docker Compose (mature) | `incus-compose` (incomplete/WIP) |
| **Ecosystem** | Enormous — Docker Hub, GitHub Actions, etc. | Growing — Linux Containers community |

### Resource Overhead

| Scenario | Docker | Incus OCI | Incus System Container |
|---|---|---|---|
| Base container overhead | ~5-10MB | ~5-10MB (comparable) | ~50-100MB (full OS) |
| 9 containers (our stack) | ~100MB overhead | ~100MB overhead | ~500-900MB overhead |
| CPU overhead | Negligible | Negligible | Negligible |
| Disk per container | Layered images (efficient) | Converted images | Full rootfs (150-500MB) |

**Bottom line**: OCI containers in Incus have comparable overhead to Docker. System containers are heavier due to systemd and OS services.

## Security & Isolation

| Aspect | Docker | Incus |
|---|---|---|
| **Kernel sharing** | Shared with host (same as Incus) | Shared with host (same as Docker) |
| **Namespace isolation** | Process, network, mount, IPC, UTS, user | Same namespaces + optional AppArmor/seccomp profiles |
| **Root in container** | Often runs as root (configurable) | System containers: full user space. OCI: same as Docker |
| **Container escape risk** | Kernel vulnerability = escape | Same risk for both |
| **Multi-tenant isolation** | Not designed for it | System containers provide stronger tenant boundaries |
| **Docker socket exposure** | Common pattern (e.g., Gitea runner) | Not needed — Incus has its own API |

**Key insight**: For our use case (single team, all trusted), the security difference is negligible. Incus's stronger isolation matters when hosting untrusted workloads or providing infrastructure to customers.

## Operational Comparison

### Day-to-Day Operations

| Task | Docker | Incus |
|---|---|---|
| Deploy all services | `docker compose up -d` | `ansible-playbook deploy.yml` |
| View running services | `docker compose ps` | `incus list` |
| View logs | `docker compose logs gitea` | `incus exec gitea -- cat /var/log/...` or `incus console gitea --type=log` |
| Restart service | `docker compose restart gitea` | `incus restart gitea` |
| Update service image | Edit compose, `docker compose up -d` | `incus stop gitea && incus delete gitea` + relaunch |
| Enter container shell | `docker compose exec gitea bash` | `incus exec gitea -- bash` |
| Deploy static site | `rsync` to host (unchanged) | `rsync` to host (unchanged) |

### Backup & Restore

| Capability | Docker | Incus |
|---|---|---|
| Database dump | `docker compose exec db pg_dump` | `incus exec db -- pg_dump` |
| Volume backup | Manual: `docker run --rm -v ... tar czf` | Built-in: `incus storage volume export` |
| Full instance snapshot | Not built-in | `incus snapshot create` (instant with ZFS) |
| Portable export | Manual tarball assembly | `incus export` (single command, includes volumes) |
| Remote backup | Needs external tool (Restic, etc.) | Same — snapshots are local only |
| Restore to different host | Manual — recreate containers, import data | `incus import` (single command) |

**Incus advantage**: Backup is a first-class feature. `incus export` creates a single portable file containing the container and all its data. With ZFS, snapshots are instant and space-efficient.

### Service Updates

| Scenario | Docker | Incus |
|---|---|---|
| Bump image version | Edit compose file, `up -d` | Delete container, relaunch with new image (data in volumes survives) |
| Config change | Edit template, `up -d` (smart restart) | Edit template on host, `incus restart` |
| Rollback | Re-pull old image, `up -d` | `incus snapshot restore` (if snapshot taken pre-upgrade) |

**Docker advantage**: Image updates are more ergonomic. Incus requires deleting and recreating the container (since OCI containers can't change their image in-place).

## When to Choose Which

### Choose Docker When
- Team is familiar with Docker (faster onboarding)
- Using many third-party services distributed as Docker images
- Want declarative stack definition (single compose file)
- Simple single-node deployment
- Ecosystem matters (GitHub Actions, CI/CD, monitoring tools all expect Docker)

### Choose Incus When
- Need VMs alongside containers (e.g., testing, customer environments)
- Want first-class backup/snapshot capabilities
- Building infrastructure as a product for customers
- Need stronger multi-tenant isolation
- Want a single platform for containers + VMs + storage
- Comfortable with Ansible-driven infrastructure (no compose equivalent)

### For Cloud Elves Specifically

**Current recommendation: Stay with Docker for production.** The stack is working, tested, and everyone knows it.

**Build the Incus branch as a parallel capability** for:
- Learning and R&D
- Potential product offering (Incus-managed customer environments)
- Testing whether Incus + ZFS backup model is simpler than Docker + Restic
- Evaluating system containers for future services that need full OS

## Migration Path

Moving from Docker to Incus (if we decide to switch):

```mermaid
sequenceDiagram
    participant Docker as Docker Server
    participant Backup as Backup Storage
    participant Incus as Incus Server

    Docker->>Backup: Run backup.sh (pg_dumps + volume tarballs)
    Note over Incus: Fresh Debian 13, Incus installed
    Incus->>Incus: Run bootstrap.yml + deploy.yml
    Backup->>Incus: Run restore.sh (import pg_dumps + volume data)
    Note over Incus: All services running with restored data
```

The data formats are identical (Postgres dumps, file tarballs). The migration is just:
1. Backup from Docker server
2. Deploy empty Incus server
3. Restore data into Incus containers

No data format conversion needed.