ce-infra/docs/incus/implementation-plan.md

# Incus Implementation Plan

## Goal

Build an Incus-based replica of our current Docker Compose infrastructure on a separate git branch (`incus`), managed by Ansible, and test it on a Proxmox VM before considering production use.

## Approach: OCI Containers in Incus

Use Incus's native OCI container support (stable since v6.3, July 2024) to run the **exact same Docker images** we use today. The orchestration layer changes from `docker compose` to `incus` CLI commands driven by Ansible.

**Why OCI over system containers?** Services like Outline, Dex, and LLDAP are only distributed as Docker images. OCI containers give us the same images with Incus's networking and storage management. No need to install anything from source.

**Gitea Runner**: Deferred. It requires Docker socket access — a separate problem that needs its own solution (likely a system container with Docker inside).

## Architecture

### Container Mapping

Every service maps 1:1 from Docker to Incus. Container names are preserved so config files (Caddyfile, dex-config.yml) work unchanged.

| Container | Image | Volumes | Exposed Ports |
|---|---|---|---|
| `caddy` | `caddy:2` | caddy-data, caddy-config, Caddyfile mount, /opt/www mount | 80, 443, 443/udp |
| `lldap` | `lldap/lldap:stable` | lldap-data | (internal: 17170, 3890) |
| `dex` | `dexidp/dex:latest` | dex-data, dex-config mount | (internal: 5556) |
| `gitea` | `gitea/gitea:1.21` | gitea-data | 2222→22 |
| `gitea-db` | `postgres:16-alpine` | gitea-db-data | (internal: 5432) |
| `outline` | `outlinewiki/outline:1.6.0` | outline-data | (internal: 3000) |
| `outline-db` | `postgres:16-alpine` | outline-db-data | (internal: 5432) |
| `outline-redis` | `redis:7-alpine` | (none) | (internal: 6379) |

### Networking

Single managed bridge (`incusbr0`) with dnsmasq providing:
- DHCP for all containers
- DNS resolution by container name (e.g., `gitea-db` resolves from any container on the bridge)
- IPv4 NAT for outbound internet access

Port forwarding via Incus proxy devices (replaces Docker's `ports:` directive):
```
caddy:    0.0.0.0:80  → 127.0.0.1:80
          0.0.0.0:443 → 127.0.0.1:443
gitea:    0.0.0.0:2222 → 127.0.0.1:22
```

### Storage

Storage pool `default` with `dir` driver for testing (switch to `zfs` for production). Named volumes created with `incus storage volume create` and attached to containers as disk devices.

### Config File Mounting

Config files templated to `/opt/ce-config/` on host, mounted into containers via disk devices:
```
/opt/ce-config/Caddyfile     → caddy:/etc/caddy/Caddyfile
/opt/ce-config/dex-config.yml → dex:/etc/dex/config.yaml
/opt/www/                     → caddy:/srv/www
```

## Ansible Role Structure (on `incus` branch)

```
roles/
  common/                        ← UNCHANGED from master
  incus/                         ← NEW (replaces roles/docker/)
    tasks/main.yml               ← Install Incus, init preseed, add OCI remote
    templates/preseed.yml.j2     ← Incus init config
  services/                      ← REWRITTEN for Incus
    tasks/
      main.yml                   ← Orchestration: config templating, calls service files
      lldap.yml                  ← LLDAP container + volume
      dex.yml                    ← Dex container + volume + config mount
      gitea.yml                  ← Gitea + Gitea-DB + volumes + proxy device
      outline.yml                ← Outline + Outline-DB + Redis + volumes
      caddy.yml                  ← Caddy + volumes + config mounts + proxy devices
    templates/
      Caddyfile.j2               ← UNCHANGED (container names match)
      dex-config.yml.j2          ← UNCHANGED (container names match)
```

## Per-Service Task Pattern

Each service file follows this idempotent pattern:

```yaml
# 1. Check if container exists
- name: Check if <service> exists
  command: incus info <service>
  register: exists
  failed_when: false
  changed_when: false

# 2. Create storage volume (idempotent — incus ignores if exists)
- name: Create <service> volume
  command: incus storage volume create {{ incus_storage_pool }} <volume-name>
  when: exists.rc != 0

# 3. Launch OCI container with environment variables
- name: Create <service> container
  command: >
    incus launch docker:<image>:<tag> <service>
    --config environment.KEY1=value1
    --config environment.KEY2=value2    
  when: exists.rc != 0

# 4. Attach storage volume
- name: Attach <volume> to <service>
  command: >
    incus config device add <service> <device-name> disk
    pool={{ incus_storage_pool }} source=<volume-name> path=<mount-path>    
  when: exists.rc != 0

# 5. Attach proxy device (if externally exposed)
- name: Add <service> proxy device
  command: >
    incus config device add <service> <name> proxy
    listen=tcp:0.0.0.0:<host-port> connect=tcp:127.0.0.1:<container-port>    
  when: exists.rc != 0

# 6. Wait for readiness
- name: Wait for <service>
  command: incus exec <service> -- <health-check-command>
  retries: 10
  delay: 3
  until: result.rc == 0
```

## Implementation Phases

### Phase 0 — Branch + Test Environment
1. `git checkout -b incus` from master
2. Create Debian 13 VM on Proxmox testlab
3. Update `inventory/hosts.yml` to target test VM
4. Verify: `ansible all -m ping`

### Phase 1 — Install Incus (`roles/incus/`)
1. Install `incus` package from Debian 13 repos
2. Initialize with preseed (storage pool + bridge)
3. Add Docker Hub OCI remote
4. Verify: `incus info` succeeds

### Phase 2 — Deploy Services (one at a time)
Deploy in dependency order, verifying each before moving on:
1. LLDAP → verify web UI responds
2. Dex → verify OIDC discovery endpoint
3. Gitea-DB → verify `pg_isready`
4. Gitea → verify web UI, add OIDC auth source
5. Outline-DB → verify `pg_isready`
6. Outline-Redis → verify `redis-cli ping`
7. Outline → verify web UI
8. Caddy → verify reverse proxy to all services

### Phase 3 — Playbook Integration
- Wire up `bootstrap.yml` (common + incus)
- Wire up `deploy.yml` (services)
- Full end-to-end test from clean VM

### Phase 4 — Backup/Restore
- Rewrite scripts using `incus exec` for pg_dump and `incus storage volume export` for volumes
- Test full backup + destroy + restore cycle

### Phase 5 — Documentation
- Complete comparison document
- Update CLAUDE.md on the incus branch

## Variables to Add

```yaml
# Incus configuration (add to inventory/group_vars/all.yml)
incus_storage_pool: "default"
incus_storage_driver: "dir"    # "zfs" for production
incus_bridge: "incusbr0"
```

All existing variables (domain, passwords, secrets, image versions) remain unchanged.

## Key Gotchas

1. **Container names must match DNS references** — Caddyfile uses `gitea:3000`, dex-config uses `lldap:3890`. Container names must be exactly `gitea`, `lldap`, etc.
2. **No depends_on** — Ansible task ordering handles startup sequence. Use `retries/delay/until` for readiness checks.
3. **OCI image pull syntax** — `incus launch docker:lldap/lldap:stable lldap` (note: `docker:` prefix)
4. **Config file updates** — Template to host, then `incus restart <container>`. Register changed state in Ansible.
5. **Restart policy** — Set `boot.autostart=true` on all containers. No automatic restart on crash (add watchdog if needed).
6. **UFW still applies** — Host firewall from `common` role still controls which ports are reachable. Current rules (22, 80, 443, 2222) are correct.
7. **First image pull is slow** — Incus downloads and converts OCI images on first launch. Subsequent launches use cache.