7.4 KiB

Raw Permalink Blame History

Incus Implementation Plan

Goal

Build an Incus-based replica of our current Docker Compose infrastructure on a separate git branch (incus), managed by Ansible, and test it on a Proxmox VM before considering production use.

Approach: OCI Containers in Incus

Use Incus's native OCI container support (stable since v6.3, July 2024) to run the exact same Docker images we use today. The orchestration layer changes from docker compose to incus CLI commands driven by Ansible.

Why OCI over system containers? Services like Outline, Dex, and LLDAP are only distributed as Docker images. OCI containers give us the same images with Incus's networking and storage management. No need to install anything from source.

Gitea Runner: Deferred. It requires Docker socket access — a separate problem that needs its own solution (likely a system container with Docker inside).

Architecture

Container Mapping

Every service maps 1:1 from Docker to Incus. Container names are preserved so config files (Caddyfile, dex-config.yml) work unchanged.

Container	Image	Volumes	Exposed Ports
`caddy`	`caddy:2`	caddy-data, caddy-config, Caddyfile mount, /opt/www mount	80, 443, 443/udp
`lldap`	`lldap/lldap:stable`	lldap-data	(internal: 17170, 3890)
`dex`	`dexidp/dex:latest`	dex-data, dex-config mount	(internal: 5556)
`gitea`	`gitea/gitea:1.21`	gitea-data	2222→22
`gitea-db`	`postgres:16-alpine`	gitea-db-data	(internal: 5432)
`outline`	`outlinewiki/outline:1.6.0`	outline-data	(internal: 3000)
`outline-db`	`postgres:16-alpine`	outline-db-data	(internal: 5432)
`outline-redis`	`redis:7-alpine`	(none)	(internal: 6379)

Networking

Single managed bridge (incusbr0) with dnsmasq providing:

DHCP for all containers
DNS resolution by container name (e.g., gitea-db resolves from any container on the bridge)
IPv4 NAT for outbound internet access

Port forwarding via Incus proxy devices (replaces Docker's ports: directive):

caddy:    0.0.0.0:80  → 127.0.0.1:80
          0.0.0.0:443 → 127.0.0.1:443
gitea:    0.0.0.0:2222 → 127.0.0.1:22

Storage

Storage pool default with dir driver for testing (switch to zfs for production). Named volumes created with incus storage volume create and attached to containers as disk devices.

Config File Mounting

Config files templated to /opt/ce-config/ on host, mounted into containers via disk devices:

/opt/ce-config/Caddyfile     → caddy:/etc/caddy/Caddyfile
/opt/ce-config/dex-config.yml → dex:/etc/dex/config.yaml
/opt/www/                     → caddy:/srv/www

Ansible Role Structure (on `incus` branch)

roles/
  common/                        ← UNCHANGED from master
  incus/                         ← NEW (replaces roles/docker/)
    tasks/main.yml               ← Install Incus, init preseed, add OCI remote
    templates/preseed.yml.j2     ← Incus init config
  services/                      ← REWRITTEN for Incus
    tasks/
      main.yml                   ← Orchestration: config templating, calls service files
      lldap.yml                  ← LLDAP container + volume
      dex.yml                    ← Dex container + volume + config mount
      gitea.yml                  ← Gitea + Gitea-DB + volumes + proxy device
      outline.yml                ← Outline + Outline-DB + Redis + volumes
      caddy.yml                  ← Caddy + volumes + config mounts + proxy devices
    templates/
      Caddyfile.j2               ← UNCHANGED (container names match)
      dex-config.yml.j2          ← UNCHANGED (container names match)

Per-Service Task Pattern

Each service file follows this idempotent pattern:

# 1. Check if container exists
- name: Check if <service> exists
  command: incus info <service>
  register: exists
  failed_when: false
  changed_when: false

# 2. Create storage volume (idempotent — incus ignores if exists)
- name: Create <service> volume
  command: incus storage volume create {{ incus_storage_pool }} <volume-name>
  when: exists.rc != 0

# 3. Launch OCI container with environment variables
- name: Create <service> container
  command: >
    incus launch docker:<image>:<tag> <service>
    --config environment.KEY1=value1
    --config environment.KEY2=value2    
  when: exists.rc != 0

# 4. Attach storage volume
- name: Attach <volume> to <service>
  command: >
    incus config device add <service> <device-name> disk
    pool={{ incus_storage_pool }} source=<volume-name> path=<mount-path>    
  when: exists.rc != 0

# 5. Attach proxy device (if externally exposed)
- name: Add <service> proxy device
  command: >
    incus config device add <service> <name> proxy
    listen=tcp:0.0.0.0:<host-port> connect=tcp:127.0.0.1:<container-port>    
  when: exists.rc != 0

# 6. Wait for readiness
- name: Wait for <service>
  command: incus exec <service> -- <health-check-command>
  retries: 10
  delay: 3
  until: result.rc == 0

Implementation Phases

Phase 0 — Branch + Test Environment

git checkout -b incus from master
Create Debian 13 VM on Proxmox testlab
Update inventory/hosts.yml to target test VM
Verify: ansible all -m ping

Phase 1 — Install Incus (`roles/incus/`)

Install incus package from Debian 13 repos
Initialize with preseed (storage pool + bridge)
Add Docker Hub OCI remote
Verify: incus info succeeds

Phase 2 — Deploy Services (one at a time)

Deploy in dependency order, verifying each before moving on:

LLDAP → verify web UI responds
Dex → verify OIDC discovery endpoint
Gitea-DB → verify pg_isready
Gitea → verify web UI, add OIDC auth source
Outline-DB → verify pg_isready
Outline-Redis → verify redis-cli ping
Outline → verify web UI
Caddy → verify reverse proxy to all services

Phase 3 — Playbook Integration

Wire up bootstrap.yml (common + incus)
Wire up deploy.yml (services)
Full end-to-end test from clean VM

Phase 4 — Backup/Restore

Rewrite scripts using incus exec for pg_dump and incus storage volume export for volumes
Test full backup + destroy + restore cycle

Phase 5 — Documentation

Complete comparison document
Update CLAUDE.md on the incus branch

Variables to Add

# Incus configuration (add to inventory/group_vars/all.yml)
incus_storage_pool: "default"
incus_storage_driver: "dir"    # "zfs" for production
incus_bridge: "incusbr0"

All existing variables (domain, passwords, secrets, image versions) remain unchanged.

Key Gotchas

Container names must match DNS references — Caddyfile uses gitea:3000, dex-config uses lldap:3890. Container names must be exactly gitea, lldap, etc.
No depends_on — Ansible task ordering handles startup sequence. Use retries/delay/until for readiness checks.
OCI image pull syntax — incus launch docker:lldap/lldap:stable lldap (note: docker: prefix)
Config file updates — Template to host, then incus restart <container>. Register changed state in Ansible.
Restart policy — Set boot.autostart=true on all containers. No automatic restart on crash (add watchdog if needed).
UFW still applies — Host firewall from common role still controls which ports are reachable. Current rules (22, 80, 443, 2222) are correct.
First image pull is slow — Incus downloads and converts OCI images on first launch. Subsequent launches use cache.

7.4 KiB Raw Permalink Blame History