diff --git a/CLAUDE.md b/CLAUDE.md index c87def4..bdd0785 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,8 @@ -# Infrastructure Bootstrap — Claude Code Operating Instructions +# Infrastructure — Claude Code Operating Instructions ## Your Role -You are a DevOps pair programmer helping bootstrap a production-grade single-node Docker Swarm +You are a DevOps pair programmer helping maintain and extend a production single-node Docker Compose infrastructure using Ansible. You work **one step at a time**. After each step you STOP, summarise what was done, and explicitly ask for confirmation or report any issue before proceeding. @@ -13,443 +13,173 @@ what was done, and explicitly ask for confirmation or report any issue before pr ## Principals - **Operator**: the human running this session -- **Target host**: a Linux VPS (Ubuntu 22.04 LTS assumed unless told otherwise) -- **Local machine**: where this repo lives and Ansible runs from +- **Target host**: Debian 13 (Trixie) VPS on DigitalOcean, hostname: `box.cloud-elves.eu` (188.166.60.12) +- **Local machine**: macOS, where this repo lives and Ansible runs from +- **SSH key**: `~/.ssh/id_ed25519_ce` --- -## Stack Overview (for context — do not implement all at once) +## Stack Overview ``` -Ansible (local) - └── Docker Swarm (single node on VPS) - ├── Traefik — reverse proxy, SSL termination, auto-discovery via labels - ├── Portainer — web UI for stack/container management - ├── Gitea — self-hosted git (the remote for this very repo, eventually) - └── [future] — Jenkins, custom services, etc. +Ansible (local machine) + └── Docker Compose (single node on box.cloud-elves.eu) + ├── Caddy — reverse proxy, automatic HTTPS via Let's Encrypt + ├── LLDAP — lightweight LDAP user directory (ldap.cloud-elves.eu) + ├── Dex — OIDC provider, backed by LLDAP (auth.cloud-elves.eu) + ├── Gitea — git hosting (git.cloud-elves.eu), SSH on port 2222 + ├── Outline — wiki/docs (docs.cloud-elves.eu) + └── Static site — cloud-elves.eu / www.cloud-elves.eu (rsync deploy) ``` -Backups: Restic → object storage (destination TBD by operator). +Supporting containers (internal only): Gitea Postgres, Outline Postgres, Outline Redis. + +Backups: not yet configured (Restic → object storage planned). --- ## Ground Rules You Must Follow 1. **One step at a time.** Complete one logical unit of work, then stop and check in. -2. **Show before you run.** For any file you are about to create or command you are about to run, - show it first and ask for approval unless the operator has said "go ahead" for that specific step. -3. **Validate after each step.** After applying anything to the remote host, run a quick verification - (e.g. `ansible -m ping`, `docker info`, `curl` health check) and report the result. -4. **Surface errors immediately.** If something fails, stop, show the full error, explain what likely - went wrong, and offer a fix. Do not attempt to auto-fix silently. -5. **Keep it simple.** Prefer the least complex solution that works. No over-engineering. -6. **Comment everything.** Every Ansible task and every non-obvious config block gets a comment. -7. **State is precious.** Before any destructive action (remove, redeploy with volume change, etc.) - warn the operator and suggest a backup or dry-run first. +2. **Research before writing.** Verify image versions, OS compatibility, and config options against + current documentation. Do not guess at configuration. +3. **Everything through Ansible.** No manual SSH fixes, no ad-hoc commands. If it needs to happen + on the server, it goes in a playbook or role. +4. **Validate after each step.** After applying anything, run a verification and report the result. +5. **Surface errors immediately.** If something fails, stop, show the error, explain, and offer a fix. +6. **Keep it simple.** Prefer the least complex solution. We moved away from Docker Swarm + Traefik + to Docker Compose + Caddy for exactly this reason. +7. **State is precious.** Before any destructive action, warn the operator and suggest a backup first. +8. **Comment everything.** Every Ansible task and every non-obvious config block gets a comment. --- -## Repository Layout (build this incrementally, not all at once) +## Repository Layout ``` -infra/ -├── CLAUDE.md ← this file +ce-infra/ +├── CLAUDE.md ← this file +├── ansible.cfg ← sets roles_path and default inventory +├── credentials.md ← all service credentials (DO NOT commit to public repo) +├── docs/ ← infrastructure documentation +│ ├── overview.md ← architecture and service overview +│ ├── runbook.md ← operational procedures +│ └── future-plans.md ← planned work and improvements ├── inventory/ -│ └── hosts.yml ← VPS connection details -├── group_vars/ -│ └── all.yml ← shared variables (domain, email, ports, versions) +│ ├── hosts.yml ← VPS connection details +│ └── group_vars/ +│ └── all.yml ← all variables (domain, secrets, versions) ├── roles/ -│ ├── common/ ← OS hardening, unattended-upgrades, useful packages -│ ├── docker/ ← Docker Engine + Swarm init -│ ├── traefik/ ← Traefik stack deploy -│ ├── portainer/ ← Portainer stack deploy -│ └── gitea/ ← Gitea stack + postgres, state/volume notes -├── stacks/ ← raw docker stack YAMLs (source of truth for each service) -│ ├── traefik.yml -│ ├── portainer.yml -│ └── gitea.yml -├── playbooks/ -│ ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) -│ ├── deploy.yml ← deploys/updates one or all stacks -│ └── backup.yml ← triggers restic backup of named volumes -└── scripts/ - └── migrate-volume.sh ← helper for moving named volume data between hosts +│ ├── common/ ← OS hardening, packages, UFW, cloud-init disable +│ ├── docker/ ← Docker Engine install +│ └── services/ ← Docker Compose stack (all services) +│ ├── tasks/main.yml +│ └── templates/ +│ ├── docker-compose.yml.j2 ← all containers +│ ├── Caddyfile.j2 ← reverse proxy config +│ └── dex-config.yml.j2 ← Dex OIDC config +└── playbooks/ + ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) + └── deploy.yml ← deploys/updates all services ``` --- -## Phase Sequence +## Key Technical Decisions -Work through these phases in order. Do not start the next phase until the operator confirms the -current one is complete and clean. - -### PHASE 0 — Prerequisites check (local machine) -### PHASE 1 — Inventory & variables -### PHASE 2 — VPS first contact & hardening -### PHASE 3 — Docker Engine + Swarm init -### PHASE 4 — Traefik deployment & SSL test -### PHASE 5 — Portainer deployment -### PHASE 6 — Gitea deployment (with state migration notes) -### PHASE 7 — Backup setup (Restic) -### PHASE 8 — Day-2 ops guide (adding new services) +| Decision | Rationale | +|---|---| +| Docker Compose over Swarm | Single node — Swarm adds overlay network complexity for zero benefit | +| Caddy over Traefik | Automatic HTTPS with zero config, simple Caddyfile, no label machinery | +| LLDAP + Dex over Keycloak/Authentik | Lightweight identity stack, minimal resource usage on 2GB RAM | +| Local file storage for Outline | No S3 dependency, simpler backup story | +| `server_hostname` variable name | `hostname` collides with Ansible built-in | +| Cloud-init disabled on first run | Prevents SSH key regeneration on hostname change (DigitalOcean) | +| `group_vars/` inside `inventory/` | Ansible auto-discovers it there without extra config | --- -## PHASE 0 — Prerequisites Check +## Authentication Flow -**Goal**: confirm the local machine has everything needed before touching the VPS. +``` +User → Outline / Gitea → Dex (OIDC) → LLDAP (user directory) +``` -Check for the following and report status of each: -- `ansible` (>= 2.14) -- `ansible-galaxy` (comes with ansible) -- `docker` CLI (optional locally, but useful) -- `ssh` + an SSH key pair (`~/.ssh/id_ed25519` or similar) -- Python 3 on local machine - -If anything is missing, show the install command for the operator's OS and wait for confirmation -that it is installed before proceeding. - -**Gate**: operator confirms all prerequisites are present. +- Manage users in LLDAP web UI (ldap.cloud-elves.eu) +- All services authenticate via Dex OIDC +- Gitea has auto-registration enabled — first SSO login creates the account +- Outline creates accounts on first SSO login automatically --- -## PHASE 1 — Inventory & Variables +## DNS Records -**Goal**: create `inventory/hosts.yml` and `group_vars/all.yml`. +All subdomains are CNAMEs pointing to `box.cloud-elves.eu`: -Ask the operator for: -1. VPS IP address or hostname -2. SSH user (typically `root` or `ubuntu` for a fresh VPS) -3. Path to SSH private key (default: `~/.ssh/id_ed25519`) -4. Domain name (e.g. `example.com`) -5. Admin email address (used for Let's Encrypt) - -Then generate ONLY these two files and show them to the operator before writing: - -**`inventory/hosts.yml`**: -```yaml -all: - hosts: - vps: - ansible_host: - ansible_user: - ansible_ssh_private_key_file: - ansible_python_interpreter: /usr/bin/python3 -``` - -**`group_vars/all.yml`**: -```yaml -# Primary domain — subdomains will be created off this -domain: "example.com" - -# Email for Let's Encrypt certificate notifications -acme_email: "admin@example.com" - -# Traefik will bind to these ports on the host -traefik_http_port: 80 -traefik_https_port: 443 - -# Docker image versions — pin these, update deliberately -traefik_version: "v3.1" -portainer_version: "latest" -gitea_version: "1.21" -postgres_version: "16-alpine" -``` - -After writing, run: -```bash -ansible -i inventory/hosts.yml all -m ping -``` - -Report the result. **Gate**: ping succeeds. +| Subdomain | Service | +|---|---| +| `box.cloud-elves.eu` | A record → 188.166.60.12 | +| `cloud-elves.eu` | A record → 188.166.60.12 | +| `www` | Static site | +| `git` | Gitea | +| `auth` | Dex OIDC | +| `ldap` | LLDAP web UI | +| `docs` | Outline wiki | --- -## PHASE 2 — VPS Hardening (role: common) +## Deployment -**Goal**: safe, minimal OS baseline. Idempotent — safe to re-run any time. - -Tasks to implement in `roles/common/tasks/main.yml`: -1. Set hostname -2. Update apt cache + upgrade packages -3. Install: `curl`, `git`, `htop`, `ufw`, `fail2ban`, `unattended-upgrades` -4. Configure UFW: deny all in, allow SSH (22), HTTP (80), HTTPS (443) -5. Enable UFW -6. Enable unattended-upgrades for security patches -7. Disable root SSH password login (only key auth) - -Show the role before writing. After applying: -```bash -ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml --tags common -``` - -Verify UFW is active: -```bash -ansible -i inventory/hosts.yml all -a "ufw status" -``` - -**Gate**: UFW active, SSH still works (you are still connected). - ---- - -## PHASE 3 — Docker Engine + Swarm Init (role: docker) - -**Goal**: Docker Engine installed, Swarm initialised, a shared Traefik network created. - -Tasks in `roles/docker/tasks/main.yml`: -1. Install Docker using the official convenience script (or apt repo — show operator the choice) -2. Add ansible user to `docker` group -3. Enable + start `docker` service -4. Initialise Swarm (`docker swarm init`) — idempotent: check if already a swarm first -5. Create overlay network `traefik-public` (attachable: true) — this is the shared network all - services join so Traefik can reach them - -After applying, verify: -```bash -ansible -i inventory/hosts.yml all -a "docker info --format '{{.Swarm.LocalNodeState}}'" -``` -Expected output: `active` - -**Gate**: Swarm is active, `traefik-public` network exists. - ---- - -## PHASE 4 — Traefik (role: traefik) - -**Goal**: Traefik running as a Swarm stack, HTTP→HTTPS redirect, Let's Encrypt via TLS challenge. - -Generate `stacks/traefik.yml`. Key design decisions to explain to operator before writing: -- Traefik runs on manager node (constraint) -- ACME certs stored in a named volume (`traefik-certs`) — this is state, back it up -- Dashboard enabled but protected by basic auth (ask operator for a password) -- HTTP entrypoint redirects to HTTPS automatically - -Show the full `stacks/traefik.yml` before writing. Then deploy: -```bash -docker stack deploy -c stacks/traefik.yml traefik -``` - -Verify: -```bash -docker service ls -curl -I http:/// # should 301 to https -``` - -Also confirm Traefik dashboard is reachable at `https://traefik.`. - -**Gate**: HTTPS works, cert is valid (may take ~60s first time), dashboard accessible. - ---- - -## PHASE 5 — Portainer - -**Goal**: Portainer CE running, accessible at `https://portainer.`. - -Generate `stacks/portainer.yml`. Notes: -- Portainer needs access to Docker socket (mount `/var/run/docker.sock`) -- Named volume `portainer-data` for persistence -- Traefik labels for routing - -Show, confirm, deploy, verify. First-run: Portainer will prompt for admin password setup. - -**Gate**: Portainer UI accessible, operator has set admin password. - ---- - -## PHASE 6 — Gitea - -**Goal**: Gitea + Postgres running at `https://git.`, with notes on state migration. - -This phase has two sub-modes: - -**6a — Fresh install**: generate `stacks/gitea.yml` with: -- Gitea service + named volume `gitea-data` -- Postgres service + named volume `gitea-db` -- Internal overlay network `gitea-internal` (Postgres not exposed to Traefik network) -- Traefik labels on Gitea only - -**6b — Migrate existing state** (if operator has an existing Gitea elsewhere): -- Stop source Gitea -- Dump Postgres: `pg_dump -U gitea gitea > gitea-backup.sql` -- Tar gitea data volume: `tar czf gitea-data.tar.gz /var/lib/docker/volumes/gitea-data` -- Use `scripts/migrate-volume.sh` to transfer and restore on new host -- Then deploy stack and verify - -Ask operator which sub-mode applies before proceeding. - -Show all files, confirm, deploy. After deploy, complete Gitea web installer. - -**Gate**: Gitea accessible, can create repo, SSH clone works. - ---- - -## PHASE 7 — Backup (Restic) - -**Goal**: daily automated backup of all named volumes to object storage. - -Ask operator for: -1. Backup destination (S3 / Backblaze B2 / SFTP / etc.) -2. Bucket name + credentials -3. Retention policy preference (default: 7 daily, 4 weekly, 6 monthly) - -Generate `playbooks/backup.yml` which: -1. Installs Restic on the host -2. Initialises the Restic repo (idempotent) -3. Creates a backup script at `/opt/restic/backup.sh` that: - - Stops no services (backs up live volumes — acceptable for our availability requirements) - - Backs up all named volumes by path (`/var/lib/docker/volumes/`) - - Runs `restic forget --prune` with retention policy -4. Installs a systemd timer (preferred over cron) to run daily at 02:00 - -Show all files, confirm, test with a manual run, verify snapshot appears: -```bash -restic -r snapshots -``` - -**Gate**: snapshot confirmed in remote repo. - ---- - -## PHASE 8 — Day-2 Ops: Adding a New Service - -**Goal**: operator understands the pattern for spinning up any new service cleanly. - -When the operator wants to add a new service, follow this checklist: - -1. **Ask**: what is the service, what image, what subdomain, does it need persistent storage? -2. **Generate** a `stacks/.yml` based on the template below -3. **Show** it, wait for approval -4. **Deploy**: `docker stack deploy -c stacks/.yml ` -5. **Verify**: check service is running, subdomain resolves, SSL cert issued -6. **Commit**: remind operator to commit the stack file to git - -**Standard stack template for a new service**: -```yaml -# stacks/.yml -# Service: -# Subdomain: . - -networks: - traefik-public: - external: true - -volumes: - -data: - -services: - : - image: : - networks: - - traefik-public - volumes: - - -data:/data - environment: - - SOME_VAR=value - deploy: - replicas: 1 - restart_policy: - condition: on-failure - delay: 5s - max_attempts: 3 - labels: - - "traefik.enable=true" - - "traefik.http.routers..rule=Host(`.`)" - - "traefik.http.routers..entrypoints=websecure" - - "traefik.http.routers..tls.certresolver=letsencrypt" - - "traefik.http.services..loadbalancer.server.port=" -``` - -**To remove a service cleanly**: -```bash -docker stack rm -# volumes are NOT removed automatically — data is safe -# to also remove volumes when sure: -docker volume rm -data -``` - ---- - -## State Migration Script Template - -Create `scripts/migrate-volume.sh`: +From a **fresh droplet** (Debian 13, SSH key injected): ```bash -#!/usr/bin/env bash -# migrate-volume.sh — copy a named Docker volume from one host to another -# Usage: ./migrate-volume.sh [ssh_key] -# -# Both hosts must have Docker installed. -# The volume must exist on the source host. -# A volume with the same name will be created on the dest host. +# 1. Accept new host keys +ssh-keygen -R box.cloud-elves.eu +ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts -set -euo pipefail +# 2. Bootstrap (OS hardening, Docker install) +ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml -VOLUME=$1 -SRC=$2 -DST=$3 -KEY=${4:-~/.ssh/id_ed25519} - -echo "==> Migrating volume '$VOLUME' from $SRC to $DST" - -echo "--> Creating volume on destination..." -ssh -i "$KEY" "$DST" "docker volume create $VOLUME" - -echo "--> Streaming data from source to destination..." -ssh -i "$KEY" "$SRC" \ - "docker run --rm -v $VOLUME:/data alpine tar czf - /data" \ - | ssh -i "$KEY" "$DST" \ - "docker run --rm -i -v $VOLUME:/data alpine tar xzf - -C /" - -echo "==> Done. Volume '$VOLUME' is available on $DST." +# 3. Deploy all services +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml ``` ---- +To **update services** after config changes: -## How to Re-deploy After a VPS Swap - -When you get a permanent VPS and want to move from the test one: - -1. Run `playbooks/backup.yml` on old host (final snapshot) -2. Provision new VPS — run `bootstrap.yml` (phases 0–3) -3. Deploy Traefik + Portainer (phases 4–5) -4. For each stateful service (Gitea etc.): use `migrate-volume.sh` to transfer volumes -5. Deploy service stacks -6. Point DNS to new IP -7. Verify SSL certs renew on new host -8. Decommission old VPS +```bash +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml +``` --- ## Quick Reference Commands ```bash -# Check all running services -docker service ls - -# View logs for a service -docker service logs -f - -# Update a service image -docker service update --image : _ - -# Re-deploy a stack (picks up config changes) -docker stack deploy -c stacks/.yml - -# Remove a stack (keeps volumes) -docker stack rm - -# List volumes -docker volume ls - -# Run ansible against host -ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml - -# Check ansible connectivity +# Check Ansible connectivity ansible -i inventory/hosts.yml all -m ping + +# Check running containers on box +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose ps" + +# View logs for a specific service +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose logs --tail 50" + +# Deploy static site +rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/ + +# Restart a single service +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose restart " ``` --- -## Begin +## Future Work -Start with **PHASE 0**. Check prerequisites on the local machine. Report findings and wait. +- **Backup setup** — Restic to object storage, systemd timer, retention policy +- **Security review** — firewall audit, container hardening, secret management (move secrets out of group_vars) +- **CI/CD** — automated deployments, documentation pipeline +- **Migration plan** — move to permanent VPS once company incorporation is complete +- **Portainer** — optional web UI for container management (low priority) diff --git a/ansible.cfg b/ansible.cfg new file mode 100644 index 0000000..f34572b --- /dev/null +++ b/ansible.cfg @@ -0,0 +1,3 @@ +[defaults] +roles_path = roles +inventory = inventory/hosts.yml diff --git a/backups/ce-backup-20260317-194405.tar b/backups/ce-backup-20260317-194405.tar new file mode 100644 index 0000000..813c034 Binary files /dev/null and b/backups/ce-backup-20260317-194405.tar differ diff --git a/credentials.md b/credentials.md new file mode 100644 index 0000000..fdad10f --- /dev/null +++ b/credentials.md @@ -0,0 +1,46 @@ +# Credentials + +## LLDAP (User Directory) + +- URL: https://ldap.cloud-elves.eu +- Admin username: `admin` +- Admin password: `cio3LFvUwUzLhO6iMXTAbQPq` +- Base DN: `dc=cloud-elves,dc=eu` +- LDAP port: 3890 (internal only) + +## Dex (OIDC Provider) + +- URL: https://auth.cloud-elves.eu +- Discovery: https://auth.cloud-elves.eu/.well-known/openid-configuration + +### Dex Clients +| Client | ID | Secret | +|---|---|---| +| Outline | `outline` | `6edfe427944e50856e65e87ebf62217389f72963e8e6fe16d19536f1d31ffb56` | +| Gitea | `gitea` | `nWotT1z2AziFbmzscORURXlZ` | + +## Gitea + +- URL: https://git.cloud-elves.eu +- SSH clone: `ssh://git@git.cloud-elves.eu:2222` +- Auth: via Dex SSO (or local admin account) + +### Gitea Postgres Database +- Host: `gitea-db` (internal) +- Database / User: `gitea` +- Password: `0cGpEZ0Ph2mdLbPJjA4kbR` + +## Outline (Wiki) + +- URL: https://docs.cloud-elves.eu +- Auth: via Dex SSO (login button: "Cloud Elves SSO") + +### Outline Postgres Database +- Host: `outline-db` (internal) +- Database / User: `outline` +- Password: `oZEdt71Jd2zkto2qWKj1MI7K` + +## Static Site + +- URL: https://cloud-elves.eu (also https://www.cloud-elves.eu) +- Deploy: `rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/` diff --git a/docs/future-plans.md b/docs/future-plans.md new file mode 100644 index 0000000..e0610db --- /dev/null +++ b/docs/future-plans.md @@ -0,0 +1,87 @@ +# Future Plans + +## Priority Overview + +| Priority | Item | Blocked on | +|---|---|---| +| High | Backup system (Restic) | Choosing backup destination | +| High | Security review | — | +| Medium | Migration to permanent VPS | Company incorporation | +| Medium | SMTP / Email | ProtonMail setup after incorporation | +| Medium | CI/CD pipeline | — | +| Low | Monitoring | — | +| Low | Group-based access control | Not needed yet (4 people) | +| Low | Move SSH to non-standard port | — | +| Low | Outline branding (remove badge) | Cosmetic, parked | + +## High Priority + +### Backup System (Restic) +- Daily automated backup of all Docker volumes to object storage +- Systemd timer (not cron) +- Retention policy: 7 daily, 4 weekly, 6 monthly +- Test restore procedure documented +- **Blocked on**: choosing a backup destination (S3 / Backblaze B2 / SFTP) + +### Security Review +- Firewall audit — verify only intended ports are exposed +- Container hardening — run containers as non-root where possible +- Secret management — move passwords/secrets out of `group_vars/all.yml` into Ansible Vault or environment files +- Review Dex/LLDAP security configuration +- Fail2ban rules review +- Check for unnecessary services or open ports +- HTTPS headers audit (HSTS, CSP, etc.) + +## Medium Priority + +### Migration to Permanent VPS + +Once company incorporation is complete and bank account is set up. + +```mermaid +sequenceDiagram + participant Old as Old VPS + participant Backup as Backup Storage + participant New as New VPS + participant DNS + + Old->>Backup: Final backup snapshot + Note over New: Provision Debian 13 + New->>New: Run bootstrap.yml + New->>New: Run deploy.yml + Backup->>New: Restore volumes + DNS->>New: Update A record to new IP + Note over New: Verify SSL certs renew + Note over Old: Decommission +``` + +### CI/CD Pipeline +- Automated deployment on git push +- Documentation pipeline (Markdown to Outline, or similar) +- Options to evaluate: Gitea Actions, Drone CI, or simple webhook-based deploy + +### SMTP / Email +- Set up ProtonMail for cloud-elves.eu (after incorporation) +- Configure Outline and Gitea to send transactional emails +- Update Let's Encrypt contact email from personal to company address + +## Low Priority + +### Monitoring +- Resource monitoring (CPU, memory, disk) +- Uptime monitoring for all services +- Options: simple solution like Uptime Kuma, or external service + +### Group-Based Access Control +- Configure Dex to pass LLDAP groups to applications +- Restrict certain services to specific groups (e.g., admin-only tools) +- Not needed now (4 people, everyone needs access everywhere) + +### Move SSH to Non-Standard Port +- Move host sshd from 22 to a high port (e.g., 2200) +- Reduces log noise from automated scanners +- Free up port 22 for Gitea SSH (nicer clone URLs) + +### Outline Branding +- Remove "Built with Outline" badge on public share pages +- Parked for now — cosmetic only diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..c81adfd --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,47 @@ +# Cloud Elves — Our Tools + +## What do we have? + +We run our own private server with a set of tools for the team. Everything is accessible from anywhere with a browser — just log in with your Cloud Elves account. + +## Your account + +You have one account that works across all our tools. Your login is managed centrally — if you can log into one tool, you can log into all of them. + +If you need an account or forgot your password, ask an admin. + +## Our tools + +### Documentation — https://docs.cloud-elves.eu + +This is where you're probably reading this. Our wiki for everything: internal docs, guides, meeting notes, public documentation — whatever we need to write down and share. + +- Log in to create and edit pages +- Some pages are public (anyone on the internet can read them) +- Private collections are only visible to the team + +### Git — https://git.cloud-elves.eu + +Our code lives here. This is like GitHub, but hosted by us. If you work with code, repositories, or anything version-controlled, this is where it goes. + +- Web interface for browsing code and pull requests +- SSH clone: `ssh://git@git.cloud-elves.eu:2222` + +### Website — https://cloud-elves.eu + +Our public website. Also available at https://www.cloud-elves.eu. + +### User Management (admin only) — https://ldap.cloud-elves.eu + +This is where admins create, edit, and remove team member accounts. Regular team members don't need to visit this — it's just for account administration. + +## How to log in + +1. Visit any of the tools above +2. Click the sign-in / SSO button +3. Enter your Cloud Elves username and password +4. You're in + +## Need help? + +Ask in the team chat, or reach out to whoever set up your account. diff --git a/docs/gitea-actions-guide.md b/docs/gitea-actions-guide.md new file mode 100644 index 0000000..ad2152d --- /dev/null +++ b/docs/gitea-actions-guide.md @@ -0,0 +1,232 @@ +# Gitea Actions — CI/CD Guide + +## What is this? + +Gitea Actions is our built-in CI/CD system. It lets you automate tasks when things happen in a git repository — like running tests on every push, deploying a site when you merge to main, or running a script on a schedule. + +If you've used GitHub Actions before, it's the same syntax. + +## How it works + +```mermaid +sequenceDiagram + participant Dev as Developer + participant Gitea as Gitea + participant Runner as Runner (box-runner) + participant Container as Job Container + + Dev->>Gitea: Push code + Gitea->>Runner: "There's a job to run" + Runner->>Container: Spin up temporary container + Container->>Container: Execute workflow steps + Container-->>Runner: Results + Runner-->>Gitea: Report status (pass/fail) + Note over Container: Container is cleaned up + Gitea-->>Dev: Show result in UI +``` + +Our runner (`box-runner`) runs on the same server as everything else. When a job triggers, it spins up a temporary Docker container, runs the steps, reports back, and cleans up. The runner itself is always running in the background, waiting for work. + +## Quick start + +### 1. Create a workflow file + +In any Gitea repository, create a file at `.gitea/workflows/` with a `.yml` extension: + +```yaml +# .gitea/workflows/hello.yml +name: Hello World +on: [push] + +jobs: + greet: + runs-on: ubuntu-latest + steps: + - run: echo "Hello from CI!" +``` + +### 2. Push it + +```bash +git add .gitea/workflows/hello.yml +git commit -m "Add CI workflow" +git push +``` + +### 3. Check the result + +Go to your repository on Gitea → **Actions** tab. You'll see the workflow run with its status. + +## Workflow syntax + +### When to run (triggers) + +```yaml +# Run on every push +on: [push] + +# Run on push to main only +on: + push: + branches: [main] + +# Run on pull request +on: [pull_request] + +# Run on a schedule (cron syntax) +on: + schedule: + - cron: "0 2 * * *" # Every day at 2 AM + +# Run manually from the UI +on: + workflow_dispatch: +``` + +### Available runners + +| Label | What it is | +|---|---| +| `ubuntu-latest` | Ubuntu container (default choice) | +| `ubuntu-24.04` | Ubuntu 24.04 specifically | +| `ubuntu-22.04` | Ubuntu 22.04 specifically | + +Use these in `runs-on`: + +```yaml +jobs: + my-job: + runs-on: ubuntu-latest +``` + +### Common steps + +**Check out the code:** +```yaml +steps: + - uses: actions/checkout@v4 +``` + +**Run a shell command:** +```yaml +steps: + - run: echo "Hello" +``` + +**Run multiple commands:** +```yaml +steps: + - run: | + echo "Step 1" + echo "Step 2" + ls -la +``` + +**Use environment variables:** +```yaml +jobs: + deploy: + runs-on: ubuntu-latest + env: + MY_VAR: "some value" + steps: + - run: echo $MY_VAR +``` + +**Use secrets (set in Gitea UI under repo Settings → Actions → Secrets):** +```yaml +steps: + - run: echo ${{ secrets.MY_SECRET }} +``` + +## Practical examples + +### Run tests on every push + +```yaml +name: Test +on: [push] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: npm install + - run: npm test +``` + +### Deploy static site on push to main + +```yaml +name: Deploy Site +on: + push: + branches: [main] + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Deploy via rsync + run: | + mkdir -p ~/.ssh + echo "${{ secrets.DEPLOY_KEY }}" > ~/.ssh/id_ed25519 + chmod 600 ~/.ssh/id_ed25519 + ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts + rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/ +``` + +> For this to work, add a `DEPLOY_KEY` secret in the repo settings containing a private SSH key that has access to the server. + +### Scheduled task (glorified cron) + +```yaml +name: Nightly Cleanup +on: + schedule: + - cron: "0 3 * * *" # 3 AM daily + +jobs: + cleanup: + runs-on: ubuntu-latest + steps: + - run: echo "Running nightly task..." + # Add your actual commands here +``` + +### Build and push a Docker image + +```yaml +name: Build Image +on: + push: + branches: [main] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: docker build -t my-app:latest . + - run: docker tag my-app:latest git.cloud-elves.eu/myorg/my-app:latest + # Push to Gitea's container registry if enabled +``` + +## Where to find things in the UI + +| What | Where | +|---|---| +| Workflow runs | Repository → **Actions** tab | +| Workflow files | Repository → `.gitea/workflows/` directory | +| Secrets | Repository → **Settings** → **Actions** → **Secrets** | +| Runner status | **Site Administration** → **Actions** → **Runners** (admin only) | + +## Tips + +- Workflows live in your repo as code — version controlled like everything else +- Each push shows a status check (green tick or red cross) next to the commit +- You can re-run failed workflows from the Actions tab +- Use `workflow_dispatch` trigger if you want a "run manually" button in the UI +- Keep workflows simple — if it's getting complex, it probably belongs in a script that the workflow just calls diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..67ac130 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,121 @@ +# Cloud Elves Infrastructure Overview + +## Architecture + +All services run on a single DigitalOcean VPS (`box.cloud-elves.eu`, Debian 13) managed by Ansible and deployed as Docker Compose containers. + +### Services + +| Service | URL | Purpose | +|---|---|---| +| Caddy | (internal) | Reverse proxy, automatic HTTPS via Let's Encrypt | +| Gitea | https://git.cloud-elves.eu | Git hosting, SSH clone on port 2222 | +| Outline | https://docs.cloud-elves.eu | Wiki and documentation | +| Dex | https://auth.cloud-elves.eu | OIDC identity provider | +| LLDAP | https://ldap.cloud-elves.eu | User directory with web UI | +| Static site | https://cloud-elves.eu | Company website (also www) | + +### Supporting Services (internal only) + +| Service | Used by | +|---|---| +| Postgres (gitea-db) | Gitea | +| Postgres (outline-db) | Outline | +| Redis | Outline | + +### How a request reaches your app + +```mermaid +sequenceDiagram + participant Browser + participant Caddy + participant App as Gitea / Outline / etc. + + Browser->>Caddy: HTTPS request (port 443) + Note over Caddy: Terminates TLS
Auto-issued Let's Encrypt cert + Caddy->>App: Forward to internal port + App-->>Caddy: Response + Caddy-->>Browser: HTTPS response +``` + +## Authentication + +All authentication flows through a centralized identity stack. + +### SSO Login Flow + +```mermaid +sequenceDiagram + participant User + participant App as Gitea / Outline + participant Dex as Dex (OIDC) + participant LLDAP as LLDAP (Directory) + + User->>App: Visit app + App->>Dex: Redirect to SSO + Dex->>User: Show login form + User->>Dex: Enter credentials + Dex->>LLDAP: Verify credentials + LLDAP-->>Dex: OK + Dex-->>App: Auth token + App-->>User: Logged in +``` + +### Onboarding a new team member + +1. Create their account in LLDAP (https://ldap.cloud-elves.eu) +2. They can immediately log into all services via SSO — accounts are auto-created on first login + +### Removing a team member + +1. Delete or disable the user in LLDAP +2. Their SSO sessions will expire (existing sessions may persist until token expiry) + +## Networking + +### Exposed Ports + +| Port | Protocol | Service | +|---|---|---| +| 22 | TCP | SSH (host access) | +| 80 | TCP | HTTP (redirects to HTTPS) | +| 443 | TCP + UDP | HTTPS (all web services) | +| 2222 | TCP | Gitea SSH (git clone) | + +All other communication happens over internal Docker networks. UFW firewall blocks everything else. + +### Internal Networks + +| Network | Connects | +|---|---| +| frontend | Caddy to all web-facing services | +| identity | Dex to LLDAP | +| gitea-backend | Gitea to its Postgres | +| outline-backend | Outline to its Postgres and Redis | + +## DNS Records + +| Record | Type | Points to | +|---|---|---| +| `cloud-elves.eu` | A | 188.166.60.12 | +| `box.cloud-elves.eu` | A | 188.166.60.12 | +| `www.cloud-elves.eu` | CNAME | box.cloud-elves.eu | +| `git.cloud-elves.eu` | CNAME | box.cloud-elves.eu | +| `auth.cloud-elves.eu` | CNAME | box.cloud-elves.eu | +| `ldap.cloud-elves.eu` | CNAME | box.cloud-elves.eu | +| `docs.cloud-elves.eu` | CNAME | box.cloud-elves.eu | + +## Data & Persistence + +All persistent data is in Docker named volumes under `/var/lib/docker/volumes/` on the host. + +| Volume | Contains | +|---|---| +| `caddy-data` | TLS certificates | +| `caddy-config` | Caddy runtime config | +| `lldap-data` | User directory (SQLite) | +| `dex-data` | Dex database (SQLite) | +| `gitea-data` | Git repositories, Gitea config | +| `gitea-db` | Gitea Postgres data | +| `outline-data` | Uploaded files/attachments | +| `outline-db` | Outline Postgres data | diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..081d9b4 --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,189 @@ +# Operational Runbook + +## Day-to-Day Operations + +### Check service health + +```bash +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose ps" +``` + +### View logs for a service + +```bash +# Replace with: caddy, gitea, outline, dex, lldap, gitea-db, outline-db, outline-redis +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose logs --tail 100" +``` + +### Restart a single service + +```bash +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose restart " +``` + +### Deploy config changes + +After editing any template in `roles/services/templates/`: + +```bash +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml +``` + +### Deploy static site + +From your local `cloud-elves.eu` repo: + +```bash +rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/ +``` + +## User Management + +### Onboarding flow + +```mermaid +sequenceDiagram + participant Admin + participant LLDAP + participant NewUser as New Team Member + participant App as Gitea / Outline + + Admin->>LLDAP: Create user account + LLDAP-->>Admin: Done + Admin->>NewUser: Share login details + NewUser->>App: Visit any app + App->>NewUser: Redirect to SSO + NewUser->>App: Log in with LLDAP credentials + Note over App: Account auto-created on first login +``` + +1. Go to https://ldap.cloud-elves.eu and log in as admin +2. Click "Create user" — fill in username, email, display name, password +3. Share credentials with the new team member +4. They can immediately log into Gitea and Outline via SSO + +### Removing a team member + +1. Go to https://ldap.cloud-elves.eu +2. Delete or disable the user + +## Adding a New Service + +### Checklist + +| Step | What | Where | +|---|---|---| +| 1 | Add container definition | `roles/services/templates/docker-compose.yml.j2` | +| 2 | Add reverse proxy entry | `roles/services/templates/Caddyfile.j2` | +| 3 | Add OIDC client (if SSO needed) | `roles/services/templates/dex-config.yml.j2` | +| 4 | Add variables (passwords, versions) | `inventory/group_vars/all.yml` | +| 5 | Create DNS CNAME | DNS provider: `` → `box.cloud-elves.eu` | +| 6 | Deploy | `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml` | +| 7 | Verify | Caddy auto-issues TLS cert on first request | + +### How it connects + +```mermaid +sequenceDiagram + participant Dev as Developer + participant Ansible + participant Box as box.cloud-elves.eu + participant Caddy + participant LE as Let's Encrypt + + Dev->>Ansible: Run deploy.yml + Ansible->>Box: Template configs + docker compose up + Box-->>Ansible: Services running + Note over Caddy: New subdomain in Caddyfile + Caddy->>LE: Request certificate + LE-->>Caddy: Certificate issued + Note over Caddy: Service live with HTTPS +``` + +## Full Rebuild from Scratch + +### When you need this + +- Droplet is destroyed +- Migrating to a new server +- Starting fresh + +### Procedure + +```bash +# 1. Provision new Debian 13 droplet with SSH key (id_ed25519_ce) + +# 2. Update DNS: point box.cloud-elves.eu A record to new IP +# Flush local DNS: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder + +# 3. Accept new host keys +ssh-keygen -R box.cloud-elves.eu +ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts + +# 4. Bootstrap (OS hardening, Docker install) +ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml + +# 5. Deploy all services +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml + +# 6. Restore data from backup (once backup system is in place) +``` + +### After a fresh deploy you'll need to + +- Set up a Gitea admin account (first registered user via SSO becomes admin) +- Re-create users in LLDAP +- All Outline content will be empty (unless restored from backup) + +## Upgrading a Service + +### Procedure + +1. Check the current version in `inventory/group_vars/all.yml` +2. Look up the latest version on Docker Hub +3. Update the version variable +4. Run: `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml` +5. Check logs for migration output: `docker compose logs --tail 50` +6. Verify the service is healthy + +### Example: upgrading Outline + +```bash +# In inventory/group_vars/all.yml, change: +# outline_version: "1.6.0" +# to: +# outline_version: "1.7.0" + +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml +``` + +## Troubleshooting + +### Service won't start + +```bash +ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ + -a "cd /opt/services && docker compose logs --tail 50" +``` + +### SSL certificate not issued + +Caddy issues certs on first request. If a cert is missing: +1. Verify DNS resolves: `dig +short .cloud-elves.eu` +2. Check Caddy logs for ACME errors +3. Restart Caddy: `docker compose restart caddy` + +### Can't SSH to the box + +1. Check if DNS is stale: `dig +short box.cloud-elves.eu` +2. Flush DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` +3. If host key changed (rebuild): `ssh-keygen -R box.cloud-elves.eu` + +### Ansible can't connect + +1. Verify SSH works: `ssh -i ~/.ssh/id_ed25519_ce root@box.cloud-elves.eu` +2. Check known_hosts for stale entries +3. Run: `ansible -i inventory/hosts.yml all -m ping` diff --git a/inventory/group_vars/all.yml b/inventory/group_vars/all.yml new file mode 100644 index 0000000..ee300ba --- /dev/null +++ b/inventory/group_vars/all.yml @@ -0,0 +1,34 @@ +# Primary domain — subdomains will be created off this +domain: "cloud-elves.eu" + +# Hostname for the server +server_hostname: "box" + +# Email for Let's Encrypt certificate notifications (used by Caddy automatically) +acme_email: "maarten@lambda-m.nl" + +# Docker image versions — pin these, update deliberately +gitea_version: "1.21" +postgres_version: "16-alpine" +outline_version: "1.6.0" +redis_version: "7-alpine" + +# LDAP base DN +ldap_base_dn: "dc=cloud-elves,dc=eu" + +# Gitea database password +gitea_db_password: "0cGpEZ0Ph2mdLbPJjA4kbR" + +# LLDAP secrets +lldap_jwt_secret: "371bc1a40f93dc2229fbaa5e080ac323" +lldap_key_seed: "1a5873dd243b29f60043d2e69ec9b7f7" +lldap_admin_password: "cio3LFvUwUzLhO6iMXTAbQPq" + +# Dex OIDC client secrets +dex_outline_client_secret: "6edfe427944e50856e65e87ebf62217389f72963e8e6fe16d19536f1d31ffb56" +dex_gitea_client_secret: "nWotT1z2AziFbmzscORURXlZ" + +# Outline secrets +outline_secret_key: "72a81e9516e29c249ad8543866b1275a34b6a15014527f81d1becab363c82fb4" +outline_utils_secret: "9219100ad29e330c0a7ea8032adc0c5831de14e487a7c3c5e050b757d022f5da" +outline_db_password: "oZEdt71Jd2zkto2qWKj1MI7K" diff --git a/inventory/hosts.yml b/inventory/hosts.yml new file mode 100644 index 0000000..ed0add0 --- /dev/null +++ b/inventory/hosts.yml @@ -0,0 +1,7 @@ +all: + hosts: + box: + ansible_host: box.cloud-elves.eu + ansible_user: root + ansible_ssh_private_key_file: ~/.ssh/id_ed25519_ce + ansible_python_interpreter: /usr/bin/python3 diff --git a/playbooks/bootstrap.yml b/playbooks/bootstrap.yml new file mode 100644 index 0000000..071531c --- /dev/null +++ b/playbooks/bootstrap.yml @@ -0,0 +1,9 @@ +--- +- name: Bootstrap and harden the VPS + hosts: all + become: true + roles: + - role: common + tags: [common] + - role: docker + tags: [docker] diff --git a/playbooks/deploy.yml b/playbooks/deploy.yml new file mode 100644 index 0000000..61ec99b --- /dev/null +++ b/playbooks/deploy.yml @@ -0,0 +1,7 @@ +--- +- name: Deploy services + hosts: all + become: true + roles: + - role: services + tags: [services] diff --git a/roles/common/handlers/main.yml b/roles/common/handlers/main.yml new file mode 100644 index 0000000..cad74d8 --- /dev/null +++ b/roles/common/handlers/main.yml @@ -0,0 +1,5 @@ +--- +- name: Restart sshd + ansible.builtin.service: + name: ssh + state: restarted diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml new file mode 100644 index 0000000..caa1359 --- /dev/null +++ b/roles/common/tasks/main.yml @@ -0,0 +1,127 @@ +--- +# Disable cloud-init so it never regenerates SSH keys or resets config on reboot +- name: Disable cloud-init + ansible.builtin.file: + path: /etc/cloud/cloud-init.disabled + state: touch + mode: "0644" + +- name: Find active cloud-init services + ansible.builtin.shell: systemctl list-unit-files 'cloud-*' --no-legend | awk '{print $1}' + register: cloud_init_services + changed_when: false + +- name: Stop cloud-init services + ansible.builtin.service: + name: "{{ item }}" + state: stopped + enabled: false + loop: "{{ cloud_init_services.stdout_lines }}" + when: cloud_init_services.stdout_lines | length > 0 + +# Set the hostname to match our inventory name +- name: Set hostname + ansible.builtin.hostname: + name: "{{ server_hostname }}" + +# Update /etc/hosts to include the new hostname +- name: Update /etc/hosts with hostname + ansible.builtin.lineinfile: + path: /etc/hosts + regexp: '^127\.0\.1\.1' + line: "127.0.1.1 {{ server_hostname }}.{{ domain }} {{ server_hostname }}" + state: present + +# Refresh apt cache before installing anything +- name: Update apt cache + ansible.builtin.apt: + update_cache: true + cache_valid_time: 3600 + +# Upgrade all installed packages to latest +- name: Upgrade all packages + ansible.builtin.apt: + upgrade: dist + autoremove: true + +# Install essential tools and security packages +- name: Install base packages + ansible.builtin.apt: + name: + - curl + - git + - htop + - rsync + - ufw + - fail2ban + - unattended-upgrades + - apt-listchanges + state: present + +# Allow SSH before enabling the firewall (don't lock ourselves out) +- name: UFW — allow SSH + community.general.ufw: + rule: allow + port: "22" + proto: tcp + +# Allow HTTP for Caddy +- name: UFW — allow HTTP + community.general.ufw: + rule: allow + port: "80" + proto: tcp + +# Allow HTTPS for Caddy +- name: UFW — allow HTTPS + community.general.ufw: + rule: allow + port: "443" + proto: tcp + +# Allow Gitea SSH +- name: UFW — allow Gitea SSH + community.general.ufw: + rule: allow + port: "2222" + proto: tcp + +# Default deny incoming, allow outgoing, then enable +- name: UFW — enable with default deny incoming + community.general.ufw: + state: enabled + default: deny + direction: incoming + +- name: UFW — default allow outgoing + community.general.ufw: + default: allow + direction: outgoing + +# Enable automatic security updates +- name: Enable unattended-upgrades + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + content: | + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Unattended-Upgrade "1"; + APT::Periodic::AutocleanInterval "7"; + mode: "0644" + +# Disable root password login — key-only auth +- name: Disable SSH password authentication + ansible.builtin.lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PasswordAuthentication' + line: 'PasswordAuthentication no' + state: present + notify: Restart sshd + +# Disable root login via password (permit key-based only) +- name: Set SSH to permit root login with key only + ansible.builtin.lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PermitRootLogin' + line: 'PermitRootLogin prohibit-password' + state: present + notify: Restart sshd diff --git a/roles/docker/tasks/main.yml b/roles/docker/tasks/main.yml new file mode 100644 index 0000000..332a957 --- /dev/null +++ b/roles/docker/tasks/main.yml @@ -0,0 +1,50 @@ +--- +# Install Docker prerequisites +- name: Install Docker dependencies + ansible.builtin.apt: + name: + - ca-certificates + - curl + - gnupg + state: present + +# Add Docker's official GPG key +- name: Create keyrings directory + ansible.builtin.file: + path: /etc/apt/keyrings + state: directory + mode: "0755" + +- name: Add Docker GPG key + ansible.builtin.get_url: + url: https://download.docker.com/linux/debian/gpg + dest: /etc/apt/keyrings/docker.asc + mode: "0644" + +# Add Docker apt repository for Debian +- name: Add Docker apt repository + ansible.builtin.apt_repository: + repo: >- + deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] + https://download.docker.com/linux/debian + {{ ansible_facts['distribution_release'] }} stable + state: present + filename: docker + +# Install Docker Engine + Compose plugin +- name: Install Docker Engine + ansible.builtin.apt: + name: + - docker-ce + - docker-ce-cli + - containerd.io + - docker-compose-plugin + state: present + update_cache: true + +# Ensure Docker service is running and enabled at boot +- name: Enable and start Docker + ansible.builtin.service: + name: docker + state: started + enabled: true diff --git a/roles/services/tasks/main.yml b/roles/services/tasks/main.yml new file mode 100644 index 0000000..b89d593 --- /dev/null +++ b/roles/services/tasks/main.yml @@ -0,0 +1,157 @@ +--- +# Create the services directory on the host +- name: Create services directory + ansible.builtin.file: + path: /opt/services + state: directory + mode: "0755" + +# Create a placeholder index page for the static site +- name: Create www directory + ansible.builtin.file: + path: /opt/www + state: directory + mode: "0755" + +- name: Create placeholder index.html + ansible.builtin.copy: + dest: /opt/www/index.html + content: | + + + Cloud Elves +

cloud-elves.eu

Coming soon.

+ + mode: "0644" + force: false + +# Template config files +- name: Deploy Caddyfile + ansible.builtin.template: + src: Caddyfile.j2 + dest: /opt/services/Caddyfile + mode: "0644" + register: caddyfile_changed + +- name: Deploy Dex config + ansible.builtin.template: + src: dex-config.yml.j2 + dest: /opt/services/dex-config.yml + mode: "0644" + +# Check if we already have a runner token saved +- name: Check for existing runner token + ansible.builtin.stat: + path: /opt/services/.runner-token + register: runner_token_file + +# Use a placeholder token for first deploy (runner won't start yet) +- name: Set runner token from file if exists + ansible.builtin.slurp: + src: /opt/services/.runner-token + register: runner_token_content + when: runner_token_file.stat.exists + +- name: Set runner token variable + ansible.builtin.set_fact: + gitea_runner_token: "{{ runner_token_content.content | b64decode | trim }}" + when: runner_token_file.stat.exists + +- name: Set placeholder runner token for first deploy + ansible.builtin.set_fact: + gitea_runner_token: "placeholder" + when: not runner_token_file.stat.exists + +# Deploy compose file +- name: Deploy docker-compose.yml + ansible.builtin.template: + src: docker-compose.yml.j2 + dest: /opt/services/docker-compose.yml + mode: "0644" + +# Start everything (runner will fail to register with placeholder token — that's OK) +- name: Start all services + ansible.builtin.command: + cmd: docker compose up -d --remove-orphans + chdir: /opt/services + changed_when: true + +# Reload Caddy if Caddyfile changed (zero-downtime) +- name: Reload Caddy config + ansible.builtin.command: + cmd: docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile + chdir: /opt/services + when: caddyfile_changed.changed + changed_when: true + +# Wait for Gitea to be ready +- name: Wait for Gitea to be ready + ansible.builtin.command: + cmd: docker compose exec -T -u git gitea gitea admin user list + chdir: /opt/services + register: gitea_ready + retries: 10 + delay: 5 + until: gitea_ready.rc == 0 + changed_when: false + +# --- Gitea OIDC setup --- + +- name: Check if Dex auth source exists in Gitea + ansible.builtin.shell: + cmd: docker compose exec -T -u git gitea gitea admin auth list 2>&1 | grep -q dex + chdir: /opt/services + register: gitea_auth_check + changed_when: false + failed_when: false + +- name: Add Dex OIDC auth source to Gitea + ansible.builtin.command: + cmd: > + docker compose exec -T -u git gitea gitea admin auth add-oauth + --name dex + --provider openidConnect + --key gitea + --secret {{ dex_gitea_client_secret }} + --auto-discover-url https://auth.{{ domain }}/.well-known/openid-configuration + --scopes "openid profile email" + chdir: /opt/services + when: gitea_auth_check.rc != 0 + changed_when: true + +# --- Gitea Actions runner setup --- + +- name: Generate runner registration token + ansible.builtin.shell: + cmd: docker compose exec -T -u git gitea gitea actions generate-runner-token 2>/dev/null + chdir: /opt/services + register: runner_token_result + changed_when: false + when: not runner_token_file.stat.exists + +- name: Save runner token + ansible.builtin.copy: + content: "{{ runner_token_result.stdout | trim }}" + dest: /opt/services/.runner-token + mode: "0600" + when: not runner_token_file.stat.exists + +# Re-template compose with real token and restart runner +- name: Set real runner token + ansible.builtin.set_fact: + gitea_runner_token: "{{ runner_token_result.stdout | trim }}" + when: not runner_token_file.stat.exists + +- name: Re-deploy docker-compose with real runner token + ansible.builtin.template: + src: docker-compose.yml.j2 + dest: /opt/services/docker-compose.yml + mode: "0644" + when: not runner_token_file.stat.exists + +- name: Restart runner with real token + ansible.builtin.command: + cmd: docker compose up -d --remove-orphans + chdir: /opt/services + when: not runner_token_file.stat.exists + changed_when: true diff --git a/roles/services/templates/Caddyfile.j2 b/roles/services/templates/Caddyfile.j2 new file mode 100644 index 0000000..cc040ea --- /dev/null +++ b/roles/services/templates/Caddyfile.j2 @@ -0,0 +1,29 @@ +{ + email {{ acme_email }} +} + +# Gitea +git.{{ domain }} { + reverse_proxy gitea:3000 +} + +# LLDAP web UI +ldap.{{ domain }} { + reverse_proxy lldap:17170 +} + +# Dex OIDC provider +auth.{{ domain }} { + reverse_proxy dex:5556 +} + +# Outline wiki +docs.{{ domain }} { + reverse_proxy outline:3000 +} + +# Static site +{{ domain }}, www.{{ domain }} { + root * /srv/www + file_server +} diff --git a/roles/services/templates/dex-config.yml.j2 b/roles/services/templates/dex-config.yml.j2 new file mode 100644 index 0000000..e6c5fe9 --- /dev/null +++ b/roles/services/templates/dex-config.yml.j2 @@ -0,0 +1,51 @@ +# Dex OIDC provider configuration +issuer: https://auth.{{ domain }} + +storage: + type: sqlite3 + config: + file: /var/dex/dex.db + +web: + http: 0.0.0.0:5556 + +connectors: + - type: ldap + id: lldap + name: LLDAP + config: + host: lldap:3890 + insecureNoSSL: true + bindDN: uid=admin,ou=people,{{ ldap_base_dn }} + bindPW: "{{ lldap_admin_password }}" + userSearch: + baseDN: ou=people,{{ ldap_base_dn }} + filter: "(objectClass=person)" + username: uid + idAttr: uid + emailAttr: mail + nameAttr: displayName + preferredUsernameAttr: uid + groupSearch: + baseDN: ou=groups,{{ ldap_base_dn }} + filter: "(objectClass=groupOfUniqueNames)" + userMatchers: + - userAttr: DN + groupAttr: uniqueMember + nameAttr: cn + +staticClients: + - id: outline + name: Outline + secret: "{{ dex_outline_client_secret }}" + redirectURIs: + - https://docs.{{ domain }}/auth/oidc.callback + + - id: gitea + name: Gitea + secret: "{{ dex_gitea_client_secret }}" + redirectURIs: + - https://git.{{ domain }}/user/oauth2/dex/callback + +oauth2: + skipApprovalScreen: true diff --git a/roles/services/templates/docker-compose.yml.j2 b/roles/services/templates/docker-compose.yml.j2 new file mode 100644 index 0000000..3933689 --- /dev/null +++ b/roles/services/templates/docker-compose.yml.j2 @@ -0,0 +1,182 @@ +# /opt/services/docker-compose.yml +# All services for box.cloud-elves.eu + +services: + + # --- Reverse Proxy --- + + caddy: + image: caddy:2 + restart: unless-stopped + ports: + - "80:80" + - "443:443" + - "443:443/udp" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile:ro + - caddy-data:/data + - caddy-config:/config + # Static site served from host directory + - /opt/www:/srv/www:ro + networks: + - frontend + + # --- Identity --- + + lldap: + image: lldap/lldap:stable + restart: unless-stopped + environment: + - LLDAP_JWT_SECRET={{ lldap_jwt_secret }} + - LLDAP_KEY_SEED={{ lldap_key_seed }} + - LLDAP_LDAP_USER_PASS={{ lldap_admin_password }} + - LLDAP_LDAP_BASE_DN={{ ldap_base_dn }} + - LLDAP_HTTP_URL=https://ldap.{{ domain }} + volumes: + - lldap-data:/data + networks: + - frontend + - identity + + dex: + image: dexidp/dex:latest + restart: unless-stopped + command: ["dex", "serve", "/etc/dex/config.yaml"] + volumes: + - ./dex-config.yml:/etc/dex/config.yaml:ro + - dex-data:/var/dex + networks: + - frontend + - identity + depends_on: + - lldap + + # --- Gitea --- + + gitea: + image: gitea/gitea:{{ gitea_version }} + restart: unless-stopped + ports: + # Gitea SSH on host port 2222 + - "2222:22" + environment: + - GITEA__database__DB_TYPE=postgres + - GITEA__database__HOST=gitea-db:5432 + - GITEA__database__NAME=gitea + - GITEA__database__USER=gitea + - GITEA__database__PASSWD={{ gitea_db_password }} + - GITEA__server__DOMAIN=git.{{ domain }} + - GITEA__server__SSH_DOMAIN=git.{{ domain }} + - GITEA__server__SSH_PORT=2222 + - GITEA__server__ROOT_URL=https://git.{{ domain }}/ + # Auto-create Gitea accounts on first SSO login + - GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION=true + - GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION=true + - GITEA__oauth2_client__ACCOUNT_LINKING=auto + # Enable Gitea Actions + - GITEA__actions__ENABLED=true + volumes: + - gitea-data:/data + - /etc/localtime:/etc/localtime:ro + networks: + - frontend + - gitea-backend + depends_on: + - gitea-db + + gitea-db: + image: postgres:{{ postgres_version }} + restart: unless-stopped + environment: + - POSTGRES_DB=gitea + - POSTGRES_USER=gitea + - POSTGRES_PASSWORD={{ gitea_db_password }} + volumes: + - gitea-db:/var/lib/postgresql/data + networks: + - gitea-backend + + # --- Gitea Runner --- + + gitea-runner: + image: gitea/act_runner:latest + restart: unless-stopped + environment: + - GITEA_INSTANCE_URL=http://gitea:3000 + - GITEA_RUNNER_REGISTRATION_TOKEN={{ gitea_runner_token }} + - GITEA_RUNNER_NAME=box-runner + volumes: + # Runner needs Docker socket to spin up job containers + - /var/run/docker.sock:/var/run/docker.sock + - gitea-runner-data:/data + networks: + - frontend + depends_on: + - gitea + + # --- Outline --- + + outline: + image: outlinewiki/outline:{{ outline_version }} + restart: unless-stopped + environment: + - URL=https://docs.{{ domain }} + - SECRET_KEY={{ outline_secret_key }} + - UTILS_SECRET={{ outline_utils_secret }} + - DATABASE_URL=postgres://outline:{{ outline_db_password }}@outline-db:5432/outline + - PGSSLMODE=disable + - REDIS_URL=redis://outline-redis:6379 + - FILE_STORAGE=local + - FILE_STORAGE_LOCAL_ROOT_DIR=/var/lib/outline/data + - FORCE_HTTPS=false + - OIDC_CLIENT_ID=outline + - OIDC_CLIENT_SECRET={{ dex_outline_client_secret }} + - OIDC_AUTH_URI=https://auth.{{ domain }}/auth + - OIDC_TOKEN_URI=https://auth.{{ domain }}/token + - OIDC_USERINFO_URI=https://auth.{{ domain }}/userinfo + - OIDC_DISPLAY_NAME=Cloud Elves SSO + - OIDC_SCOPES=openid profile email offline_access + volumes: + - outline-data:/var/lib/outline/data + networks: + - frontend + - outline-backend + depends_on: + - outline-db + - outline-redis + - dex + + outline-db: + image: postgres:{{ postgres_version }} + restart: unless-stopped + environment: + - POSTGRES_DB=outline + - POSTGRES_USER=outline + - POSTGRES_PASSWORD={{ outline_db_password }} + volumes: + - outline-db:/var/lib/postgresql/data + networks: + - outline-backend + + outline-redis: + image: redis:{{ redis_version }} + restart: unless-stopped + networks: + - outline-backend + +networks: + frontend: + identity: + gitea-backend: + outline-backend: + +volumes: + caddy-data: + caddy-config: + lldap-data: + dex-data: + gitea-data: + gitea-db: + gitea-runner-data: + outline-data: + outline-db: diff --git a/scripts/backup.sh b/scripts/backup.sh new file mode 100755 index 0000000..673cefa --- /dev/null +++ b/scripts/backup.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# backup.sh — Create a full backup of all Cloud Elves services +# +# Usage: ./scripts/backup.sh [destination_dir] +# +# Creates a timestamped tarball containing volume-level backups of all services. +# Run from the ce-infra repo root directory. +# Requires SSH access to box.cloud-elves.eu as root. + +set -euo pipefail + +HOST="root@box.cloud-elves.eu" +SSH_KEY="${SSH_KEY:-$HOME/.ssh/id_ed25519_ce}" +SSH="ssh -i $SSH_KEY $HOST" +SCP="scp -i $SSH_KEY" +TIMESTAMP=$(date +%Y%m%d-%H%M%S) +DEST_DIR="${1:-./backups}" +BACKUP_NAME="ce-backup-${TIMESTAMP}" +REMOTE_TMP="/tmp/${BACKUP_NAME}" + +echo "=== Cloud Elves Backup — ${TIMESTAMP} ===" +echo "" + +# Create local destination +mkdir -p "${DEST_DIR}" + +# Create remote temp directory +echo "[1/7] Preparing..." +$SSH "mkdir -p ${REMOTE_TMP}" + +# Stop services for consistent backup (databases stay up for pg_dump) +echo "[2/7] Stopping application services for consistent backup..." +$SSH "cd /opt/services && docker compose stop gitea outline dex lldap gitea-runner caddy" + +# Gitea: pg_dump + data volume +echo "[3/7] Backing up Gitea (database + data)..." +$SSH "cd /opt/services && docker compose exec -T gitea-db pg_dump -U gitea gitea | gzip > ${REMOTE_TMP}/gitea-db.sql.gz" +$SSH "docker run --rm -v services_gitea-data:/data -v ${REMOTE_TMP}:/backup alpine tar czf /backup/gitea-data.tar.gz -C /data ." + +# Outline: pg_dump + data volume +echo "[4/7] Backing up Outline (database + uploads)..." +$SSH "cd /opt/services && docker compose exec -T outline-db pg_dump -U outline outline | gzip > ${REMOTE_TMP}/outline-db.sql.gz" +$SSH "docker run --rm -v services_outline-data:/data -v ${REMOTE_TMP}:/backup alpine tar czf /backup/outline-data.tar.gz -C /data ." + +# LLDAP: SQLite volume +echo "[5/7] Backing up LLDAP (user directory)..." +$SSH "docker run --rm -v services_lldap-data:/data -v ${REMOTE_TMP}:/backup alpine tar czf /backup/lldap-data.tar.gz -C /data ." + +# Dex: SQLite volume +echo "[6/7] Backing up Dex (auth database)..." +$SSH "docker run --rm -v services_dex-data:/dex -v ${REMOTE_TMP}:/backup alpine tar czf /backup/dex-data.tar.gz -C /dex ." + +# Restart services +echo "[7/7] Restarting services..." +$SSH "cd /opt/services && docker compose up -d" + +# Bundle everything and download +echo "" +echo "Downloading backup..." +$SSH "cd /tmp && tar cf ${BACKUP_NAME}.tar -C ${REMOTE_TMP} ." +$SCP "${HOST}:/tmp/${BACKUP_NAME}.tar" "${DEST_DIR}/${BACKUP_NAME}.tar" + +# Clean up remote +$SSH "rm -rf ${REMOTE_TMP} /tmp/${BACKUP_NAME}.tar" + +# Show result +BACKUP_SIZE=$(du -h "${DEST_DIR}/${BACKUP_NAME}.tar" | cut -f1) +echo "" +echo "=== Backup complete ===" +echo "File: ${DEST_DIR}/${BACKUP_NAME}.tar (${BACKUP_SIZE})" +echo "" +echo "Contents:" +tar tf "${DEST_DIR}/${BACKUP_NAME}.tar" | sed 's/^/ /' +echo "" +echo "Downtime was ~30 seconds while services were stopped for consistency." diff --git a/scripts/restore.sh b/scripts/restore.sh new file mode 100755 index 0000000..f21fa26 --- /dev/null +++ b/scripts/restore.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# restore.sh — Restore Cloud Elves services from a backup +# +# Usage: ./scripts/restore.sh +# +# Prerequisites: +# - Target server already bootstrapped: ansible-playbook playbooks/bootstrap.yml +# - Services already deployed: ansible-playbook playbooks/deploy.yml +# - This script stops services, replaces data, and restarts +# +# Run from the ce-infra repo root directory. +# Requires SSH access to box.cloud-elves.eu as root. + +set -euo pipefail + +if [ $# -eq 0 ]; then + echo "Usage: $0 " + echo "Example: $0 ./backups/ce-backup-20260317-180000.tar" + exit 1 +fi + +BACKUP_FILE="$1" +HOST="root@box.cloud-elves.eu" +SSH_KEY="${SSH_KEY:-$HOME/.ssh/id_ed25519_ce}" +SSH="ssh -i $SSH_KEY $HOST" +SCP="scp -i $SSH_KEY" +REMOTE_TMP="/tmp/ce-restore" + +if [ ! -f "${BACKUP_FILE}" ]; then + echo "Error: Backup file not found: ${BACKUP_FILE}" + exit 1 +fi + +echo "=== Cloud Elves Restore ===" +echo "Backup: ${BACKUP_FILE}" +echo "" +echo "Contents:" +tar tf "${BACKUP_FILE}" | sed 's/^/ /' +echo "" +echo "WARNING: This will stop all services and replace their data." +echo "Press Enter to continue, Ctrl+C to abort." +read -r + +# Upload backup +echo "[1/7] Uploading backup to server..." +$SSH "mkdir -p ${REMOTE_TMP}" +$SCP "${BACKUP_FILE}" "${HOST}:${REMOTE_TMP}/backup.tar" +$SSH "cd ${REMOTE_TMP} && tar xf backup.tar && rm backup.tar" + +# Stop all services +echo "[2/7] Stopping all services..." +$SSH "cd /opt/services && docker compose down" + +# Restore LLDAP volume +echo "[3/7] Restoring LLDAP (user directory)..." +$SSH "docker volume rm services_lldap-data 2>/dev/null || true && docker volume create services_lldap-data" +$SSH "docker run --rm -v services_lldap-data:/data -v ${REMOTE_TMP}:/backup alpine tar xzf /backup/lldap-data.tar.gz -C /data" + +# Restore Dex volume +echo "[4/7] Restoring Dex (auth database)..." +$SSH "docker volume rm services_dex-data 2>/dev/null || true && docker volume create services_dex-data" +$SSH "docker run --rm -v services_dex-data:/dex -v ${REMOTE_TMP}:/backup alpine tar xzf /backup/dex-data.tar.gz -C /dex" + +# Restore Gitea data volume +echo "[5/7] Restoring Gitea (data + database)..." +$SSH "docker volume rm services_gitea-data 2>/dev/null || true && docker volume create services_gitea-data" +$SSH "docker run --rm -v services_gitea-data:/data -v ${REMOTE_TMP}:/backup alpine tar xzf /backup/gitea-data.tar.gz -C /data" + +# Start databases only so we can restore into them +$SSH "cd /opt/services && docker compose up -d gitea-db outline-db" +echo " Waiting for databases to start..." +sleep 5 + +# Restore Gitea database +$SSH "cd /opt/services && docker compose exec -T gitea-db dropdb -U gitea --if-exists gitea" +$SSH "cd /opt/services && docker compose exec -T gitea-db createdb -U gitea gitea" +$SSH "gunzip -c ${REMOTE_TMP}/gitea-db.sql.gz | docker compose -f /opt/services/docker-compose.yml exec -T gitea-db psql -U gitea gitea" + +# Restore Outline data volume +echo "[6/7] Restoring Outline (uploads + database)..." +$SSH "docker volume rm services_outline-data 2>/dev/null || true && docker volume create services_outline-data" +$SSH "docker run --rm -v services_outline-data:/data -v ${REMOTE_TMP}:/backup alpine tar xzf /backup/outline-data.tar.gz -C /data" + +# Restore Outline database +$SSH "cd /opt/services && docker compose exec -T outline-db dropdb -U outline --if-exists outline" +$SSH "cd /opt/services && docker compose exec -T outline-db createdb -U outline outline" +$SSH "gunzip -c ${REMOTE_TMP}/outline-db.sql.gz | docker compose -f /opt/services/docker-compose.yml exec -T outline-db psql -U outline outline" + +# Start everything +echo "[7/7] Starting all services..." +$SSH "cd /opt/services && docker compose up -d" + +# Wait for Gitea and regenerate hooks +echo " Waiting for services to start..." +sleep 10 + +echo " Regenerating Gitea hooks..." +$SSH "cd /opt/services && docker compose exec -T -u git gitea gitea admin regenerate hooks" || echo " (no repos to regenerate hooks for)" + +# Clean up +$SSH "rm -rf ${REMOTE_TMP}" + +echo "" +echo "=== Restore complete ===" +echo "" +echo "Please verify:" +echo " - https://ldap.cloud-elves.eu — LLDAP (users should be there)" +echo " - https://auth.cloud-elves.eu — Dex (SSO should work)" +echo " - https://git.cloud-elves.eu — Gitea (repos, users)" +echo " - https://docs.cloud-elves.eu — Outline (documents)" +echo " - https://cloud-elves.eu — Static site (deploy via rsync separately)" +echo "" +echo "Note: Users will need to log in again (SSO sessions were reset)."