ce-infra/docs/runbook.md

Operational Runbook

Day-to-Day Operations

Check service health

ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose ps"

View logs for a service

# Replace <service> with: caddy, gitea, outline, dex, lldap, gitea-db, outline-db, outline-redis
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose logs <service> --tail 100"

Restart a single service

ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose restart <service>"

Deploy config changes

After editing any template in roles/services/templates/:

ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

Deploy static site

From your local cloud-elves.eu repo:

rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/

User Management

Onboarding flow

sequenceDiagram
    participant Admin
    participant LLDAP
    participant NewUser as New Team Member
    participant App as Gitea / Outline

    Admin->>LLDAP: Create user account
    LLDAP-->>Admin: Done
    Admin->>NewUser: Share login details
    NewUser->>App: Visit any app
    App->>NewUser: Redirect to SSO
    NewUser->>App: Log in with LLDAP credentials
    Note over App: Account auto-created on first login

1. Go to https://ldap.cloud-elves.eu and log in as admin
2. Click "Create user" — fill in username, email, display name, password
3. Share credentials with the new team member
4. They can immediately log into Gitea and Outline via SSO

Removing a team member

1. Go to https://ldap.cloud-elves.eu
2. Delete or disable the user

Adding a New Service

Checklist

Step	What	Where
1	Add container definition	roles/services/templates/docker-compose.yml.j2
2	Add reverse proxy entry	roles/services/templates/Caddyfile.j2
3	Add OIDC client (if SSO needed)	roles/services/templates/dex-config.yml.j2
4	Add variables (passwords, versions)	inventory/group_vars/all.yml
5	Create DNS CNAME	DNS provider: <subdomain> → box.cloud-elves.eu
6	Deploy	ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
7	Verify	Caddy auto-issues TLS cert on first request

How it connects

sequenceDiagram
    participant Dev as Developer
    participant Ansible
    participant Box as box.cloud-elves.eu
    participant Caddy
    participant LE as Let's Encrypt

    Dev->>Ansible: Run deploy.yml
    Ansible->>Box: Template configs + docker compose up
    Box-->>Ansible: Services running
    Note over Caddy: New subdomain in Caddyfile
    Caddy->>LE: Request certificate
    LE-->>Caddy: Certificate issued
    Note over Caddy: Service live with HTTPS

Full Rebuild from Scratch

When you need this

• Droplet is destroyed
• Migrating to a new server
• Starting fresh

Procedure

# 1. Provision new Debian 13 droplet with SSH key (id_ed25519_ce)

# 2. Update DNS: point box.cloud-elves.eu A record to new IP
#    Flush local DNS: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

# 3. Accept new host keys
ssh-keygen -R box.cloud-elves.eu
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts

# 4. Bootstrap (OS hardening, Docker install)
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml

# 5. Deploy all services
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

# 6. Restore data from backup (once backup system is in place)

After a fresh deploy you'll need to

• Set up a Gitea admin account (first registered user via SSO becomes admin)
• Re-create users in LLDAP
• All Outline content will be empty (unless restored from backup)

Upgrading a Service

Procedure

1. Check the current version in inventory/group_vars/all.yml
2. Look up the latest version on Docker Hub
3. Update the version variable
4. Run: ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
5. Check logs for migration output: docker compose logs <service> --tail 50
6. Verify the service is healthy

Example: upgrading Outline

# In inventory/group_vars/all.yml, change:
#   outline_version: "1.6.0"
# to:
#   outline_version: "1.7.0"

ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

Troubleshooting

Service won't start

ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose logs <service> --tail 50"

SSL certificate not issued

Caddy issues certs on first request. If a cert is missing:

1. Verify DNS resolves: dig +short <subdomain>.cloud-elves.eu
2. Check Caddy logs for ACME errors
3. Restart Caddy: docker compose restart caddy

Can't SSH to the box

1. Check if DNS is stale: dig +short box.cloud-elves.eu
2. Flush DNS: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
3. If host key changed (rebuild): ssh-keygen -R box.cloud-elves.eu

Ansible can't connect

1. Verify SSH works: ssh -i ~/.ssh/id_ed25519_ce root@box.cloud-elves.eu
2. Check known_hosts for stale entries
3. Run: ansible -i inventory/hosts.yml all -m ping