ce-infra/docs/runbook.md

# Operational Runbook

## Day-to-Day Operations

### Check service health

```bash
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose ps"
```

### View logs for a service

```bash
# Replace <service> with: caddy, gitea, outline, dex, lldap, gitea-db, outline-db, outline-redis
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose logs <service> --tail 100"
```

### Restart a single service

```bash
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose restart <service>"
```

### Deploy config changes

After editing any template in `roles/services/templates/`:

```bash
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
```

### Deploy static site

From your local `cloud-elves.eu` repo:

```bash
rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/
```

## User Management

### Onboarding flow

```mermaid
sequenceDiagram
    participant Admin
    participant LLDAP
    participant NewUser as New Team Member
    participant App as Gitea / Outline

    Admin->>LLDAP: Create user account
    LLDAP-->>Admin: Done
    Admin->>NewUser: Share login details
    NewUser->>App: Visit any app
    App->>NewUser: Redirect to SSO
    NewUser->>App: Log in with LLDAP credentials
    Note over App: Account auto-created on first login
```

1. Go to https://ldap.cloud-elves.eu and log in as admin
2. Click "Create user" — fill in username, email, display name, password
3. Share credentials with the new team member
4. They can immediately log into Gitea and Outline via SSO

### Removing a team member

1. Go to https://ldap.cloud-elves.eu
2. Delete or disable the user

## Adding a New Service

### Checklist

| Step | What | Where |
|---|---|---|
| 1 | Add container definition | `roles/services/templates/docker-compose.yml.j2` |
| 2 | Add reverse proxy entry | `roles/services/templates/Caddyfile.j2` |
| 3 | Add OIDC client (if SSO needed) | `roles/services/templates/dex-config.yml.j2` |
| 4 | Add variables (passwords, versions) | `inventory/group_vars/all.yml` |
| 5 | Create DNS CNAME | DNS provider: `<subdomain>` → `box.cloud-elves.eu` |
| 6 | Deploy | `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml` |
| 7 | Verify | Caddy auto-issues TLS cert on first request |

### How it connects

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Ansible
    participant Box as box.cloud-elves.eu
    participant Caddy
    participant LE as Let's Encrypt

    Dev->>Ansible: Run deploy.yml
    Ansible->>Box: Template configs + docker compose up
    Box-->>Ansible: Services running
    Note over Caddy: New subdomain in Caddyfile
    Caddy->>LE: Request certificate
    LE-->>Caddy: Certificate issued
    Note over Caddy: Service live with HTTPS
```

## Full Rebuild from Scratch

### When you need this

- Droplet is destroyed
- Migrating to a new server
- Starting fresh

### Procedure

```bash
# 1. Provision new Debian 13 droplet with SSH key (id_ed25519_ce)

# 2. Update DNS: point box.cloud-elves.eu A record to new IP
#    Flush local DNS: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

# 3. Accept new host keys
ssh-keygen -R box.cloud-elves.eu
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts

# 4. Bootstrap (OS hardening, Docker install)
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml

# 5. Deploy all services
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

# 6. Restore data from backup (once backup system is in place)
```

### After a fresh deploy you'll need to

- Set up a Gitea admin account (first registered user via SSO becomes admin)
- Re-create users in LLDAP
- All Outline content will be empty (unless restored from backup)

## Upgrading a Service

### Procedure

1. Check the current version in `inventory/group_vars/all.yml`
2. Look up the latest version on Docker Hub
3. Update the version variable
4. Run: `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml`
5. Check logs for migration output: `docker compose logs <service> --tail 50`
6. Verify the service is healthy

### Example: upgrading Outline

```bash
# In inventory/group_vars/all.yml, change:
#   outline_version: "1.6.0"
# to:
#   outline_version: "1.7.0"

ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
```

## Troubleshooting

### Service won't start

```bash
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
  -a "cd /opt/services && docker compose logs <service> --tail 50"
```

### SSL certificate not issued

Caddy issues certs on first request. If a cert is missing:
1. Verify DNS resolves: `dig +short <subdomain>.cloud-elves.eu`
2. Check Caddy logs for ACME errors
3. Restart Caddy: `docker compose restart caddy`

### Can't SSH to the box

1. Check if DNS is stale: `dig +short box.cloud-elves.eu`
2. Flush DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
3. If host key changed (rebuild): `ssh-keygen -R box.cloud-elves.eu`

### Ansible can't connect

1. Verify SSH works: `ssh -i ~/.ssh/id_ed25519_ce root@box.cloud-elves.eu`
2. Check known_hosts for stale entries
3. Run: `ansible -i inventory/hosts.yml all -m ping`