190 lines
5.1 KiB
Markdown
190 lines
5.1 KiB
Markdown
# Operational Runbook
|
|
|
|
## Day-to-Day Operations
|
|
|
|
### Check service health
|
|
|
|
```bash
|
|
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
|
|
-a "cd /opt/services && docker compose ps"
|
|
```
|
|
|
|
### View logs for a service
|
|
|
|
```bash
|
|
# Replace <service> with: caddy, gitea, outline, dex, lldap, gitea-db, outline-db, outline-redis
|
|
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
|
|
-a "cd /opt/services && docker compose logs <service> --tail 100"
|
|
```
|
|
|
|
### Restart a single service
|
|
|
|
```bash
|
|
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
|
|
-a "cd /opt/services && docker compose restart <service>"
|
|
```
|
|
|
|
### Deploy config changes
|
|
|
|
After editing any template in `roles/services/templates/`:
|
|
|
|
```bash
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
|
|
```
|
|
|
|
### Deploy static site
|
|
|
|
From your local `cloud-elves.eu` repo:
|
|
|
|
```bash
|
|
rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/
|
|
```
|
|
|
|
## User Management
|
|
|
|
### Onboarding flow
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Admin
|
|
participant LLDAP
|
|
participant NewUser as New Team Member
|
|
participant App as Gitea / Outline
|
|
|
|
Admin->>LLDAP: Create user account
|
|
LLDAP-->>Admin: Done
|
|
Admin->>NewUser: Share login details
|
|
NewUser->>App: Visit any app
|
|
App->>NewUser: Redirect to SSO
|
|
NewUser->>App: Log in with LLDAP credentials
|
|
Note over App: Account auto-created on first login
|
|
```
|
|
|
|
1. Go to https://ldap.cloud-elves.eu and log in as admin
|
|
2. Click "Create user" — fill in username, email, display name, password
|
|
3. Share credentials with the new team member
|
|
4. They can immediately log into Gitea and Outline via SSO
|
|
|
|
### Removing a team member
|
|
|
|
1. Go to https://ldap.cloud-elves.eu
|
|
2. Delete or disable the user
|
|
|
|
## Adding a New Service
|
|
|
|
### Checklist
|
|
|
|
| Step | What | Where |
|
|
|---|---|---|
|
|
| 1 | Add container definition | `roles/services/templates/docker-compose.yml.j2` |
|
|
| 2 | Add reverse proxy entry | `roles/services/templates/Caddyfile.j2` |
|
|
| 3 | Add OIDC client (if SSO needed) | `roles/services/templates/dex-config.yml.j2` |
|
|
| 4 | Add variables (passwords, versions) | `inventory/group_vars/all.yml` |
|
|
| 5 | Create DNS CNAME | DNS provider: `<subdomain>` → `box.cloud-elves.eu` |
|
|
| 6 | Deploy | `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml` |
|
|
| 7 | Verify | Caddy auto-issues TLS cert on first request |
|
|
|
|
### How it connects
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Dev as Developer
|
|
participant Ansible
|
|
participant Box as box.cloud-elves.eu
|
|
participant Caddy
|
|
participant LE as Let's Encrypt
|
|
|
|
Dev->>Ansible: Run deploy.yml
|
|
Ansible->>Box: Template configs + docker compose up
|
|
Box-->>Ansible: Services running
|
|
Note over Caddy: New subdomain in Caddyfile
|
|
Caddy->>LE: Request certificate
|
|
LE-->>Caddy: Certificate issued
|
|
Note over Caddy: Service live with HTTPS
|
|
```
|
|
|
|
## Full Rebuild from Scratch
|
|
|
|
### When you need this
|
|
|
|
- Droplet is destroyed
|
|
- Migrating to a new server
|
|
- Starting fresh
|
|
|
|
### Procedure
|
|
|
|
```bash
|
|
# 1. Provision new Debian 13 droplet with SSH key (id_ed25519_ce)
|
|
|
|
# 2. Update DNS: point box.cloud-elves.eu A record to new IP
|
|
# Flush local DNS: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
|
|
|
|
# 3. Accept new host keys
|
|
ssh-keygen -R box.cloud-elves.eu
|
|
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts
|
|
|
|
# 4. Bootstrap (OS hardening, Docker install)
|
|
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml
|
|
|
|
# 5. Deploy all services
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
|
|
|
|
# 6. Restore data from backup (once backup system is in place)
|
|
```
|
|
|
|
### After a fresh deploy you'll need to
|
|
|
|
- Set up a Gitea admin account (first registered user via SSO becomes admin)
|
|
- Re-create users in LLDAP
|
|
- All Outline content will be empty (unless restored from backup)
|
|
|
|
## Upgrading a Service
|
|
|
|
### Procedure
|
|
|
|
1. Check the current version in `inventory/group_vars/all.yml`
|
|
2. Look up the latest version on Docker Hub
|
|
3. Update the version variable
|
|
4. Run: `ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml`
|
|
5. Check logs for migration output: `docker compose logs <service> --tail 50`
|
|
6. Verify the service is healthy
|
|
|
|
### Example: upgrading Outline
|
|
|
|
```bash
|
|
# In inventory/group_vars/all.yml, change:
|
|
# outline_version: "1.6.0"
|
|
# to:
|
|
# outline_version: "1.7.0"
|
|
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Service won't start
|
|
|
|
```bash
|
|
ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
|
|
-a "cd /opt/services && docker compose logs <service> --tail 50"
|
|
```
|
|
|
|
### SSL certificate not issued
|
|
|
|
Caddy issues certs on first request. If a cert is missing:
|
|
1. Verify DNS resolves: `dig +short <subdomain>.cloud-elves.eu`
|
|
2. Check Caddy logs for ACME errors
|
|
3. Restart Caddy: `docker compose restart caddy`
|
|
|
|
### Can't SSH to the box
|
|
|
|
1. Check if DNS is stale: `dig +short box.cloud-elves.eu`
|
|
2. Flush DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
|
|
3. If host key changed (rebuild): `ssh-keygen -R box.cloud-elves.eu`
|
|
|
|
### Ansible can't connect
|
|
|
|
1. Verify SSH works: `ssh -i ~/.ssh/id_ed25519_ce root@box.cloud-elves.eu`
|
|
2. Check known_hosts for stale entries
|
|
3. Run: `ansible -i inventory/hosts.yml all -m ping`
|