maarten
/
ce-infra
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add backup/restore docs, update all docs for consistency 

- Add docs/backup-restore.md with tested backup & restore procedure
- Add .gitignore to exclude backup files from repo
- Update CLAUDE.md: add gitea runner to stack, add scripts to layout,
  update future work (backup & CI/CD now done)
- Update docs/overview.md: add runner to services, clarify volume names
  and backup status, add gitea-runner-data volume
- Update docs/future-plans.md: move backup & CI/CD to completed section,
  rename Restic section to automated remote backups

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
M. van der Woord 2026-03-19 22:18:45 +01:00
parent 72da841c2b
commit b0e0f64439
5 changed files with 173 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
# Backup files — keep these off the repo, store separately
backups/

22
CLAUDE.md
View File

@ -28,13 +28,14 @@ Ansible (local machine)
├── LLDAP — lightweight LDAP user directory (ldap.cloud-elves.eu) ├── LLDAP — lightweight LDAP user directory (ldap.cloud-elves.eu)
├── Dex — OIDC provider, backed by LLDAP (auth.cloud-elves.eu) ├── Dex — OIDC provider, backed by LLDAP (auth.cloud-elves.eu)
├── Gitea — git hosting (git.cloud-elves.eu), SSH on port 2222 ├── Gitea — git hosting (git.cloud-elves.eu), SSH on port 2222
├── Gitea Runner — CI/CD runner for Gitea Actions (act_runner)
├── Outline — wiki/docs (docs.cloud-elves.eu) ├── Outline — wiki/docs (docs.cloud-elves.eu)
└── Static site — cloud-elves.eu / www.cloud-elves.eu (rsync deploy) └── Static site — cloud-elves.eu / www.cloud-elves.eu (rsync deploy)
``` ```
Supporting containers (internal only): Gitea Postgres, Outline Postgres, Outline Redis. Supporting containers (internal only): Gitea Postgres, Outline Postgres, Outline Redis.
Backups: not yet configured (Restic → object storage planned). Backups: manual via `scripts/backup.sh` and `scripts/restore.sh`. Tested end-to-end.
--- ---
@ -64,6 +65,9 @@ ce-infra/
├── docs/ ← infrastructure documentation ├── docs/ ← infrastructure documentation
│ ├── overview.md ← architecture and service overview │ ├── overview.md ← architecture and service overview
│ ├── runbook.md ← operational procedures │ ├── runbook.md ← operational procedures
│ ├── backup-restore.md ← backup and restore procedures
│ ├── gitea-actions-guide.md ← CI/CD usage guide
│ ├── getting-started.md ← non-technical team onboarding
│ └── future-plans.md ← planned work and improvements │ └── future-plans.md ← planned work and improvements
├── inventory/ ├── inventory/
│ ├── hosts.yml ← VPS connection details │ ├── hosts.yml ← VPS connection details
@ -78,9 +82,12 @@ ce-infra/
│ ├── docker-compose.yml.j2 ← all containers │ ├── docker-compose.yml.j2 ← all containers
│ ├── Caddyfile.j2 ← reverse proxy config │ ├── Caddyfile.j2 ← reverse proxy config
│ └── dex-config.yml.j2 ← Dex OIDC config │ └── dex-config.yml.j2 ← Dex OIDC config
└── playbooks/ ├── playbooks/
├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) │ ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run)
└── deploy.yml ← deploys/updates all services │ └── deploy.yml ← deploys/updates all services
└── scripts/
├── backup.sh ← full backup to local tarball
└── restore.sh ← restore from backup onto fresh deploy
``` ```
--- ---
@ -178,8 +185,9 @@ ansible -i inventory/hosts.yml all -m ansible.builtin.shell \
## Future Work ## Future Work
- **Backup setup** — Restic to object storage, systemd timer, retention policy - **Automated remote backups** — Restic to object storage, systemd timer, retention policy
- **Security review** — firewall audit, container hardening, secret management (move secrets out of group_vars) - **Security review** — firewall audit, container hardening, secret management (move secrets out of group_vars)
- **CI/CD** — automated deployments, documentation pipeline
- **Migration plan** — move to permanent VPS once company incorporation is complete - **Migration plan** — move to permanent VPS once company incorporation is complete
- **Portainer** — optional web UI for container management (low priority) - **SMTP** — ProtonMail for cloud-elves.eu after incorporation
See `docs/future-plans.md` for the full prioritized list.

122
docs/backup-restore.md Normal file
View File

@ -0,0 +1,122 @@
# Backup & Restore
## Overview
We have a tested backup and restore process that captures all state from our server and can fully restore it onto a fresh VPS. This has been verified end-to-end: backup, rebuild droplet, bootstrap, deploy, restore — everything comes back including repos, documents, users, and SSO.
## What gets backed up
| Component | What's captured | How |
|---|---|---|
| Gitea | Repositories, config, users, issues | Postgres dump + data volume |
| Outline | All documents, collections, settings | Postgres dump + file uploads volume |
| LLDAP | All user accounts and groups | SQLite database file |
| Dex | Auth tokens, signing keys | SQLite database file |
## What is NOT backed up
| Component | Why |
|---|---|
| Static site (cloud-elves.eu) | Deployed via rsync from your local repo — redeploy after restore |
| TLS certificates | Caddy re-issues them automatically in seconds |
| Gitea runner token | Regenerated automatically during deploy |
| Docker images | Re-pulled automatically on startup |
## Running a backup
From your laptop, in the `ce-infra` repo directory:
```bash
./scripts/backup.sh
```
This will:
1. Stop application services (~30 seconds downtime)
2. Dump both Postgres databases
3. Copy all data volumes (Gitea, Outline, LLDAP, Dex)
4. Restart all services
5. Download a single `.tar` file to `./backups/`
The backup file is named with a timestamp, e.g. `ce-backup-20260317-195615.tar`.
## Full restore procedure
### When you need this
- Droplet was destroyed or rebuilt
- Migrating to a new server
- Disaster recovery
### Steps
```mermaid
sequenceDiagram
participant You
participant VPS as New VPS
participant Backup as Backup File
You->>VPS: Provision Debian 13 droplet
You->>You: Fix SSH known hosts
You->>VPS: Run bootstrap.yml
You->>VPS: Run deploy.yml (will partially fail — expected)
Backup->>VPS: Run restore.sh
Note over VPS: All data restored
You->>VPS: Verify all services
```
**1. Provision a new Debian 13 droplet** with your SSH key (`id_ed25519_ce`).
**2. Update DNS** if the IP changed — point `box.cloud-elves.eu` A record to the new IP.
**3. Fix SSH known hosts:**
```bash
ssh-keygen -R box.cloud-elves.eu
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts
```
**4. Bootstrap the server:**
```bash
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml
```
**5. Deploy services:**
```bash
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
```
This will partially fail (Gitea isn't installed yet). That's expected — the restore will fix it.
**6. Restore from backup:**
```bash
./scripts/restore.sh ./backups/ce-backup-YYYYMMDD-HHMMSS.tar
```
**7. Redeploy static site** (not included in backup):
```bash
cd /path/to/cloud-elves.eu && rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/
```
**8. Verify everything:**
| Service | URL | What to check |
|---|---|---|
| LLDAP | https://ldap.cloud-elves.eu | Users are there |
| Dex | https://auth.cloud-elves.eu | SSO login works |
| Gitea | https://git.cloud-elves.eu | Repos and code are there |
| Outline | https://docs.cloud-elves.eu | Documents are there |
| Static site | https://cloud-elves.eu | Page loads |
Users will need to log in again — SSO sessions are reset during restore.
## Important notes
- **SECRET_KEY and UTILS_SECRET** (for Outline) must be preserved. These are stored in `inventory/group_vars/all.yml`. If you lose these, encrypted data in Outline becomes permanently inaccessible.
- **Keep this repo safe** — it contains all the configuration and secrets needed to rebuild everything.
- **Keep backups off the server** — the backup script downloads the file to your laptop. Store copies somewhere safe.
- **Test restores periodically** — a backup you haven't tested is not a backup.

34
docs/future-plans.md
View File

@ -2,25 +2,36 @@
## Priority Overview ## Priority Overview
| Priority | Item | Blocked on | | Priority | Item | Status |
|---|---|---| |---|---|---|
| High | Backup system (Restic) | Choosing backup destination | | High | Backup & restore | Done — `scripts/backup.sh` and `scripts/restore.sh`, tested end-to-end |
| High | Security review | — | | High | CI/CD | Done — Gitea Actions with act_runner, see `docs/gitea-actions-guide.md` |
| Medium | Migration to permanent VPS | Company incorporation | | High | Security review | Not started |
| Medium | SMTP / Email | ProtonMail setup after incorporation | | Medium | Automated remote backups (Restic) | Blocked on choosing backup destination |
| Medium | CI/CD pipeline | — | | Medium | Migration to permanent VPS | Blocked on company incorporation |
| Medium | SMTP / Email | Blocked on ProtonMail setup after incorporation |
| Low | Monitoring | — | | Low | Monitoring | — |
| Low | Group-based access control | Not needed yet (4 people) | | Low | Group-based access control | Not needed yet (4 people) |
| Low | Move SSH to non-standard port | — | | Low | Move SSH to non-standard port | — |
| Low | Outline branding (remove badge) | Cosmetic, parked | | Low | Outline branding (remove badge) | Cosmetic, parked |
## Completed
### Backup & Restore
- Manual backup/restore scripts in `scripts/` — tested end-to-end
- See `docs/backup-restore.md` for full procedure
### CI/CD — Gitea Actions
- Gitea Actions enabled with act_runner
- GitHub Actions compatible workflow syntax
- See `docs/gitea-actions-guide.md` for usage guide
## High Priority ## High Priority
### Backup System (Restic) ### Automated Remote Backups (Restic)
- Daily automated backup of all Docker volumes to object storage - Automate the existing backup script to run daily and push to remote storage
- Systemd timer (not cron) - Systemd timer (not cron)
- Retention policy: 7 daily, 4 weekly, 6 monthly - Retention policy: 7 daily, 4 weekly, 6 monthly
- Test restore procedure documented
- **Blocked on**: choosing a backup destination (S3 / Backblaze B2 / SFTP) - **Blocked on**: choosing a backup destination (S3 / Backblaze B2 / SFTP)
### Security Review ### Security Review
@ -55,11 +66,6 @@ sequenceDiagram
Note over Old: Decommission Note over Old: Decommission
``` ```
### CI/CD Pipeline
- Automated deployment on git push
- Documentation pipeline (Markdown to Outline, or similar)
- Options to evaluate: Gitea Actions, Drone CI, or simple webhook-based deploy
### SMTP / Email ### SMTP / Email
- Set up ProtonMail for cloud-elves.eu (after incorporation) - Set up ProtonMail for cloud-elves.eu (after incorporation)
- Configure Outline and Gitea to send transactional emails - Configure Outline and Gitea to send transactional emails

26
docs/overview.md
View File

@ -19,9 +19,10 @@ All services run on a single DigitalOcean VPS (`box.cloud-elves.eu`, Debian 13)
| Service | Used by | | Service | Used by |
|---|---| |---|---|
| Gitea Runner (act_runner) | Gitea Actions CI/CD — runs pipelines |
| Postgres (gitea-db) | Gitea | | Postgres (gitea-db) | Gitea |
| Postgres (outline-db) | Outline | | Postgres (outline-db) | Outline |
| Redis | Outline | | Redis | Outline (caching/sessions) |
### How a request reaches your app ### How a request reaches your app
@ -107,15 +108,16 @@ All other communication happens over internal Docker networks. UFW firewall bloc
## Data & Persistence ## Data & Persistence
All persistent data is in Docker named volumes under `/var/lib/docker/volumes/` on the host. All persistent data is in Docker named volumes under `/var/lib/docker/volumes/` on the host. Docker Compose prefixes them with `services_` (e.g. `services_gitea-data`).
| Volume | Contains | | Volume | Contains | Backed up? |
|---|---| |---|---|---|
| `caddy-data` | TLS certificates | | `lldap-data` | User directory (SQLite) | Yes |
| `caddy-config` | Caddy runtime config | | `dex-data` | Dex auth database (SQLite) | Yes |
| `lldap-data` | User directory (SQLite) | | `gitea-data` | Git repositories, Gitea config | Yes |
| `dex-data` | Dex database (SQLite) | | `gitea-db` | Gitea Postgres data | Yes |
| `gitea-data` | Git repositories, Gitea config | | `outline-data` | Uploaded files/attachments | Yes |
| `gitea-db` | Gitea Postgres data | | `outline-db` | Outline Postgres data | Yes |
| `outline-data` | Uploaded files/attachments | | `caddy-data` | TLS certificates | No — re-issued automatically |
| `outline-db` | Outline Postgres data | | `caddy-config` | Caddy runtime config | No — regenerated from Caddyfile |
| `gitea-runner-data` | Runner registration state | No — regenerated on deploy |