From b0e0f6443932021dceb80c7bf60c6de87602daac Mon Sep 17 00:00:00 2001 From: "M. van der Woord" Date: Thu, 19 Mar 2026 22:18:45 +0100 Subject: [PATCH] Add backup/restore docs, update all docs for consistency - Add docs/backup-restore.md with tested backup & restore procedure - Add .gitignore to exclude backup files from repo - Update CLAUDE.md: add gitea runner to stack, add scripts to layout, update future work (backup & CI/CD now done) - Update docs/overview.md: add runner to services, clarify volume names and backup status, add gitea-runner-data volume - Update docs/future-plans.md: move backup & CI/CD to completed section, rename Restic section to automated remote backups Co-Authored-By: Claude Opus 4.6 (1M context) --- .gitignore | 2 + CLAUDE.md | 22 +++++--- docs/backup-restore.md | 122 +++++++++++++++++++++++++++++++++++++++++ docs/future-plans.md | 34 +++++++----- docs/overview.md | 26 +++++---- 5 files changed, 173 insertions(+), 33 deletions(-) create mode 100644 .gitignore create mode 100644 docs/backup-restore.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..05fb9a2 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +# Backup files — keep these off the repo, store separately +backups/ diff --git a/CLAUDE.md b/CLAUDE.md index bdd0785..6882d0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,13 +28,14 @@ Ansible (local machine) ├── LLDAP — lightweight LDAP user directory (ldap.cloud-elves.eu) ├── Dex — OIDC provider, backed by LLDAP (auth.cloud-elves.eu) ├── Gitea — git hosting (git.cloud-elves.eu), SSH on port 2222 + ├── Gitea Runner — CI/CD runner for Gitea Actions (act_runner) ├── Outline — wiki/docs (docs.cloud-elves.eu) └── Static site — cloud-elves.eu / www.cloud-elves.eu (rsync deploy) ``` Supporting containers (internal only): Gitea Postgres, Outline Postgres, Outline Redis. -Backups: not yet configured (Restic → object storage planned). +Backups: manual via `scripts/backup.sh` and `scripts/restore.sh`. Tested end-to-end. --- @@ -64,6 +65,9 @@ ce-infra/ ├── docs/ ← infrastructure documentation │ ├── overview.md ← architecture and service overview │ ├── runbook.md ← operational procedures +│ ├── backup-restore.md ← backup and restore procedures +│ ├── gitea-actions-guide.md ← CI/CD usage guide +│ ├── getting-started.md ← non-technical team onboarding │ └── future-plans.md ← planned work and improvements ├── inventory/ │ ├── hosts.yml ← VPS connection details @@ -78,9 +82,12 @@ ce-infra/ │ ├── docker-compose.yml.j2 ← all containers │ ├── Caddyfile.j2 ← reverse proxy config │ └── dex-config.yml.j2 ← Dex OIDC config -└── playbooks/ - ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) - └── deploy.yml ← deploys/updates all services +├── playbooks/ +│ ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) +│ └── deploy.yml ← deploys/updates all services +└── scripts/ + ├── backup.sh ← full backup to local tarball + └── restore.sh ← restore from backup onto fresh deploy ``` --- @@ -178,8 +185,9 @@ ansible -i inventory/hosts.yml all -m ansible.builtin.shell \ ## Future Work -- **Backup setup** — Restic to object storage, systemd timer, retention policy +- **Automated remote backups** — Restic to object storage, systemd timer, retention policy - **Security review** — firewall audit, container hardening, secret management (move secrets out of group_vars) -- **CI/CD** — automated deployments, documentation pipeline - **Migration plan** — move to permanent VPS once company incorporation is complete -- **Portainer** — optional web UI for container management (low priority) +- **SMTP** — ProtonMail for cloud-elves.eu after incorporation + +See `docs/future-plans.md` for the full prioritized list. diff --git a/docs/backup-restore.md b/docs/backup-restore.md new file mode 100644 index 0000000..6adf527 --- /dev/null +++ b/docs/backup-restore.md @@ -0,0 +1,122 @@ +# Backup & Restore + +## Overview + +We have a tested backup and restore process that captures all state from our server and can fully restore it onto a fresh VPS. This has been verified end-to-end: backup, rebuild droplet, bootstrap, deploy, restore — everything comes back including repos, documents, users, and SSO. + +## What gets backed up + +| Component | What's captured | How | +|---|---|---| +| Gitea | Repositories, config, users, issues | Postgres dump + data volume | +| Outline | All documents, collections, settings | Postgres dump + file uploads volume | +| LLDAP | All user accounts and groups | SQLite database file | +| Dex | Auth tokens, signing keys | SQLite database file | + +## What is NOT backed up + +| Component | Why | +|---|---| +| Static site (cloud-elves.eu) | Deployed via rsync from your local repo — redeploy after restore | +| TLS certificates | Caddy re-issues them automatically in seconds | +| Gitea runner token | Regenerated automatically during deploy | +| Docker images | Re-pulled automatically on startup | + +## Running a backup + +From your laptop, in the `ce-infra` repo directory: + +```bash +./scripts/backup.sh +``` + +This will: +1. Stop application services (~30 seconds downtime) +2. Dump both Postgres databases +3. Copy all data volumes (Gitea, Outline, LLDAP, Dex) +4. Restart all services +5. Download a single `.tar` file to `./backups/` + +The backup file is named with a timestamp, e.g. `ce-backup-20260317-195615.tar`. + +## Full restore procedure + +### When you need this + +- Droplet was destroyed or rebuilt +- Migrating to a new server +- Disaster recovery + +### Steps + +```mermaid +sequenceDiagram + participant You + participant VPS as New VPS + participant Backup as Backup File + + You->>VPS: Provision Debian 13 droplet + You->>You: Fix SSH known hosts + You->>VPS: Run bootstrap.yml + You->>VPS: Run deploy.yml (will partially fail — expected) + Backup->>VPS: Run restore.sh + Note over VPS: All data restored + You->>VPS: Verify all services +``` + +**1. Provision a new Debian 13 droplet** with your SSH key (`id_ed25519_ce`). + +**2. Update DNS** if the IP changed — point `box.cloud-elves.eu` A record to the new IP. + +**3. Fix SSH known hosts:** + +```bash +ssh-keygen -R box.cloud-elves.eu +sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder +ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts +``` + +**4. Bootstrap the server:** + +```bash +ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml +``` + +**5. Deploy services:** + +```bash +ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml +``` + +This will partially fail (Gitea isn't installed yet). That's expected — the restore will fix it. + +**6. Restore from backup:** + +```bash +./scripts/restore.sh ./backups/ce-backup-YYYYMMDD-HHMMSS.tar +``` + +**7. Redeploy static site** (not included in backup): + +```bash +cd /path/to/cloud-elves.eu && rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/ +``` + +**8. Verify everything:** + +| Service | URL | What to check | +|---|---|---| +| LLDAP | https://ldap.cloud-elves.eu | Users are there | +| Dex | https://auth.cloud-elves.eu | SSO login works | +| Gitea | https://git.cloud-elves.eu | Repos and code are there | +| Outline | https://docs.cloud-elves.eu | Documents are there | +| Static site | https://cloud-elves.eu | Page loads | + +Users will need to log in again — SSO sessions are reset during restore. + +## Important notes + +- **SECRET_KEY and UTILS_SECRET** (for Outline) must be preserved. These are stored in `inventory/group_vars/all.yml`. If you lose these, encrypted data in Outline becomes permanently inaccessible. +- **Keep this repo safe** — it contains all the configuration and secrets needed to rebuild everything. +- **Keep backups off the server** — the backup script downloads the file to your laptop. Store copies somewhere safe. +- **Test restores periodically** — a backup you haven't tested is not a backup. diff --git a/docs/future-plans.md b/docs/future-plans.md index e0610db..db9241a 100644 --- a/docs/future-plans.md +++ b/docs/future-plans.md @@ -2,25 +2,36 @@ ## Priority Overview -| Priority | Item | Blocked on | +| Priority | Item | Status | |---|---|---| -| High | Backup system (Restic) | Choosing backup destination | -| High | Security review | — | -| Medium | Migration to permanent VPS | Company incorporation | -| Medium | SMTP / Email | ProtonMail setup after incorporation | -| Medium | CI/CD pipeline | — | +| High | Backup & restore | Done — `scripts/backup.sh` and `scripts/restore.sh`, tested end-to-end | +| High | CI/CD | Done — Gitea Actions with act_runner, see `docs/gitea-actions-guide.md` | +| High | Security review | Not started | +| Medium | Automated remote backups (Restic) | Blocked on choosing backup destination | +| Medium | Migration to permanent VPS | Blocked on company incorporation | +| Medium | SMTP / Email | Blocked on ProtonMail setup after incorporation | | Low | Monitoring | — | | Low | Group-based access control | Not needed yet (4 people) | | Low | Move SSH to non-standard port | — | | Low | Outline branding (remove badge) | Cosmetic, parked | +## Completed + +### Backup & Restore +- Manual backup/restore scripts in `scripts/` — tested end-to-end +- See `docs/backup-restore.md` for full procedure + +### CI/CD — Gitea Actions +- Gitea Actions enabled with act_runner +- GitHub Actions compatible workflow syntax +- See `docs/gitea-actions-guide.md` for usage guide + ## High Priority -### Backup System (Restic) -- Daily automated backup of all Docker volumes to object storage +### Automated Remote Backups (Restic) +- Automate the existing backup script to run daily and push to remote storage - Systemd timer (not cron) - Retention policy: 7 daily, 4 weekly, 6 monthly -- Test restore procedure documented - **Blocked on**: choosing a backup destination (S3 / Backblaze B2 / SFTP) ### Security Review @@ -55,11 +66,6 @@ sequenceDiagram Note over Old: Decommission ``` -### CI/CD Pipeline -- Automated deployment on git push -- Documentation pipeline (Markdown to Outline, or similar) -- Options to evaluate: Gitea Actions, Drone CI, or simple webhook-based deploy - ### SMTP / Email - Set up ProtonMail for cloud-elves.eu (after incorporation) - Configure Outline and Gitea to send transactional emails diff --git a/docs/overview.md b/docs/overview.md index 67ac130..d49b894 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -19,9 +19,10 @@ All services run on a single DigitalOcean VPS (`box.cloud-elves.eu`, Debian 13) | Service | Used by | |---|---| +| Gitea Runner (act_runner) | Gitea Actions CI/CD — runs pipelines | | Postgres (gitea-db) | Gitea | | Postgres (outline-db) | Outline | -| Redis | Outline | +| Redis | Outline (caching/sessions) | ### How a request reaches your app @@ -107,15 +108,16 @@ All other communication happens over internal Docker networks. UFW firewall bloc ## Data & Persistence -All persistent data is in Docker named volumes under `/var/lib/docker/volumes/` on the host. +All persistent data is in Docker named volumes under `/var/lib/docker/volumes/` on the host. Docker Compose prefixes them with `services_` (e.g. `services_gitea-data`). -| Volume | Contains | -|---|---| -| `caddy-data` | TLS certificates | -| `caddy-config` | Caddy runtime config | -| `lldap-data` | User directory (SQLite) | -| `dex-data` | Dex database (SQLite) | -| `gitea-data` | Git repositories, Gitea config | -| `gitea-db` | Gitea Postgres data | -| `outline-data` | Uploaded files/attachments | -| `outline-db` | Outline Postgres data | +| Volume | Contains | Backed up? | +|---|---|---| +| `lldap-data` | User directory (SQLite) | Yes | +| `dex-data` | Dex auth database (SQLite) | Yes | +| `gitea-data` | Git repositories, Gitea config | Yes | +| `gitea-db` | Gitea Postgres data | Yes | +| `outline-data` | Uploaded files/attachments | Yes | +| `outline-db` | Outline Postgres data | Yes | +| `caddy-data` | TLS certificates | No — re-issued automatically | +| `caddy-config` | Caddy runtime config | No — regenerated from Caddyfile | +| `gitea-runner-data` | Runner registration state | No — regenerated on deploy |