123 lines
3.8 KiB
Markdown
123 lines
3.8 KiB
Markdown
# Backup & Restore
|
|
|
|
## Overview
|
|
|
|
We have a tested backup and restore process that captures all state from our server and can fully restore it onto a fresh VPS. This has been verified end-to-end: backup, rebuild droplet, bootstrap, deploy, restore — everything comes back including repos, documents, users, and SSO.
|
|
|
|
## What gets backed up
|
|
|
|
| Component | What's captured | How |
|
|
|---|---|---|
|
|
| Gitea | Repositories, config, users, issues | Postgres dump + data volume |
|
|
| Outline | All documents, collections, settings | Postgres dump + file uploads volume |
|
|
| LLDAP | All user accounts and groups | SQLite database file |
|
|
| Dex | Auth tokens, signing keys | SQLite database file |
|
|
|
|
## What is NOT backed up
|
|
|
|
| Component | Why |
|
|
|---|---|
|
|
| Static site (cloud-elves.eu) | Deployed via rsync from your local repo — redeploy after restore |
|
|
| TLS certificates | Caddy re-issues them automatically in seconds |
|
|
| Gitea runner token | Regenerated automatically during deploy |
|
|
| Docker images | Re-pulled automatically on startup |
|
|
|
|
## Running a backup
|
|
|
|
From your laptop, in the `ce-infra` repo directory:
|
|
|
|
```bash
|
|
./scripts/backup.sh
|
|
```
|
|
|
|
This will:
|
|
1. Stop application services (~30 seconds downtime)
|
|
2. Dump both Postgres databases
|
|
3. Copy all data volumes (Gitea, Outline, LLDAP, Dex)
|
|
4. Restart all services
|
|
5. Download a single `.tar` file to `./backups/`
|
|
|
|
The backup file is named with a timestamp, e.g. `ce-backup-20260317-195615.tar`.
|
|
|
|
## Full restore procedure
|
|
|
|
### When you need this
|
|
|
|
- Droplet was destroyed or rebuilt
|
|
- Migrating to a new server
|
|
- Disaster recovery
|
|
|
|
### Steps
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant You
|
|
participant VPS as New VPS
|
|
participant Backup as Backup File
|
|
|
|
You->>VPS: Provision Debian 13 droplet
|
|
You->>You: Fix SSH known hosts
|
|
You->>VPS: Run bootstrap.yml
|
|
You->>VPS: Run deploy.yml (will partially fail — expected)
|
|
Backup->>VPS: Run restore.sh
|
|
Note over VPS: All data restored
|
|
You->>VPS: Verify all services
|
|
```
|
|
|
|
**1. Provision a new Debian 13 droplet** with your SSH key (`id_ed25519_ce`).
|
|
|
|
**2. Update DNS** if the IP changed — point `box.cloud-elves.eu` A record to the new IP.
|
|
|
|
**3. Fix SSH known hosts:**
|
|
|
|
```bash
|
|
ssh-keygen -R box.cloud-elves.eu
|
|
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
|
|
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts
|
|
```
|
|
|
|
**4. Bootstrap the server:**
|
|
|
|
```bash
|
|
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml
|
|
```
|
|
|
|
**5. Deploy services:**
|
|
|
|
```bash
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
|
|
```
|
|
|
|
This will partially fail (Gitea isn't installed yet). That's expected — the restore will fix it.
|
|
|
|
**6. Restore from backup:**
|
|
|
|
```bash
|
|
./scripts/restore.sh ./backups/ce-backup-YYYYMMDD-HHMMSS.tar
|
|
```
|
|
|
|
**7. Redeploy static site** (not included in backup):
|
|
|
|
```bash
|
|
cd /path/to/cloud-elves.eu && rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/
|
|
```
|
|
|
|
**8. Verify everything:**
|
|
|
|
| Service | URL | What to check |
|
|
|---|---|---|
|
|
| LLDAP | https://ldap.cloud-elves.eu | Users are there |
|
|
| Dex | https://auth.cloud-elves.eu | SSO login works |
|
|
| Gitea | https://git.cloud-elves.eu | Repos and code are there |
|
|
| Outline | https://docs.cloud-elves.eu | Documents are there |
|
|
| Static site | https://cloud-elves.eu | Page loads |
|
|
|
|
Users will need to log in again — SSO sessions are reset during restore.
|
|
|
|
## Important notes
|
|
|
|
- **SECRET_KEY and UTILS_SECRET** (for Outline) must be preserved. These are stored in `inventory/group_vars/all.yml`. If you lose these, encrypted data in Outline becomes permanently inaccessible.
|
|
- **Keep this repo safe** — it contains all the configuration and secrets needed to rebuild everything.
|
|
- **Keep backups off the server** — the backup script downloads the file to your laptop. Store copies somewhere safe.
|
|
- **Test restores periodically** — a backup you haven't tested is not a backup.
|