ce-infra/docs/backup-restore.md

Backup & Restore

Overview

We have a tested backup and restore process that captures all state from our server and can fully restore it onto a fresh VPS. This has been verified end-to-end: backup, rebuild droplet, bootstrap, deploy, restore — everything comes back including repos, documents, users, and SSO.

What gets backed up

Component	What's captured	How
Gitea	Repositories, config, users, issues	Postgres dump + data volume
Outline	All documents, collections, settings	Postgres dump + file uploads volume
LLDAP	All user accounts and groups	SQLite database file
Dex	Auth tokens, signing keys	SQLite database file

What is NOT backed up

Component	Why
Static site (cloud-elves.eu)	Deployed via rsync from your local repo — redeploy after restore
TLS certificates	Caddy re-issues them automatically in seconds
Gitea runner token	Regenerated automatically during deploy
Docker images	Re-pulled automatically on startup

Running a backup

From your laptop, in the ce-infra repo directory:

./scripts/backup.sh

This will:

1. Stop application services (~30 seconds downtime)
2. Dump both Postgres databases
3. Copy all data volumes (Gitea, Outline, LLDAP, Dex)
4. Restart all services
5. Download a single .tar file to ./backups/

The backup file is named with a timestamp, e.g. ce-backup-20260317-195615.tar.

Full restore procedure

When you need this

• Droplet was destroyed or rebuilt
• Migrating to a new server
• Disaster recovery

Steps

sequenceDiagram
    participant You
    participant VPS as New VPS
    participant Backup as Backup File

    You->>VPS: Provision Debian 13 droplet
    You->>You: Fix SSH known hosts
    You->>VPS: Run bootstrap.yml
    You->>VPS: Run deploy.yml (will partially fail — expected)
    Backup->>VPS: Run restore.sh
    Note over VPS: All data restored
    You->>VPS: Verify all services

1. Provision a new Debian 13 droplet with your SSH key (id_ed25519_ce).

2. Update DNS if the IP changed — point box.cloud-elves.eu A record to the new IP.

3. Fix SSH known hosts:

ssh-keygen -R box.cloud-elves.eu
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts

4. Bootstrap the server:

ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml

5. Deploy services:

ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

This will partially fail (Gitea isn't installed yet). That's expected — the restore will fix it.

6. Restore from backup:

./scripts/restore.sh ./backups/ce-backup-YYYYMMDD-HHMMSS.tar

7. Redeploy static site (not included in backup):

cd /path/to/cloud-elves.eu && rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/

8. Verify everything:

Service	URL	What to check
LLDAP	https://ldap.cloud-elves.eu	Users are there
Dex	https://auth.cloud-elves.eu	SSO login works
Gitea	https://git.cloud-elves.eu	Repos and code are there
Outline	https://docs.cloud-elves.eu	Documents are there
Static site	https://cloud-elves.eu	Page loads

Users will need to log in again — SSO sessions are reset during restore.

Important notes

• SECRET_KEY and UTILS_SECRET (for Outline) must be preserved. These are stored in inventory/group_vars/all.yml. If you lose these, encrypted data in Outline becomes permanently inaccessible.
• Keep this repo safe — it contains all the configuration and secrets needed to rebuild everything.
• Keep backups off the server — the backup script downloads the file to your laptop. Store copies somewhere safe.
• Test restores periodically — a backup you haven't tested is not a backup.