# Backup & Restore

## Overview

We have a tested backup and restore process that captures all state from our server and can fully restore it onto a fresh VPS. This has been verified end-to-end: backup, rebuild droplet, bootstrap, deploy, restore — everything comes back including repos, documents, users, and SSO.

## What gets backed up

| Component | What's captured | How |
|---|---|---|
| Gitea | Repositories, config, users, issues | Postgres dump + data volume |
| Outline | All documents, collections, settings | Postgres dump + file uploads volume |
| LLDAP | All user accounts and groups | SQLite database file |
| Dex | Auth tokens, signing keys | SQLite database file |

## What is NOT backed up

| Component | Why |
|---|---|
| Static site (cloud-elves.eu) | Deployed via rsync from your local repo — redeploy after restore |
| TLS certificates | Caddy re-issues them automatically in seconds |
| Gitea runner token | Regenerated automatically during deploy |
| Docker images | Re-pulled automatically on startup |

## Running a backup

From your laptop, in the `ce-infra` repo directory:

```bash
./scripts/backup.sh
```

This will:
1. Stop application services (~30 seconds downtime)
2. Dump both Postgres databases
3. Copy all data volumes (Gitea, Outline, LLDAP, Dex)
4. Restart all services
5. Download a single `.tar` file to `./backups/`

The backup file is named with a timestamp, e.g. `ce-backup-20260317-195615.tar`.

## Full restore procedure

### When you need this

- Droplet was destroyed or rebuilt
- Migrating to a new server
- Disaster recovery

### Steps

```mermaid
sequenceDiagram
    participant You
    participant VPS as New VPS
    participant Backup as Backup File

    You->>VPS: Provision Debian 13 droplet
    You->>You: Fix SSH known hosts
    You->>VPS: Run bootstrap.yml
    You->>VPS: Run deploy.yml (will partially fail — expected)
    Backup->>VPS: Run restore.sh
    Note over VPS: All data restored
    You->>VPS: Verify all services
```

**1. Provision a new Debian 13 droplet** with your SSH key (`id_ed25519_ce`).

**2. Update DNS** if the IP changed — point `box.cloud-elves.eu` A record to the new IP.

**3. Fix SSH known hosts:**

```bash
ssh-keygen -R box.cloud-elves.eu
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
ssh-keyscan box.cloud-elves.eu >> ~/.ssh/known_hosts
```

**4. Bootstrap the server:**

```bash
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml
```

**5. Deploy services:**

```bash
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml
```

This will partially fail (Gitea isn't installed yet). That's expected — the restore will fix it.

**6. Restore from backup:**

```bash
./scripts/restore.sh ./backups/ce-backup-YYYYMMDD-HHMMSS.tar
```

**7. Redeploy static site** (not included in backup):

```bash
cd /path/to/cloud-elves.eu && rsync -avz --delete site/ root@box.cloud-elves.eu:/opt/www/
```

**8. Verify everything:**

| Service | URL | What to check |
|---|---|---|
| LLDAP | https://ldap.cloud-elves.eu | Users are there |
| Dex | https://auth.cloud-elves.eu | SSO login works |
| Gitea | https://git.cloud-elves.eu | Repos and code are there |
| Outline | https://docs.cloud-elves.eu | Documents are there |
| Static site | https://cloud-elves.eu | Page loads |

Users will need to log in again — SSO sessions are reset during restore.

## Important notes

- **SECRET_KEY and UTILS_SECRET** (for Outline) must be preserved. These are stored in `inventory/group_vars/all.yml`. If you lose these, encrypted data in Outline becomes permanently inaccessible.
- **Keep this repo safe** — it contains all the configuration and secrets needed to rebuild everything.
- **Keep backups off the server** — the backup script downloads the file to your laptop. Store copies somewhere safe.
- **Test restores periodically** — a backup you haven't tested is not a backup.