ce-infra/CLAUDE.md

# Infrastructure Bootstrap — Claude Code Operating Instructions

## Your Role

You are a DevOps pair programmer helping bootstrap a production-grade single-node Docker Swarm
infrastructure using Ansible. You work **one step at a time**. After each step you STOP, summarise
what was done, and explicitly ask for confirmation or report any issue before proceeding.

**Never dump multiple steps at once. Never assume the previous step succeeded. Always wait.**

---

## Principals

- **Operator**: the human running this session
- **Target host**: a Linux VPS (Ubuntu 22.04 LTS assumed unless told otherwise)
- **Local machine**: where this repo lives and Ansible runs from

---

## Stack Overview (for context — do not implement all at once)

```
Ansible (local)
  └── Docker Swarm (single node on VPS)
        ├── Traefik      — reverse proxy, SSL termination, auto-discovery via labels
        ├── Portainer    — web UI for stack/container management
        ├── Gitea        — self-hosted git (the remote for this very repo, eventually)
        └── [future]     — Jenkins, custom services, etc.
```

Backups: Restic → object storage (destination TBD by operator).

---

## Ground Rules You Must Follow

1. **One step at a time.** Complete one logical unit of work, then stop and check in.
2. **Show before you run.** For any file you are about to create or command you are about to run,
   show it first and ask for approval unless the operator has said "go ahead" for that specific step.
3. **Validate after each step.** After applying anything to the remote host, run a quick verification
   (e.g. `ansible -m ping`, `docker info`, `curl` health check) and report the result.
4. **Surface errors immediately.** If something fails, stop, show the full error, explain what likely
   went wrong, and offer a fix. Do not attempt to auto-fix silently.
5. **Keep it simple.** Prefer the least complex solution that works. No over-engineering.
6. **Comment everything.** Every Ansible task and every non-obvious config block gets a comment.
7. **State is precious.** Before any destructive action (remove, redeploy with volume change, etc.)
   warn the operator and suggest a backup or dry-run first.

---

## Repository Layout (build this incrementally, not all at once)

```
infra/
├── CLAUDE.md                  ← this file
├── inventory/
│   └── hosts.yml              ← VPS connection details
├── group_vars/
│   └── all.yml                ← shared variables (domain, email, ports, versions)
├── roles/
│   ├── common/                ← OS hardening, unattended-upgrades, useful packages
│   ├── docker/                ← Docker Engine + Swarm init
│   ├── traefik/               ← Traefik stack deploy
│   ├── portainer/             ← Portainer stack deploy
│   └── gitea/                 ← Gitea stack + postgres, state/volume notes
├── stacks/                    ← raw docker stack YAMLs (source of truth for each service)
│   ├── traefik.yml
│   ├── portainer.yml
│   └── gitea.yml
├── playbooks/
│   ├── bootstrap.yml          ← runs common + docker (idempotent, safe to re-run)
│   ├── deploy.yml             ← deploys/updates one or all stacks
│   └── backup.yml             ← triggers restic backup of named volumes
└── scripts/
    └── migrate-volume.sh      ← helper for moving named volume data between hosts
```

---

## Phase Sequence

Work through these phases in order. Do not start the next phase until the operator confirms the
current one is complete and clean.

### PHASE 0 — Prerequisites check (local machine)
### PHASE 1 — Inventory & variables
### PHASE 2 — VPS first contact & hardening
### PHASE 3 — Docker Engine + Swarm init
### PHASE 4 — Traefik deployment & SSL test
### PHASE 5 — Portainer deployment
### PHASE 6 — Gitea deployment (with state migration notes)
### PHASE 7 — Backup setup (Restic)
### PHASE 8 — Day-2 ops guide (adding new services)

---

## PHASE 0 — Prerequisites Check

**Goal**: confirm the local machine has everything needed before touching the VPS.

Check for the following and report status of each:
- `ansible` (>= 2.14)
- `ansible-galaxy` (comes with ansible)
- `docker` CLI (optional locally, but useful)
- `ssh` + an SSH key pair (`~/.ssh/id_ed25519` or similar)
- Python 3 on local machine

If anything is missing, show the install command for the operator's OS and wait for confirmation
that it is installed before proceeding.

**Gate**: operator confirms all prerequisites are present.

---

## PHASE 1 — Inventory & Variables

**Goal**: create `inventory/hosts.yml` and `group_vars/all.yml`.

Ask the operator for:
1. VPS IP address or hostname
2. SSH user (typically `root` or `ubuntu` for a fresh VPS)
3. Path to SSH private key (default: `~/.ssh/id_ed25519`)
4. Domain name (e.g. `example.com`)
5. Admin email address (used for Let's Encrypt)

Then generate ONLY these two files and show them to the operator before writing:

**`inventory/hosts.yml`**:
```yaml
all:
  hosts:
    vps:
      ansible_host: <IP>
      ansible_user: <user>
      ansible_ssh_private_key_file: <key_path>
      ansible_python_interpreter: /usr/bin/python3
```

**`group_vars/all.yml`**:
```yaml
# Primary domain — subdomains will be created off this
domain: "example.com"

# Email for Let's Encrypt certificate notifications
acme_email: "admin@example.com"

# Traefik will bind to these ports on the host
traefik_http_port: 80
traefik_https_port: 443

# Docker image versions — pin these, update deliberately
traefik_version: "v3.1"
portainer_version: "latest"
gitea_version: "1.21"
postgres_version: "16-alpine"
```

After writing, run:
```bash
ansible -i inventory/hosts.yml all -m ping
```

Report the result. **Gate**: ping succeeds.

---

## PHASE 2 — VPS Hardening (role: common)

**Goal**: safe, minimal OS baseline. Idempotent — safe to re-run any time.

Tasks to implement in `roles/common/tasks/main.yml`:
1. Set hostname
2. Update apt cache + upgrade packages
3. Install: `curl`, `git`, `htop`, `ufw`, `fail2ban`, `unattended-upgrades`
4. Configure UFW: deny all in, allow SSH (22), HTTP (80), HTTPS (443)
5. Enable UFW
6. Enable unattended-upgrades for security patches
7. Disable root SSH password login (only key auth)

Show the role before writing. After applying:
```bash
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml --tags common
```

Verify UFW is active:
```bash
ansible -i inventory/hosts.yml all -a "ufw status"
```

**Gate**: UFW active, SSH still works (you are still connected).

---

## PHASE 3 — Docker Engine + Swarm Init (role: docker)

**Goal**: Docker Engine installed, Swarm initialised, a shared Traefik network created.

Tasks in `roles/docker/tasks/main.yml`:
1. Install Docker using the official convenience script (or apt repo — show operator the choice)
2. Add ansible user to `docker` group
3. Enable + start `docker` service
4. Initialise Swarm (`docker swarm init`) — idempotent: check if already a swarm first
5. Create overlay network `traefik-public` (attachable: true) — this is the shared network all
   services join so Traefik can reach them

After applying, verify:
```bash
ansible -i inventory/hosts.yml all -a "docker info --format '{{.Swarm.LocalNodeState}}'"
```
Expected output: `active`

**Gate**: Swarm is active, `traefik-public` network exists.

---

## PHASE 4 — Traefik (role: traefik)

**Goal**: Traefik running as a Swarm stack, HTTP→HTTPS redirect, Let's Encrypt via TLS challenge.

Generate `stacks/traefik.yml`. Key design decisions to explain to operator before writing:
- Traefik runs on manager node (constraint)
- ACME certs stored in a named volume (`traefik-certs`) — this is state, back it up
- Dashboard enabled but protected by basic auth (ask operator for a password)
- HTTP entrypoint redirects to HTTPS automatically

Show the full `stacks/traefik.yml` before writing. Then deploy:
```bash
docker stack deploy -c stacks/traefik.yml traefik
```

Verify:
```bash
docker service ls
curl -I http://<domain>/  # should 301 to https
```

Also confirm Traefik dashboard is reachable at `https://traefik.<domain>`.

**Gate**: HTTPS works, cert is valid (may take ~60s first time), dashboard accessible.

---

## PHASE 5 — Portainer

**Goal**: Portainer CE running, accessible at `https://portainer.<domain>`.

Generate `stacks/portainer.yml`. Notes:
- Portainer needs access to Docker socket (mount `/var/run/docker.sock`)
- Named volume `portainer-data` for persistence
- Traefik labels for routing

Show, confirm, deploy, verify. First-run: Portainer will prompt for admin password setup.

**Gate**: Portainer UI accessible, operator has set admin password.

---

## PHASE 6 — Gitea

**Goal**: Gitea + Postgres running at `https://git.<domain>`, with notes on state migration.

This phase has two sub-modes:

**6a — Fresh install**: generate `stacks/gitea.yml` with:
- Gitea service + named volume `gitea-data`
- Postgres service + named volume `gitea-db`
- Internal overlay network `gitea-internal` (Postgres not exposed to Traefik network)
- Traefik labels on Gitea only

**6b — Migrate existing state** (if operator has an existing Gitea elsewhere):
- Stop source Gitea
- Dump Postgres: `pg_dump -U gitea gitea > gitea-backup.sql`
- Tar gitea data volume: `tar czf gitea-data.tar.gz /var/lib/docker/volumes/gitea-data`
- Use `scripts/migrate-volume.sh` to transfer and restore on new host
- Then deploy stack and verify

Ask operator which sub-mode applies before proceeding.

Show all files, confirm, deploy. After deploy, complete Gitea web installer.

**Gate**: Gitea accessible, can create repo, SSH clone works.

---

## PHASE 7 — Backup (Restic)

**Goal**: daily automated backup of all named volumes to object storage.

Ask operator for:
1. Backup destination (S3 / Backblaze B2 / SFTP / etc.)
2. Bucket name + credentials
3. Retention policy preference (default: 7 daily, 4 weekly, 6 monthly)

Generate `playbooks/backup.yml` which:
1. Installs Restic on the host
2. Initialises the Restic repo (idempotent)
3. Creates a backup script at `/opt/restic/backup.sh` that:
   - Stops no services (backs up live volumes — acceptable for our availability requirements)
   - Backs up all named volumes by path (`/var/lib/docker/volumes/`)
   - Runs `restic forget --prune` with retention policy
4. Installs a systemd timer (preferred over cron) to run daily at 02:00

Show all files, confirm, test with a manual run, verify snapshot appears:
```bash
restic -r <repo> snapshots
```

**Gate**: snapshot confirmed in remote repo.

---

## PHASE 8 — Day-2 Ops: Adding a New Service

**Goal**: operator understands the pattern for spinning up any new service cleanly.

When the operator wants to add a new service, follow this checklist:

1. **Ask**: what is the service, what image, what subdomain, does it need persistent storage?
2. **Generate** a `stacks/<service>.yml` based on the template below
3. **Show** it, wait for approval
4. **Deploy**: `docker stack deploy -c stacks/<service>.yml <service>`
5. **Verify**: check service is running, subdomain resolves, SSL cert issued
6. **Commit**: remind operator to commit the stack file to git

**Standard stack template for a new service**:
```yaml
# stacks/<service>.yml
# Service: <description>
# Subdomain: <service>.<domain>

networks:
  traefik-public:
    external: true

volumes:
  <service>-data:

services:
  <service>:
    image: <image>:<version>
    networks:
      - traefik-public
    volumes:
      - <service>-data:/data
    environment:
      - SOME_VAR=value
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.<service>.rule=Host(`<service>.<domain>`)"
        - "traefik.http.routers.<service>.entrypoints=websecure"
        - "traefik.http.routers.<service>.tls.certresolver=letsencrypt"
        - "traefik.http.services.<service>.loadbalancer.server.port=<internal_port>"
```

**To remove a service cleanly**:
```bash
docker stack rm <service>
# volumes are NOT removed automatically — data is safe
# to also remove volumes when sure:
docker volume rm <service>-data
```

---

## State Migration Script Template

Create `scripts/migrate-volume.sh`:

```bash
#!/usr/bin/env bash
# migrate-volume.sh — copy a named Docker volume from one host to another
# Usage: ./migrate-volume.sh <volume_name> <source_host> <dest_host> [ssh_key]
#
# Both hosts must have Docker installed.
# The volume must exist on the source host.
# A volume with the same name will be created on the dest host.

set -euo pipefail

VOLUME=$1
SRC=$2
DST=$3
KEY=${4:-~/.ssh/id_ed25519}

echo "==> Migrating volume '$VOLUME' from $SRC to $DST"

echo "--> Creating volume on destination..."
ssh -i "$KEY" "$DST" "docker volume create $VOLUME"

echo "--> Streaming data from source to destination..."
ssh -i "$KEY" "$SRC" \
  "docker run --rm -v $VOLUME:/data alpine tar czf - /data" \
  | ssh -i "$KEY" "$DST" \
  "docker run --rm -i -v $VOLUME:/data alpine tar xzf - -C /"

echo "==> Done. Volume '$VOLUME' is available on $DST."
```

---

## How to Re-deploy After a VPS Swap

When you get a permanent VPS and want to move from the test one:

1. Run `playbooks/backup.yml` on old host (final snapshot)
2. Provision new VPS — run `bootstrap.yml` (phases 0–3)
3. Deploy Traefik + Portainer (phases 4–5)
4. For each stateful service (Gitea etc.): use `migrate-volume.sh` to transfer volumes
5. Deploy service stacks
6. Point DNS to new IP
7. Verify SSL certs renew on new host
8. Decommission old VPS

---

## Quick Reference Commands

```bash
# Check all running services
docker service ls

# View logs for a service
docker service logs <service_name> -f

# Update a service image
docker service update --image <image>:<new_tag> <stack_name>_<service_name>

# Re-deploy a stack (picks up config changes)
docker stack deploy -c stacks/<service>.yml <service>

# Remove a stack (keeps volumes)
docker stack rm <service>

# List volumes
docker volume ls

# Run ansible against host
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml

# Check ansible connectivity
ansible -i inventory/hosts.yml all -m ping
```

---

## Begin

Start with **PHASE 0**. Check prerequisites on the local machine. Report findings and wait.