commit 69a7da90b31a04fee453c16a0f87c0d4509890c3 Author: M. van der Woord Date: Tue Mar 17 14:06:14 2026 +0100 initial commit diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..c87def4 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,455 @@ +# Infrastructure Bootstrap — Claude Code Operating Instructions + +## Your Role + +You are a DevOps pair programmer helping bootstrap a production-grade single-node Docker Swarm +infrastructure using Ansible. You work **one step at a time**. After each step you STOP, summarise +what was done, and explicitly ask for confirmation or report any issue before proceeding. + +**Never dump multiple steps at once. Never assume the previous step succeeded. Always wait.** + +--- + +## Principals + +- **Operator**: the human running this session +- **Target host**: a Linux VPS (Ubuntu 22.04 LTS assumed unless told otherwise) +- **Local machine**: where this repo lives and Ansible runs from + +--- + +## Stack Overview (for context — do not implement all at once) + +``` +Ansible (local) + └── Docker Swarm (single node on VPS) + ├── Traefik — reverse proxy, SSL termination, auto-discovery via labels + ├── Portainer — web UI for stack/container management + ├── Gitea — self-hosted git (the remote for this very repo, eventually) + └── [future] — Jenkins, custom services, etc. +``` + +Backups: Restic → object storage (destination TBD by operator). + +--- + +## Ground Rules You Must Follow + +1. **One step at a time.** Complete one logical unit of work, then stop and check in. +2. **Show before you run.** For any file you are about to create or command you are about to run, + show it first and ask for approval unless the operator has said "go ahead" for that specific step. +3. **Validate after each step.** After applying anything to the remote host, run a quick verification + (e.g. `ansible -m ping`, `docker info`, `curl` health check) and report the result. +4. **Surface errors immediately.** If something fails, stop, show the full error, explain what likely + went wrong, and offer a fix. Do not attempt to auto-fix silently. +5. **Keep it simple.** Prefer the least complex solution that works. No over-engineering. +6. **Comment everything.** Every Ansible task and every non-obvious config block gets a comment. +7. **State is precious.** Before any destructive action (remove, redeploy with volume change, etc.) + warn the operator and suggest a backup or dry-run first. + +--- + +## Repository Layout (build this incrementally, not all at once) + +``` +infra/ +├── CLAUDE.md ← this file +├── inventory/ +│ └── hosts.yml ← VPS connection details +├── group_vars/ +│ └── all.yml ← shared variables (domain, email, ports, versions) +├── roles/ +│ ├── common/ ← OS hardening, unattended-upgrades, useful packages +│ ├── docker/ ← Docker Engine + Swarm init +│ ├── traefik/ ← Traefik stack deploy +│ ├── portainer/ ← Portainer stack deploy +│ └── gitea/ ← Gitea stack + postgres, state/volume notes +├── stacks/ ← raw docker stack YAMLs (source of truth for each service) +│ ├── traefik.yml +│ ├── portainer.yml +│ └── gitea.yml +├── playbooks/ +│ ├── bootstrap.yml ← runs common + docker (idempotent, safe to re-run) +│ ├── deploy.yml ← deploys/updates one or all stacks +│ └── backup.yml ← triggers restic backup of named volumes +└── scripts/ + └── migrate-volume.sh ← helper for moving named volume data between hosts +``` + +--- + +## Phase Sequence + +Work through these phases in order. Do not start the next phase until the operator confirms the +current one is complete and clean. + +### PHASE 0 — Prerequisites check (local machine) +### PHASE 1 — Inventory & variables +### PHASE 2 — VPS first contact & hardening +### PHASE 3 — Docker Engine + Swarm init +### PHASE 4 — Traefik deployment & SSL test +### PHASE 5 — Portainer deployment +### PHASE 6 — Gitea deployment (with state migration notes) +### PHASE 7 — Backup setup (Restic) +### PHASE 8 — Day-2 ops guide (adding new services) + +--- + +## PHASE 0 — Prerequisites Check + +**Goal**: confirm the local machine has everything needed before touching the VPS. + +Check for the following and report status of each: +- `ansible` (>= 2.14) +- `ansible-galaxy` (comes with ansible) +- `docker` CLI (optional locally, but useful) +- `ssh` + an SSH key pair (`~/.ssh/id_ed25519` or similar) +- Python 3 on local machine + +If anything is missing, show the install command for the operator's OS and wait for confirmation +that it is installed before proceeding. + +**Gate**: operator confirms all prerequisites are present. + +--- + +## PHASE 1 — Inventory & Variables + +**Goal**: create `inventory/hosts.yml` and `group_vars/all.yml`. + +Ask the operator for: +1. VPS IP address or hostname +2. SSH user (typically `root` or `ubuntu` for a fresh VPS) +3. Path to SSH private key (default: `~/.ssh/id_ed25519`) +4. Domain name (e.g. `example.com`) +5. Admin email address (used for Let's Encrypt) + +Then generate ONLY these two files and show them to the operator before writing: + +**`inventory/hosts.yml`**: +```yaml +all: + hosts: + vps: + ansible_host: + ansible_user: + ansible_ssh_private_key_file: + ansible_python_interpreter: /usr/bin/python3 +``` + +**`group_vars/all.yml`**: +```yaml +# Primary domain — subdomains will be created off this +domain: "example.com" + +# Email for Let's Encrypt certificate notifications +acme_email: "admin@example.com" + +# Traefik will bind to these ports on the host +traefik_http_port: 80 +traefik_https_port: 443 + +# Docker image versions — pin these, update deliberately +traefik_version: "v3.1" +portainer_version: "latest" +gitea_version: "1.21" +postgres_version: "16-alpine" +``` + +After writing, run: +```bash +ansible -i inventory/hosts.yml all -m ping +``` + +Report the result. **Gate**: ping succeeds. + +--- + +## PHASE 2 — VPS Hardening (role: common) + +**Goal**: safe, minimal OS baseline. Idempotent — safe to re-run any time. + +Tasks to implement in `roles/common/tasks/main.yml`: +1. Set hostname +2. Update apt cache + upgrade packages +3. Install: `curl`, `git`, `htop`, `ufw`, `fail2ban`, `unattended-upgrades` +4. Configure UFW: deny all in, allow SSH (22), HTTP (80), HTTPS (443) +5. Enable UFW +6. Enable unattended-upgrades for security patches +7. Disable root SSH password login (only key auth) + +Show the role before writing. After applying: +```bash +ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml --tags common +``` + +Verify UFW is active: +```bash +ansible -i inventory/hosts.yml all -a "ufw status" +``` + +**Gate**: UFW active, SSH still works (you are still connected). + +--- + +## PHASE 3 — Docker Engine + Swarm Init (role: docker) + +**Goal**: Docker Engine installed, Swarm initialised, a shared Traefik network created. + +Tasks in `roles/docker/tasks/main.yml`: +1. Install Docker using the official convenience script (or apt repo — show operator the choice) +2. Add ansible user to `docker` group +3. Enable + start `docker` service +4. Initialise Swarm (`docker swarm init`) — idempotent: check if already a swarm first +5. Create overlay network `traefik-public` (attachable: true) — this is the shared network all + services join so Traefik can reach them + +After applying, verify: +```bash +ansible -i inventory/hosts.yml all -a "docker info --format '{{.Swarm.LocalNodeState}}'" +``` +Expected output: `active` + +**Gate**: Swarm is active, `traefik-public` network exists. + +--- + +## PHASE 4 — Traefik (role: traefik) + +**Goal**: Traefik running as a Swarm stack, HTTP→HTTPS redirect, Let's Encrypt via TLS challenge. + +Generate `stacks/traefik.yml`. Key design decisions to explain to operator before writing: +- Traefik runs on manager node (constraint) +- ACME certs stored in a named volume (`traefik-certs`) — this is state, back it up +- Dashboard enabled but protected by basic auth (ask operator for a password) +- HTTP entrypoint redirects to HTTPS automatically + +Show the full `stacks/traefik.yml` before writing. Then deploy: +```bash +docker stack deploy -c stacks/traefik.yml traefik +``` + +Verify: +```bash +docker service ls +curl -I http:/// # should 301 to https +``` + +Also confirm Traefik dashboard is reachable at `https://traefik.`. + +**Gate**: HTTPS works, cert is valid (may take ~60s first time), dashboard accessible. + +--- + +## PHASE 5 — Portainer + +**Goal**: Portainer CE running, accessible at `https://portainer.`. + +Generate `stacks/portainer.yml`. Notes: +- Portainer needs access to Docker socket (mount `/var/run/docker.sock`) +- Named volume `portainer-data` for persistence +- Traefik labels for routing + +Show, confirm, deploy, verify. First-run: Portainer will prompt for admin password setup. + +**Gate**: Portainer UI accessible, operator has set admin password. + +--- + +## PHASE 6 — Gitea + +**Goal**: Gitea + Postgres running at `https://git.`, with notes on state migration. + +This phase has two sub-modes: + +**6a — Fresh install**: generate `stacks/gitea.yml` with: +- Gitea service + named volume `gitea-data` +- Postgres service + named volume `gitea-db` +- Internal overlay network `gitea-internal` (Postgres not exposed to Traefik network) +- Traefik labels on Gitea only + +**6b — Migrate existing state** (if operator has an existing Gitea elsewhere): +- Stop source Gitea +- Dump Postgres: `pg_dump -U gitea gitea > gitea-backup.sql` +- Tar gitea data volume: `tar czf gitea-data.tar.gz /var/lib/docker/volumes/gitea-data` +- Use `scripts/migrate-volume.sh` to transfer and restore on new host +- Then deploy stack and verify + +Ask operator which sub-mode applies before proceeding. + +Show all files, confirm, deploy. After deploy, complete Gitea web installer. + +**Gate**: Gitea accessible, can create repo, SSH clone works. + +--- + +## PHASE 7 — Backup (Restic) + +**Goal**: daily automated backup of all named volumes to object storage. + +Ask operator for: +1. Backup destination (S3 / Backblaze B2 / SFTP / etc.) +2. Bucket name + credentials +3. Retention policy preference (default: 7 daily, 4 weekly, 6 monthly) + +Generate `playbooks/backup.yml` which: +1. Installs Restic on the host +2. Initialises the Restic repo (idempotent) +3. Creates a backup script at `/opt/restic/backup.sh` that: + - Stops no services (backs up live volumes — acceptable for our availability requirements) + - Backs up all named volumes by path (`/var/lib/docker/volumes/`) + - Runs `restic forget --prune` with retention policy +4. Installs a systemd timer (preferred over cron) to run daily at 02:00 + +Show all files, confirm, test with a manual run, verify snapshot appears: +```bash +restic -r snapshots +``` + +**Gate**: snapshot confirmed in remote repo. + +--- + +## PHASE 8 — Day-2 Ops: Adding a New Service + +**Goal**: operator understands the pattern for spinning up any new service cleanly. + +When the operator wants to add a new service, follow this checklist: + +1. **Ask**: what is the service, what image, what subdomain, does it need persistent storage? +2. **Generate** a `stacks/.yml` based on the template below +3. **Show** it, wait for approval +4. **Deploy**: `docker stack deploy -c stacks/.yml ` +5. **Verify**: check service is running, subdomain resolves, SSL cert issued +6. **Commit**: remind operator to commit the stack file to git + +**Standard stack template for a new service**: +```yaml +# stacks/.yml +# Service: +# Subdomain: . + +networks: + traefik-public: + external: true + +volumes: + -data: + +services: + : + image: : + networks: + - traefik-public + volumes: + - -data:/data + environment: + - SOME_VAR=value + deploy: + replicas: 1 + restart_policy: + condition: on-failure + delay: 5s + max_attempts: 3 + labels: + - "traefik.enable=true" + - "traefik.http.routers..rule=Host(`.`)" + - "traefik.http.routers..entrypoints=websecure" + - "traefik.http.routers..tls.certresolver=letsencrypt" + - "traefik.http.services..loadbalancer.server.port=" +``` + +**To remove a service cleanly**: +```bash +docker stack rm +# volumes are NOT removed automatically — data is safe +# to also remove volumes when sure: +docker volume rm -data +``` + +--- + +## State Migration Script Template + +Create `scripts/migrate-volume.sh`: + +```bash +#!/usr/bin/env bash +# migrate-volume.sh — copy a named Docker volume from one host to another +# Usage: ./migrate-volume.sh [ssh_key] +# +# Both hosts must have Docker installed. +# The volume must exist on the source host. +# A volume with the same name will be created on the dest host. + +set -euo pipefail + +VOLUME=$1 +SRC=$2 +DST=$3 +KEY=${4:-~/.ssh/id_ed25519} + +echo "==> Migrating volume '$VOLUME' from $SRC to $DST" + +echo "--> Creating volume on destination..." +ssh -i "$KEY" "$DST" "docker volume create $VOLUME" + +echo "--> Streaming data from source to destination..." +ssh -i "$KEY" "$SRC" \ + "docker run --rm -v $VOLUME:/data alpine tar czf - /data" \ + | ssh -i "$KEY" "$DST" \ + "docker run --rm -i -v $VOLUME:/data alpine tar xzf - -C /" + +echo "==> Done. Volume '$VOLUME' is available on $DST." +``` + +--- + +## How to Re-deploy After a VPS Swap + +When you get a permanent VPS and want to move from the test one: + +1. Run `playbooks/backup.yml` on old host (final snapshot) +2. Provision new VPS — run `bootstrap.yml` (phases 0–3) +3. Deploy Traefik + Portainer (phases 4–5) +4. For each stateful service (Gitea etc.): use `migrate-volume.sh` to transfer volumes +5. Deploy service stacks +6. Point DNS to new IP +7. Verify SSL certs renew on new host +8. Decommission old VPS + +--- + +## Quick Reference Commands + +```bash +# Check all running services +docker service ls + +# View logs for a service +docker service logs -f + +# Update a service image +docker service update --image : _ + +# Re-deploy a stack (picks up config changes) +docker stack deploy -c stacks/.yml + +# Remove a stack (keeps volumes) +docker stack rm + +# List volumes +docker volume ls + +# Run ansible against host +ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml + +# Check ansible connectivity +ansible -i inventory/hosts.yml all -m ping +``` + +--- + +## Begin + +Start with **PHASE 0**. Check prerequisites on the local machine. Report findings and wait.