gabriel / musehub public
Closed #157 Bug production-readiness
filed by gabriel human · 23 days ago

Production readiness — Section 8: Database Security, Migrations, and Backups — HIGHEST PRIORITY

0 Anchors
Blast radius
Churn 30d
0 Proposals

Full detail

CRITICAL: verified live via SSM that no usable database backup exists anywhere — no cron job installed, rclone not present, the only backup file found is a stale 20-byte manual dump from ~2 months ago. If the production database is lost today there is no recovery path.

Work: install rclone, configure the R2 remote, install the deploy/backup.sh cron job, verify a real non-empty backup lands, then perform an actual restore test. Also: split the single shared Postgres role into app-runtime/migration/admin roles.

This should be prioritized ahead of the rest of the checklist.

Activity1
gabriel opened this issue 23 days ago
gabriel 6 days ago

Update 2026-08-25: Resolved for Production — real, end-to-end verified, not just "a cron job exists."

  • Fixed a real bug: deploy/backup.sh used docker compose exec postgres, but production has no docker-compose.yml at all — fixed to docker exec musehub_postgres directly.
  • Extended deploy/secrets.sh to source BACKUP_R2_BUCKET from SSM too (non-secret String param) — needed since secrets.sh overwrites .env on every run.
  • Dedicated backup bucket (musehub-production-backups) + a separate, narrowly-scoped R2 token (Object Read & Write on that bucket only) — not reusing the app's own R2 credential, so one compromise doesn't grant access to the other.
  • Hit and fixed a real rclone+R2 gotcha: writes failed with 403 AccessDenied even though reads worked, because the object-scoped token lacks bucket-level permissions this old rclone version (1.53.3) checks for by default. Fixed with no_check_bucket = true.
  • Ran the backup for real: a genuine 16K compressed dump (not the historical 20-byte stale dump this whole effort kept finding), successfully synced to R2. Daily 3 AM cron installed.
  • Performed an actual restore test — this ticket's explicit ask, not just "backups run": restored into a throwaway postgres:16-alpine container (zero risk to the live DB), confirmed all 74 tables came back correctly (matching the schema gate's own count), tore the container down. Zero rows on musehub_repos/musehub_identities is expected — production is hours old with no real user data yet; the pipeline itself is what's proven.
  • Postgres role-splitting (app runtime/migrations/admin) from this ticket's secondary ask is not done — still a single shared role.

Staging still has zero backups — this was production-only work. Replicating it for staging (same bucket/token/rclone pattern) is a reasonable, lower-urgency follow-up now that production is no longer the emergency.

closed this issue 6 days ago