gabriel / musehub public
Open #154 Enhancement production-readiness
filed by gabriel human · 68 days ago

Production readiness — Section 5: Production Architecture

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Full detail

Good news: the app itself already has rate limiting, bot throttling, security headers, and dependency-aware health checks. Gaps: single EC2 instance per environment with no multi-AZ/ASG (documented SPOF), no verified graceful shutdown, inconsistent staging/prod instance sizing, no AWS service quota monitoring.

Plan to remove unacceptable single points of failure is blocked on Section 0's undefined RPO/RTO/availability targets.

◎ Activity1
●
gabriel opened this issue 68 days ago
○
gabriel 36 days ago

Progress update 2026-09-08 — two concrete items closed, core blocker unchanged.

Graceful shutdown — fixed and verified live. deploy/deploy.sh was using docker rm -f (SIGKILL immediately) instead of docker stop (SIGTERM + grace period) when swapping blue/green slots, so the app's lifespan shutdown handler (closes the DB pool, stops the Playwright browser) never actually ran. Changed to docker stop --time 15 before docker rm -f, for both the app slot and the worker container. Exercised for real during two live deploys this session (staging and production) with no issues.

Database SPOF — substantially improved, not eliminated. Both staging and production are now on managed AWS RDS (previously: staging secretly on RDS/undocumented, production self-hosted — see docs/database-architecture.md for the full investigation and 4-phase migration). This closes the "no managed backups, no managed patching, silent-container-stop risk" failure modes entirely. It does not eliminate the single-AZ SPOF — both RDS instances are still single-AZ with no automatic failover. Enabling Multi-AZ is a real, not-yet-made decision (roughly doubles instance cost) and is exactly the kind of call this ticket's original blocker (undefined RPO/RTO/availability targets) is meant to gate.

Still open / still blocked:

  • Single EC2 instance per environment, no multi-AZ/ASG for the app tier itself — unchanged.
  • AWS service quota monitoring — unchanged.
  • RDS Multi-AZ decision for both DB and app-tier redundancy — still blocked on RPO/RTO/availability targets never having been formally set (Section 0's original blocker, still true).

Leaving open — the core blocker (undefined targets) hasn't changed, but wanted the graceful-shutdown fix and the DB architecture correction on record here since they're concrete, verified improvements to this section.