Section 5 — Production Architecture

Companion to musehub-production-readiness-checklist.md.

Correction, 2026-09-08 (Phases 0–2 complete): while re-verifying this section, discovered that staging had actually been running on managed AWS RDS (not self-hosted Postgres) since 2026-04-05 — see docs/database-architecture.md, the canonical source of truth for database architecture. Production has since been migrated to RDS too (2026-09-08). Item 2 below is now partially resolved, not eliminated: both environments get managed patching, automated backups, and point-in-time recovery from RDS, closing the "no managed failover infrastructure at all" gap. But both instances are still deliberately single-AZ (MultiAZ: false) — a real outage or AZ failure still takes the database down; RDS just makes recovery fast and reliable rather than eliminating the single point of failure outright. Enabling Multi-AZ (near-zero-downtime automatic failover) remains a genuine, not-yet-made decision — tracked as a Phase 4 follow-up, gated on whether the added cost (roughly 2x the instance cost) is justified by current availability targets (#149).

Good news first — the app itself is more hardened than the infra around it

Grepping musehub/ turned up real, already-implemented protections that Section 0's infrastructure-level inventory didn't surface:

• Health checks with real dependency probes: /healthz (musehub/main.py:323) checks both DB and object storage, not just "process is up." Used by deploy.sh for the blue/green health gate.
• Rate limiting (musehub/rate_limits.py): per-route slowapi limits (wire push 30/min, MCP human/agent/anonymous tiers, asset requests), plus a 300/min global per-IP baseline.
• Bot throttling (musehub/middleware/bot_throttle.py): blocks unauthenticated scanners on write paths while explicitly exempting authenticated (MSign) traffic and safe GET/HEAD reads — a deliberate, well-reasoned design, not a blunt UA blocklist.
• Security headers already set in main.py: X-Frame-Options, X-Content-Type-Options, Content-Security-Policy, Strict-Transport-Security — and again at the nginx layer (deploy/nginx-cf.conf): HSTS with includeSubDomains, nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy.
• Deliberate per-route timeouts in nginx — not a single blanket value: 5s for fast health checks, 60s default, 120–300s for heavier routes, 3600s for streaming (MCP/SSE) endpoints. This is intentional, not an oversight.
• No cookies at all — MSign is header-based (Ed25519 signatures), so cookie flag items (Secure/HttpOnly/SameSite) are N/A, not gaps.

Real gaps

• Single EC2 instance per environment — no ASG, no multi-AZ. Blue/green happens within one instance (two Docker containers on different ports), not across instances. If the instance itself fails (hardware, AZ outage), there is no automatic failover. This is the single largest intentional SPOF and needs an explicit decision: accept it (documented technical debt, matches the "keep it simple" architecture direction from Section 0) or plan multi-instance/multi-AZ.
• No graceful shutdown / SIGTERM handling found in musehub/main.py — uvicorn's defaults apply. In-flight requests during a container stop may be dropped rather than drained. Given blue/green already waits for the new slot's /healthz before flipping nginx, the new slot is protected, but the old slot's shutdown during docker rm isn't verified graceful.
• No autoscaling, no documented justification for fixed capacity. Fixed t3.small (prod) / t3.medium (staging) — inconsistent sizing between environments with no stated reason. Worth either matching sizes or documenting why staging is intentionally larger.
• No AWS WAF — and none needed the way the checklist assumes. There's no ALB/CloudFront for AWS WAF to attach to. DDoS/abuse protection is Cloudflare's job here (edge proxy in front of everything), which is a reasonable substitution — but Cloudflare's own WAF/rate-limiting rules are unverified from this environment (no Cloudflare API access). Needs Gabriel to check the Cloudflare dashboard for what's actually configured there.
• No AWS service quota monitoring.
• Background job idempotency (musehub-runner CI jobs) not verified in this pass — would need a closer read of the runner's job-claiming logic; flagging as unverified rather than assuming.
• Migration/rolling-deploy compatibility: the deploy pipeline does run migrations before the slot flip (per docs/infrastructure.md's pipeline steps), which is the right shape, but expand-and-contract discipline for schema changes hasn't been reviewed — deferred to Section 8.

Documented intentional single points of failure (per the checklist's explicit ask)

1. One EC2 instance per environment — no instance-level redundancy.
2. Updated 2026-09-08: one RDS Postgres instance per environment (previously self-hosted for production, self-hosted-vs-RDS inconsistent between environments before this date — see docs/database-architecture.md) — single-AZ, no automatic failover. RDS Multi-AZ would close this remaining gap; not yet enabled, tracked as a Phase 4 decision pending availability targets.
3. One AWS account, shared with Stori (Section 2).
4. One Cloudflare account/zone — DNS, TLS, and (per Section 6) possibly object storage all depend on Cloudflare being up and correctly configured.
5. Deploys depend on one human running a script with a shared credential from a laptop — no redundant deployer today (Section 1/3).

Plan to remove unacceptable ones: not yet decided which of the above are acceptable long-term vs. must-fix-before-launch — this needs Gabriel's input once RPO/RTO/availability targets (Section 0, still open) are defined. A SPOF that's fine for a beta with a handful of users may not be fine at real scale.