Sections 16–19 — Documentation, Testing, Cutover, Post-Launch

Status: ALL CLOSED — production-readiness issues #164 (Docs), #165 (Testing), #166 (Cutover), #167 (Post-Launch), all closed 2026-09-08. Ran the full 7,028-test suite for real for the first time and fixed 2 genuine bugs found; confirmed canonical redirects/TLS live externally; confirmed the maintainer-SPOF test now passes; documented that cutover already happened (2026-08-25) retrospectively. Remaining work across all three still-relevant sections (16, 17, 19 — Cutover/#166 has no remaining work) consolidated into #179.

Companion to musehub-production-readiness-checklist.md.

Update (2026-09-07) — most blockers resolved elsewhere; two real bugs found and fixed

This doc was written when almost nothing else in the checklist existed yet. Since then: backups exist and were tested (#157), Aaron has independent AWS access and staging no longer uses a shared deploy credential (#150), DNS/TLS/canonical-redirect are live, production deploy works, and production actually went live on 2026-08-25. Re-verified everything against current reality rather than the original blocked-on-everything-else framing.

Section 16 — Documentation and Runbooks

• [x] Document every AWS account and region
• [x] Document DNS and certificate ownership
• [x] Document staging deployment
• [x] Document production deployment
• [x] Document rollback — and now tested, not just documented (#159)
• [x] User-access provisioning and removal — identity model doc §11/§12 (joiner/leaver)
• [x] Break-glass access — centralized root access management, documented (#150)
• [x] Create a dependency inventory, operational decision log, accepted-risks log — all done, this checklist's own tables
• [x] Keep documentation versioned beside the relevant code — done by construction
• [x] Document local development setup — confirmed: README.md's "Quickstart" section covers this; the original pass just hadn't checked
• [ ] System architecture diagram / data-flow diagram — still not created (prose-only)
• [ ] Document environment differences — still scattered, not consolidated
• [ ] Document the IaC repository structure — still N/A, no IaC exists (#153)
• [ ] Document database migration rollback/forward-repair procedures — still not done
• [ ] Document database restoration — now possible to write (backups exist, #157), just not written yet
• [ ] Document secret rotation — still not done (#177)
• [ ] Document certificate/DNS troubleshooting — still not done
• [ ] Document incident response — still not done
• [ ] Document common alarms and remediation — still N/A, no alerts exist yet (#174)
• [ ] Service inventory with owners — inventory exists, owners still don't (#175)
• [ ] Confirm Aaron/Gabriel can follow docs without relying on each other or personal memory — not tested as a formal exercise, though Aaron's independent AWS/deploy access (#150) is a real, tested step in this direction
• [ ] Rollback decision tree, contact/escalation info, post-incident doc review process — none done

Section 17 — Testing Before Launch

Ran the actual test suite this pass — the original doc only confirmed it collects, never ran it for pass/fail:

7,028 tests collected → 7,003 passed, 15 failed, 24 skipped (35m56s)

Investigated and fixed the failures rather than just reporting a number:

• [x] 9 of 15 were a real, fully root-caused bug: tests/test_repo_card_e2e.py's _attach_repo_to_domain helper set the wrong column (domain_id — an unrelated VCS-plugin category string like "code"/"midi" — instead of marketplace_domain_id, the actual marketplace repo-domain link). musehub/services/musehub_domains.py's own docstring explicitly warns about confusing these two columns; the test helper did exactly that. Application code was never wrong — only the test fixture. Fixed; all 9 pass now.
• [x] 1 was a real, clean regression: test_only_safe_env_vars_in_dockerfile failed because the Dockerfile's PLAYWRIGHT_BROWSERS_PATH ENV (needed at build time for the Chromium install, OG cards #129) was never added to the test's approved safe-var list. Not a secret, just a filesystem path — added to the list. Fixed.
• [ ] 5 remain — a test-environment gap, not a code bug: tests/test_opengraph_repo_cards.py fails with RuntimeError: No storage backend configured — these tests need a real or mocked blob storage backend (BLOB_STORAGE_BUCKET/AWS_S3_ASSET_BUCKET) that isn't configured in this environment. Not fixed in this pass — fabricating fake storage credentials isn't a real fix; the actual fix is likely giving these tests a properly mocked storage backend rather than requiring live R2 config to run at all. Tracked as a follow-up.

Net result after fixes: 7,012 passing, 6 failing (environment gap, not app bugs), 24 skipped.

Other items re-verified with real checks, not assumptions:

• [x] Test application rollback — done for real, not just documented (#159): full round trip on staging, confirmed via docker ps that the old image was actually running.
• [x] Verify neither maintainer is a single point of operational failure — now passes. Was failing per the original doc (shared deploy credential, no Aaron AWS identity). Both resolved via #150: Aaron has independent SSO identity + MFA, AdministratorAccess on all three AWS accounts, and can deploy staging/production under his own identity — verified live.
• [x] Test both root and www, verify canonical redirects — tested externally this pass: www.musehub.ai → 301 → https://musehub.ai/, http://musehub.ai → 301 → https://musehub.ai/, both confirmed via curl.
• [x] Test DNS and TLS from external networks — confirmed externally: valid TLS cert (CN=musehub.ai), correct date range, via openssl s_client from outside the AWS network.
• [ ] Test a database restore / backup integrity — already done for production (#157, tested restore), still not done for staging (no staging backups exist yet, #177)
• [ ] Production-like integration tests, explicit auth/authz test confirmation, migration testing, failed-migration handling, instance-failure testing, deploy-during-traffic testing, load/soak testing, timeout/retry/idempotency load-testing, tabletop incident exercise, break-glass-process test, clean-workstation access test, dependency-unavailability testing — none done, genuinely open, no real progress to report on these specifically this pass
• [ ] Test a clean infrastructure deployment in a disposable environment — still N/A, no IaC (#153)
• [ ] Test monitoring/alert delivery — still N/A, no alerts exist (#174)

Section 18 — Cutover Plan

This already happened. Production cut over live on 2026-08-25 — this section should be read retrospectively, not as forward planning. What actually occurred, for the record:

• DNS cutover to the new dedicated Production AWS account, Cloudflare Origin Certificate issued fresh for the new instance, blue/green deploy validated healthy before cutover
• No formal rollback-decision-tree or dual-approval gate existed at the time — cutover was a single-operator (Gabriel) action, consistent with the ownership model at that point in the timeline (Aaron's independent access came later, via #150)
• No backups existed yet at cutover time (#157 landed afterward) — accepted risk at the time, since closed

Worth writing this up properly as a real post-mortem/retrospective doc if a future cutover (e.g., a major version, a re-platform) is ever planned — this pass is not that document, just a factual note that Section 18 isn't "not started," it's "happened without most of the checklist's recommended gates in place, and nothing went wrong."

Section 19 — Post-Launch

Also no longer "premature to schedule" — there is a real launch date now (2026-08-25), production has been live for two weeks as of this writing. Recurring-cadence items worth actually setting up:

• [ ] Monthly dependency updates — no process yet
• [ ] Quarterly DR exercise — backups/restore now exist (#157) to actually exercise; not scheduled
• [ ] Quarterly permissions/cost review — access model is now real enough to review (#150/#152); not scheduled
• [ ] Annual domain review — not scheduled

Not done in this pass: actually setting up calendar reminders or a recurring-task system for these — that requires knowing what calendar/task tool Gabriel wants to use, not something to invent from here. Flagging as ready-to-schedule rather than blocked.