Sections 16–19 — Documentation, Testing, Cutover, Post-Launch

Companion to musehub-production-readiness-checklist.md. Covered together: all four are meta/process sections that depend entirely on Sections 0–15 being real. None of their content can be "done" ahead of the underlying work — this pass records what already exists, what's genuinely blocked until earlier sections close, and one concrete finding (the test suite's actual size).

Section 16 — Documentation and Runbooks

This checklist effort is the start of this section — docs/production-readiness/*.md now covers most of the "document X" items in spirit, even though they weren't written with runbook formatting in mind.

• [ ] System architecture diagram — not created; Section 0/5's prose description could become one
• [ ] Data-flow diagram — not created
• [x] Document every AWS account and region — done, Section 0
• [x] Document DNS and certificate ownership — done, Section 6 (with the Cloudflare correction)
• [ ] Document environment differences — partially, scattered across Sections 0/5; not consolidated
• [ ] Document the IaC repository structure — N/A, no IaC exists yet (Section 4)
• [ ] Document local development setup — not reviewed in this pass (likely exists elsewhere in the repo, e.g. a README — not verified)
• [x] Document staging deployment — already well-documented in docs/infrastructure.md
• [x] Document production deployment — same, docs/infrastructure.md (though prod is currently broken, Section 0)
• [x] Document rollback — documented (IMAGE_TAG=<tag>), not yet tested (Section 11)
• [ ] Document database migration procedures — partially (Section 8); rollback/forward-repair not documented
• [ ] Document database restoration — cannot be documented meaningfully, no backups exist (Section 8)
• [ ] Document secret rotation — not done (Section 7)
• [ ] Document certificate and DNS troubleshooting — not done (though Section 6 documents current state well)
• [ ] Document user-access provisioning and removal — not done (Section 1)
• [ ] Document break-glass access — not done (Section 1)
• [ ] Document incident response — not done
• [ ] Document backup retention and restoration tests — N/A, no backups exist
• [ ] Document common alarms and remediation — N/A, no alarms exist (Section 12)
• [ ] Create a service inventory with owners — partial (Section 0's resource inventory exists; owners don't, Section 1)
• [x] Create a dependency inventory — done, Section 0
• [x] Create an operational decision log — done, this checklist's own Decision Log table
• [x] Record accepted risks and deferred work — done, this checklist's own Accepted Risks table
• [x] Keep documentation versioned beside the relevant code — done by construction (all of this lives in docs/ in the MuseHub repo)
• [ ] Confirm Aaron can follow the documentation without Gabriel's help — not tested
• [ ] Confirm Gabriel can follow it without relying on personal memory — not tested
• [ ] Add a production launch checklist and rollback decision tree — this checklist's Section 18 is the launch checklist; a decision tree specifically doesn't exist
• [ ] Add contact information and escalation paths — not done
• [ ] Review documentation after every incident or major deployment — process doesn't exist yet

Section 17 — Testing Before Launch

• [x] Automated test suite exists and collects cleanly — verified: pytest --collect-only finds 7,028 tests across the repo, no collection errors. Actually running the full suite (pass/fail status) wasn't done in this pass — that's real Section 17 work, not a doc-sweep item.
• [ ] Production-like integration tests — not verified whether any target a staging-like environment specifically
• [ ] Test authentication and authorization — likely covered somewhere in the 7,028 tests; not confirmed
• [ ] Test a clean infrastructure deployment in a disposable environment — N/A, no IaC exists yet
• [ ] Test migration from the current schema — not done as an explicit pre-launch exercise
• [ ] Test application rollback — not done, explicitly required, currently just a documented command
• [ ] Test failed migration handling — not done
• [ ] Test a database restore — cannot happen, no backups exist (Section 8) — this is the actual blocker for several Section 17 items
• [ ] Test backup integrity — same blocker
• [ ] Test task or instance failure — not done
• [ ] Test deployment during active traffic — not done
• [ ] Test autoscaling — N/A, none exists
• [ ] Run basic load and soak tests — not done
• [ ] Verify timeout/retry/idempotency behavior — partially covered by code review (Sections 5/9), not load-tested
• [ ] Test DNS and TLS from external networks — partially done this session (live cert check); not a full external-network sweep
• [ ] Test both root and www, verify canonical redirects — cannot fully pass, no canonical redirect exists yet (Section 6)
• [ ] Test monitoring and alert delivery — N/A, no alerts exist yet (Section 12)
• [ ] Run a tabletop incident exercise — not done
• [ ] Test the break-glass process — N/A, doesn't exist yet (Section 1)
• [ ] Verify neither maintainer is a single point of operational failure — currently fails this test, per Section 1's findings (shared credential, no Aaron AWS identity)
• [ ] Test production access from a clean workstation — not done
• [ ] Test what happens when a secret/DB/queue/external API is unavailable — not done

Section 18 — Cutover Plan

Entirely future work — every item depends on Sections 0–15 being substantially complete first (backups existing, RPO/RTO defined, monitoring live, Aaron having independent access, prod deploy actually working). Not meaningfully startable yet; listing here as "correctly blocked" rather than walking each checkbox individually. The one prerequisite worth calling out now: prod deploy is currently broken (Section 0 — no IAM instance profile), so no cutover can happen until that's fixed, independent of everything else in this checklist.

Section 19 — Post-Launch

Same as Section 18 — entirely contingent on launch having happened. The recurring-schedule items (monthly dependency updates, quarterly DR exercise, quarterly permissions/cost review, annual domain review) are worth setting up as calendar reminders once launch is real, but scheduling them now would be premature — there's no launch date yet, and several are only meaningful once the underlying capability exists (e.g. "quarterly DR exercise" needs backups and a restore procedure to exist first).