# Sections 16–19 — Documentation, Testing, Cutover, Post-Launch > Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md). > Covered together: all four are meta/process sections that depend entirely on Sections 0–15 > being real. None of their content can be "done" ahead of the underlying work — this pass records > what already exists, what's genuinely blocked until earlier sections close, and one concrete > finding (the test suite's actual size). ## Section 16 — Documentation and Runbooks This checklist effort *is* the start of this section — `docs/production-readiness/*.md` now covers most of the "document X" items in spirit, even though they weren't written with runbook formatting in mind. - [ ] System architecture diagram — not created; Section 0/5's prose description could become one - [ ] Data-flow diagram — not created - [x] Document every AWS account and region — done, Section 0 - [x] Document DNS and certificate ownership — done, Section 6 (with the Cloudflare correction) - [ ] Document environment differences — partially, scattered across Sections 0/5; not consolidated - [ ] Document the IaC repository structure — N/A, no IaC exists yet (Section 4) - [ ] Document local development setup — not reviewed in this pass (likely exists elsewhere in the repo, e.g. a README — not verified) - [x] Document staging deployment — already well-documented in `docs/infrastructure.md` - [x] Document production deployment — same, `docs/infrastructure.md` (though prod is currently broken, Section 0) - [x] Document rollback — documented (`IMAGE_TAG=`), not yet tested (Section 11) - [ ] Document database migration procedures — partially (Section 8); rollback/forward-repair not documented - [ ] Document database restoration — **cannot be documented meaningfully, no backups exist** (Section 8) - [ ] Document secret rotation — not done (Section 7) - [ ] Document certificate and DNS troubleshooting — not done (though Section 6 documents current state well) - [ ] Document user-access provisioning and removal — not done (Section 1) - [ ] Document break-glass access — not done (Section 1) - [ ] Document incident response — not done - [ ] Document backup retention and restoration tests — N/A, no backups exist - [ ] Document common alarms and remediation — N/A, no alarms exist (Section 12) - [ ] Create a service inventory with owners — partial (Section 0's resource inventory exists; owners don't, Section 1) - [x] Create a dependency inventory — done, Section 0 - [x] Create an operational decision log — done, this checklist's own Decision Log table - [x] Record accepted risks and deferred work — done, this checklist's own Accepted Risks table - [x] Keep documentation versioned beside the relevant code — done by construction (all of this lives in `docs/` in the MuseHub repo) - [ ] Confirm Aaron can follow the documentation without Gabriel's help — not tested - [ ] Confirm Gabriel can follow it without relying on personal memory — not tested - [ ] Add a production launch checklist and rollback decision tree — this checklist's Section 18 is the launch checklist; a decision tree specifically doesn't exist - [ ] Add contact information and escalation paths — not done - [ ] Review documentation after every incident or major deployment — process doesn't exist yet ## Section 17 — Testing Before Launch - [x] Automated test suite exists and collects cleanly — **verified**: `pytest --collect-only` finds **7,028 tests** across the repo, no collection errors. Actually running the full suite (pass/fail status) wasn't done in this pass — that's real Section 17 work, not a doc-sweep item. - [ ] Production-like integration tests — not verified whether any target a staging-like environment specifically - [ ] Test authentication and authorization — likely covered somewhere in the 7,028 tests; not confirmed - [ ] Test a clean infrastructure deployment in a disposable environment — N/A, no IaC exists yet - [ ] Test migration from the current schema — not done as an explicit pre-launch exercise - [ ] Test application rollback — **not done**, explicitly required, currently just a documented command - [ ] Test failed migration handling — not done - [ ] Test a database restore — **cannot happen, no backups exist** (Section 8) — this is the actual blocker for several Section 17 items - [ ] Test backup integrity — same blocker - [ ] Test task or instance failure — not done - [ ] Test deployment during active traffic — not done - [ ] Test autoscaling — N/A, none exists - [ ] Run basic load and soak tests — not done - [ ] Verify timeout/retry/idempotency behavior — partially covered by code review (Sections 5/9), not load-tested - [ ] Test DNS and TLS from external networks — partially done this session (live cert check); not a full external-network sweep - [ ] Test both root and `www`, verify canonical redirects — **cannot fully pass**, no canonical redirect exists yet (Section 6) - [ ] Test monitoring and alert delivery — N/A, no alerts exist yet (Section 12) - [ ] Run a tabletop incident exercise — not done - [ ] Test the break-glass process — N/A, doesn't exist yet (Section 1) - [ ] Verify neither maintainer is a single point of operational failure — **currently fails this test**, per Section 1's findings (shared credential, no Aaron AWS identity) - [ ] Test production access from a clean workstation — not done - [ ] Test what happens when a secret/DB/queue/external API is unavailable — not done ## Section 18 — Cutover Plan Entirely future work — every item depends on Sections 0–15 being substantially complete first (backups existing, RPO/RTO defined, monitoring live, Aaron having independent access, prod deploy actually working). Not meaningfully startable yet; listing here as "correctly blocked" rather than walking each checkbox individually. The one prerequisite worth calling out now: **prod deploy is currently broken** (Section 0 — no IAM instance profile), so no cutover can happen until that's fixed, independent of everything else in this checklist. ## Section 19 — Post-Launch Same as Section 18 — entirely contingent on launch having happened. The recurring-schedule items (monthly dependency updates, quarterly DR exercise, quarterly permissions/cost review, annual domain review) are worth setting up as calendar reminders once launch is real, but scheduling them now would be premature — there's no launch date yet, and several are only meaningful once the underlying capability exists (e.g. "quarterly DR exercise" needs backups and a restore procedure to exist first).