# MuseHub Production-Readiness Checklist

> Master checklist for promoting MuseHub from staging to production, automating deployments, sharing operational ownership across the team (Gabriel, Aaron, and interns Sol/Toby/Ben), and applying security, reliability, backup, documentation, and infrastructure best practices.
>
> **Team access tiers** (decided 2026-08-08, see [`production-readiness/runbook-deploy-onboarding.md`](production-readiness/runbook-deploy-onboarding.md)):
> Gabriel and Aaron have full staging + production access; interns (Sol, Toby, Ben) are staging-only
> by design. Interim: the current shared AWS credential is not distributed to interns — real
> staging access for them waits on Section 3's individual-identity rollout, which is now scoped for
> all 5 people from the start.

> **Working method:** sections are swept in order, once each. Unresolved items are flagged inline
> and rolled up into the Decision Log / Accepted Risks tables at the bottom rather than blocking
> progress on later sections. See `docs/production-readiness/` for the per-section companion docs
> with full verification detail.

## 0. Decisions and Inventory

> Full detail: [`production-readiness/00-inventory-and-decisions.md`](production-readiness/00-inventory-and-decisions.md)

- [ ] Document the current staging architecture.
  - [x] AWS account and region — `992382692655`, `us-east-1` (verified live)
  - [x] Compute platform: ECS, EC2, App Runner, Elastic Beanstalk, Lambda, EKS, etc. — raw EC2 (`t3.small`/`t3.medium`), Docker Compose, no managed compute service
  - [x] Load balancer, API Gateway, CloudFront, or other ingress — none; Cloudflare proxy → nginx on the instance
  - [x] Database engine and topology — self-hosted PostgreSQL 16, single instance, no RDS, no replica
  - [x] Redis, queues, object storage, search, background workers — background worker (`musehub-runner`) confirmed; object storage confirmed **live on Cloudflare R2** (bucket `musehub-staging`, proven by a successful push this session — `docs/infrastructure.md` corrected); no Redis/search identified
  - [x] VPC, subnets, route tables, NAT gateways, and security groups — default VPC, no custom subnets, no NAT gateway, `musehub-sg` verified live (443 from Cloudflare ranges only)
  - [x] Secrets and configuration sources — hand-provisioned `.env` per instance (not SSM-automated); confirmed it includes working R2 credentials `docs/infrastructure.md` didn't document. Open question for Gabriel: should production adopt the drafted SSM pipeline (`deploy/secrets.sh`) from day one instead of hand-provisioning like staging?
  - [x] Logging and monitoring — corrected (see Section 12): structured JSON logs **are** centralized live in CloudWatch (`/musehub/staging`, ~1.7GB, no retention limit set); alarms/SNS/dashboards on top of them do not exist
  - [x] Current deployment process — documented: manual, human-triggered `deploy/push.sh`, no CI/CD automation
- [x] Inventory every manually created AWS resource. — 2 EC2 + 2 EIP, 1 SG, 1 IAM user, 1 IAM instance role, 1 ECR repo, 1 S3 bucket, 3 SSM params (all listed in companion doc)
- [ ] Inventory every external dependency.
  - [x] GitHub — backup-mirror only, never VCS/CI/collaboration (ecosystem policy)
  - [x] Namecheap — registrar + DNS host for `musehub.ai`
  - [x] Email provider — confirmed none exists
  - [x] OAuth providers — confirmed none exist (MSign/Ed25519 only)
  - [x] Payment or billing providers — confirmed none exist
  - [x] Error tracking — confirmed none exists
  - [ ] Third-party APIs — Cloudflare R2/Worker usage inferred from scripts but not independently verified against the Cloudflare dashboard
- [x] Identify all stateful and mission-critical data. — Postgres DB, object store, `musehub-releases` S3 bucket, SSM params (all listed in companion doc)
- [ ] Define expected initial traffic and reasonable growth assumptions. — **not yet defined, needs Gabriel**
- [ ] Define availability expectations. — **not yet defined, needs Gabriel**
- [ ] Define acceptable downtime during deployment. — **not yet defined, needs Gabriel**
- [ ] Define recovery objectives.
  - [ ] **RPO:** maximum acceptable data loss — **not yet defined, needs Gabriel**
  - [ ] **RTO:** maximum acceptable restoration time — **not yet defined, needs Gabriel**
- [ ] Decide the canonical public URLs.
  - [x] Root (`musehub.ai`) — exists, live
  - [x] `www.musehub.ai` — exists, live
  - [ ] API hostname — none exists; MuseHub serves API + web from the same origin, no `api.` subdomain in use today
  - [x] Staging (`staging.musehub.ai`) — exists, live
- [ ] Choose either the root domain or `www` as canonical. — **not yet decided, needs Gabriel**
- [ ] Redirect the noncanonical hostname permanently to the canonical one. — no redirect logic found in nginx config; not implemented
- [ ] Decide whether DNS remains hosted at Namecheap or moves to Route 53. — **not yet decided, needs Gabriel**
- [x] Decide whether production and non-production will use separate AWS accounts. — **Decided:** single account accepted as technical debt for now (see Decision Log)

---

## 1. Ownership and Shared Access

> No shared human accounts, shared passwords, or shared AWS access keys.
> Full detail: [`production-readiness/01-ownership-and-access.md`](production-readiness/01-ownership-and-access.md).
> "GitHub organization" items below are the MuseHub-native equivalent per the Decision Log.

- [ ] Confirm the domain is held by the company rather than an individual wherever practical. — **needs Gabriel (Namecheap console)**
- [ ] Confirm AWS account ownership and billing contacts. — **needs Gabriel (AWS console)**
- [x] Confirm MuseHub repo ownership/collaboration model. — `gabriel` owns both repos; `aaronrene` has `write` collaborator access on `musehub` and `muse` (verified via `muse hub collaborator list`)
- [ ] Add both Gabriel and Aaron as appropriate organizational owners or administrators. — done for MuseHub repos (Aaron has `write`); **not done for AWS/Namecheap — needs Gabriel**
- [ ] Ensure neither person is the sole recovery path for:
  - [ ] Namecheap — **needs Gabriel**
  - [ ] AWS — **needs Gabriel**
  - [x] MuseHub — Aaron already has `write` access independent of Gabriel
  - [ ] DNS — **needs Gabriel**
  - [ ] Database — **needs Gabriel** (currently only reachable via `musehub-infra` shared AWS credential)
  - [ ] Production secrets — **needs Gabriel**
  - [ ] Billing — **needs Gabriel**
  - [ ] Backups — **needs Gabriel** (also blocked on resolving Section 0 Contradiction 2)
- [ ] Use company-controlled email addresses for infrastructure accounts. — **needs Gabriel**
- [ ] Configure at least two trusted recovery paths. — **needs Gabriel**
- [ ] Store emergency recovery documentation in a location accessible to both founders. — **needs Gabriel**
- [ ] Create a documented joiner–mover–leaver access procedure. — not started
- [ ] Create a quarterly access-review process. — not started
- [ ] Record ownership for every operational area:
  - [ ] AWS administration — currently Gabriel only, undocumented
  - [ ] DNS and certificates — currently Gabriel only, undocumented
  - [ ] Deployments — currently Gabriel only (shared `musehub-infra` credential), undocumented
  - [ ] Database administration — currently Gabriel only, undocumented
  - [ ] Incident response — no process exists
  - [ ] Billing — currently Gabriel only, undocumented
  - [ ] Backups — no automated backups exist yet (Section 0 Contradiction 2)
- [ ] Create an emergency break-glass procedure. — not started
- [ ] Test that Aaron can independently:
  - [ ] Sign in to AWS — **cannot test — no AWS identity exists for Aaron yet**
  - [ ] Inspect production — **cannot test**
  - [ ] Deploy staging — **cannot test as designed** — today's deploy uses a shared AWS credential (`musehub-infra`), not an individual one; this violates the "no shared AWS access keys" principle this section opens with
  - [ ] Approve or deploy production — **cannot test** — prod deploy is currently broken regardless (Section 0: no IAM instance profile on the prod instance)
  - [ ] Inspect logs and alarms — **cannot test — no logs/alarms exist yet** (Section 12)
  - [ ] Restore from backup — **cannot test — no backups exist yet** (Section 0 Contradiction 2)
  - [ ] Update DNS when authorized — **cannot test — Aaron's Namecheap access unverified**
- [ ] Test that Gabriel can perform the same tasks without relying on undocumented knowledge. — not started; current process is entirely undocumented outside `docs/infrastructure.md`

---

## 2. AWS Account Structure

> Full detail: [`production-readiness/02-aws-account-structure.md`](production-readiness/02-aws-account-structure.md)

### Recommended Target

- [ ] Create or verify an AWS Organization. — **deferred, see Decision Log** (single account accepted as technical debt)
- [ ] Use distinct AWS accounts for:
  - [ ] Management — deferred
  - [ ] Production — deferred
  - [ ] Non-production — deferred
- [ ] Consider separate security and log-archive accounts as the operation grows. — deferred
- [ ] Place production and non-production into separate organizational units. — deferred (no Organization exists)
- [ ] Apply Service Control Policies only after testing them carefully. — N/A while deferred
- [ ] Centralize billing. — N/A, single account
- [ ] Enable account-level budget alerts. — not done; tracked under Section 15 (Cost Controls) instead since it applies within the single account regardless of structure — **needs Gabriel (AWS console access required)**
- [ ] Set alternate security, operations, and billing contacts on every account. — **needs Gabriel**
- [ ] Restrict enabled AWS regions to the regions MuseHub actually uses, where feasible. — **needs Gabriel**; note the account is shared with Stori (region `us-east-2` bucket observed), so any region restriction must account for both projects
- [x] Document any temporary decision to keep staging and production in one AWS account as accepted technical debt. — done, see Decision Log and Accepted Risks in this checklist

---

## 3. Identity, Authentication, and Permissions

> Full detail: [`production-readiness/03-identity-and-permissions.md`](production-readiness/03-identity-and-permissions.md)
> Document-only pass — no IAM changes made. Real build happens after the full sweep.

### Root and Recovery Access

- [ ] Protect every AWS root account with phishing-resistant MFA where possible. — **needs Gabriel (AWS console)**
- [ ] Do not create root access keys. — **needs Gabriel to confirm**
- [ ] Do not use root for routine administration. — **needs Gabriel to confirm**
- [ ] Secure the root email account with MFA. — **needs Gabriel**
- [ ] Store root recovery information securely. — **needs Gabriel**
- [ ] Document the legitimate uses of root. — not started
- [ ] Configure multi-person approval for emergency root use if practical. — not started

### Human Access

- [ ] Configure AWS IAM Identity Center. — not started; visibility blocked (`sso-admin` denied) — **needs Gabriel to check/build**
- [ ] Create separate identities for Gabriel and Aaron. — not started (blocks Section 1's independent-operation test)
- [ ] Require MFA for both. — not started
- [ ] Prefer hardware security keys or passkeys. — not started
- [x] Define permission sets, for example:
  - [x] `Administrator` — decided: Gabriel
  - [x] `ProductionOperator` — decided: Aaron (staging + production)
  - [x] `StagingOperator` — decided: Sol, Toby, Ben (interns — staging-only, no production access; see [`runbook-deploy-onboarding.md`](production-readiness/runbook-deploy-onboarding.md))
  - [ ] `ReadOnly` — not started
  - [ ] `Billing` — not started
- [ ] Give routine access through the least-privileged applicable role. — not started
- [ ] Require explicit elevation for administrative work. — not started
- [ ] Avoid permanent `AdministratorAccess` for everyday use where practical. — not started
- [ ] Use IAM Access Analyzer to inspect and refine permissions. — not started; needs broader read access
- [ ] Review unused roles, users, policies, and credentials. — not started; needs broader read access
- [ ] Document who can approve and perform production changes. — not started

### Workload Access

- [x] Give every application or service its own IAM role. — 3 separate IAM users confirmed live (`musehub-infra`, `musehub-app`, `stori-assets-app`), no shared workload credential across services
- [x] Do not place AWS access keys in source code. — confirmed via grep, none found
- [x] Do not place long-lived AWS access keys in GitHub secrets. — N/A, no GitHub Actions/secrets used (see Decision Log); equivalent check for MuseHub-runner CI config deferred to Section 9
- [ ] Restrict every deployment role to:
  - [ ] The correct repository — N/A, no deployment role exists yet (deploys use a human + shared credential)
  - [ ] The correct branch, tag, or GitHub environment — N/A, same reason
  - [ ] The correct AWS account — N/A, same reason
  - [ ] The required AWS resources only — partially true of `musehub-infra`'s scoping, but it's a human-shared credential, not a deployment role
- [ ] Separate staging and production deployment roles. — **not done** — both environments share the `musehub-infra` credential
- [ ] Validate trust policies and permission boundaries. — not started; needs broader read access
- [ ] Prefer temporary credentials everywhere. — **not done** — all current credentials are long-lived IAM user access keys

---

## 4. Infrastructure as Code

> Full detail: [`production-readiness/04-iac.md`](production-readiness/04-iac.md)
> Document-only pass. No IaC written yet — deferred to the build phase after the full sweep.

- [ ] Select one Infrastructure-as-Code system:
  - [ ] Terraform/OpenTofu — **recommended**, not yet formally chosen by Gabriel
  - [ ] AWS CDK
  - [ ] CloudFormation
- [ ] Capture the existing staging infrastructure in code. — not started; full resource list is in the [Section 0 doc](production-readiness/00-inventory-and-decisions.md)
- [ ] Stop making undocumented console-only changes. — behavioral change, not yet in effect
- [ ] Build reusable modules rather than duplicating entire configurations. — N/A until IaC exists
- [ ] Provide separate environment configuration for:
  - [ ] Staging — N/A until IaC exists
  - [ ] Production — N/A until IaC exists
- [ ] Keep production and staging state isolated. — N/A until IaC exists
- [ ] Encrypt remote IaC state. — N/A until IaC exists
- [ ] Enable versioning and locking for remote state. — N/A until IaC exists
- [ ] Restrict access to IaC state because it may contain sensitive metadata. — N/A until IaC exists
- [ ] Pin provider, module, and action versions. — N/A until IaC exists
- [ ] Add formatting and validation checks. — N/A until IaC exists
- [ ] Add static security scanning. — N/A until IaC exists
- [ ] Require a reviewed plan before production changes. — N/A until IaC exists
- [ ] Detect configuration drift. — N/A until IaC exists
- [ ] Document how to import existing resources into IaC. — not started; note this must be *import*, not recreate, since real resources already exist and are live
- [ ] Add environment-standard tags:
  - [ ] Application — not applied anywhere currently
  - [ ] Environment — not applied anywhere currently
  - [ ] Owner — not applied anywhere currently
  - [ ] Managed-by — not applied anywhere currently
  - [ ] Cost-center — not applied anywhere currently
  - [ ] Data-classification — not applied anywhere currently
- [ ] Make a fresh environment reproducible from source control and documented prerequisites. — not started
- [ ] Add a teardown guard so production cannot be accidentally destroyed. — N/A until IaC exists
- [ ] Add deletion protection to critical resources. — not started (also relevant to Section 8 — DB deletion protection specifically)
- [ ] Add policy checks that prevent production resources from being publicly exposed unintentionally. — N/A until IaC exists
- [ ] Ensure IaC changes are reviewed by someone other than the author when practical. — N/A until IaC exists

---

## 5. Production Architecture

> Full detail: [`production-readiness/05-production-architecture.md`](production-readiness/05-production-architecture.md)

- [ ] Reassess whether staging's sizing and topology are production-appropriate. — flagged: staging is `t3.medium`, prod is `t3.small` — inconsistent, no documented reason
- [x] Do not copy staging data into production unless explicitly sanitized and required. — not done, and correctly not done — no evidence of staging→prod data copying anywhere
- [ ] Use separate production:
  - [x] Database — separate instance entirely (different EC2 host)
  - [ ] Credentials — unresolved pending Section 0 Contradiction 2
  - [ ] Secrets — unresolved pending Section 0 Contradiction 2
  - [ ] Object-storage buckets — unresolved pending Section 0 Contradiction 1
  - [ ] Queues — none exist in either environment
  - [ ] Cache — none exist in either environment
  - [ ] Logs — not separated in any meaningful way; no centralized logging exists yet (Section 12)
  - [ ] Encryption keys where appropriate — not reviewed yet
- [ ] Deploy production across at least two Availability Zones where supported and justified. — **not done**; single instance, no ASG, documented as intentional SPOF below
- [x] Keep databases and internal services in private subnets. — partially true in spirit (DB only reachable via the app's Docker network, not the public internet) but not true architecturally (default VPC, no real private subnet); security group is the actual control here
- [x] Expose only the public ingress layer. — confirmed: only 443 open, restricted to Cloudflare IP ranges
- [x] Eliminate direct public database access. — confirmed, DB has no public exposure
- [x] Use security groups by service relationship rather than broad CIDR ranges. — reasonable substitution given the architecture: scoped to Cloudflare's published IP ranges, not `0.0.0.0/0`
- [x] Prohibit unrestricted administrative ports such as public SSH. — confirmed, no port 22 open
- [x] Use AWS Systems Manager Session Manager rather than public SSH where applicable. — confirmed, this is already the only access path
- [x] Configure health checks. — confirmed: `/healthz` probes both DB and object storage, gates blue/green flips
- [ ] Configure graceful application shutdown. — **not verified** — no SIGTERM handling found in `musehub/main.py`; uvicorn defaults apply
- [ ] Configure connection draining. — partial: blue/green waits for new slot health before flip, but old slot's in-flight requests during `docker rm` aren't verified graceful
- [x] Set sensible application and load-balancer timeouts. — confirmed: deliberate per-route nginx timeouts (5s health, 60s default, up to 3600s for streaming/MCP)
- [ ] Configure autoscaling or document why fixed capacity is appropriate. — neither done; fixed capacity with no written justification
- [ ] Set resource requests, limits, or instance sizes deliberately. — instance types chosen but not justified with load data
- [ ] Verify production can survive a routine instance or task replacement. — **not verified**; single-instance architecture means instance failure has no automatic failover (documented SPOF below)
- [x] Confirm background jobs are idempotent or safely retryable. — confirmed: `claim_next_job` uses `SELECT ... FOR UPDATE SKIP LOCKED` for safe concurrent claiming, `reclaim_stale_jobs` resets crashed-worker jobs back to pending (see Section 9 doc)
- [ ] Confirm database migrations are compatible with rolling deployments. — pipeline runs migrations before slot flip (right shape); expand/contract discipline not reviewed — deferred to Section 8
- [x] Add rate limiting and abuse controls. — confirmed: `slowapi`-based per-route limits + global baseline + bot-throttle middleware, already implemented
- [ ] Consider AWS WAF for public endpoints. — N/A as literally stated (no ALB/CloudFront); Cloudflare is the substitute layer, but its WAF/rate-limit config is unverified — **needs Gabriel to check Cloudflare dashboard**
- [x] Configure DDoS protections appropriate to the architecture. — Cloudflare edge proxy already provides this; not independently verified beyond "Cloudflare is in front of everything"
- [ ] Establish AWS service quota monitoring for relevant resources. — not started
- [x] Document every intentional single point of failure. — done, see companion doc: single EC2 per environment, single Postgres instance, single AWS account (shared with Stori), single Cloudflare account, single human deployer with shared credential
- [ ] Create a plan to remove unacceptable single points of failure. — **needs Gabriel** — depends on RPO/RTO/availability targets that are still undefined (Section 0)

---

## 6. DNS, Domain, and TLS

> Full detail: [`production-readiness/06-dns-domain-tls.md`](production-readiness/06-dns-domain-tls.md)
> ⚠️ **Finding:** DNS is actually hosted at **Cloudflare**, not Namecheap — verified via live
> `dig NS musehub.ai`. `docs/infrastructure.md` is stale on this point. Namecheap is registrar
> only. This changes several items below from "Namecheap vs Route 53" to "Cloudflare vs Route 53."

### Namecheap Account

- [ ] Enable MFA on Namecheap for every authorized administrator. — **needs Gabriel**
- [ ] Use unique individual access rather than shared credentials where the account model permits. — **needs Gabriel**
- [ ] Enable domain lock. — **needs Gabriel**
- [ ] Verify registrant and recovery contact details. — **needs Gabriel**
- [ ] Enable automatic renewal. — **needs Gabriel**
- [ ] Verify the payment method will not unexpectedly expire. — **needs Gabriel**
- [ ] Document who may change nameservers or DNS. — **needs Gabriel**; note DNS itself is now on Cloudflare, so this also means documenting Cloudflare access, not just Namecheap
- [ ] Export or document all existing DNS records before making changes. — **needs Gabriel, from the Cloudflare dashboard**, not Namecheap's Advanced DNS tab
- [ ] Preserve mail-related MX, SPF, DKIM, and DMARC records. — **needs Gabriel to confirm these exist and are preserved in Cloudflare's DNS**
- [ ] Confirm both Gabriel and Aaron have an approved recovery path. — **needs Gabriel**, for both Namecheap and Cloudflare

### DNS Design

- [ ] Decide between:
  - [x] Namecheap-hosted DNS — **not the current state**; superseded by the finding above
  - [ ] Route 53-hosted DNS with Namecheap remaining registrar — open; recommendation below
- [x] ~~Prefer Route 53...~~ — **recommendation revised**: current DNS host is Cloudflare (not Namecheap), and Cloudflare is also doing edge proxy/TLS/possibly R2 — recommend staying on Cloudflare DNS rather than fragmenting the stack by moving to Route 53, unless Gabriel has a specific reason to want AWS-native alias records
- [ ] Create a Route 53 hosted zone if selected. — N/A per above recommendation, pending Gabriel's decision
- [x] Reproduce every required existing record before switching nameservers. — N/A, no switch planned per recommendation
- [ ] Lower DNS TTL ahead of cutover. — deferred to Section 18 (Cutover Plan)
- [x] Configure the root/apex record. — exists, live, resolves via Cloudflare
- [x] Configure `www`. — exists, live
- [ ] Configure `api` if needed. — no `api.` hostname exists; MuseHub serves API + web from one origin — confirm this is intentional
- [x] Keep staging on a clearly separate hostname. — confirmed, `staging.musehub.ai`
- [x] Do not expose internal AWS service hostnames unnecessarily. — confirmed, only the public hostnames resolve; no raw AWS DNS names exposed
- [ ] Verify IPv4 and IPv6 behavior. — not tested in this pass
- [ ] Configure a canonical-host redirect. — **not implemented** — no redirect logic found; root and `www` both resolve without a canonical redirect
- [ ] Test DNS from multiple resolvers after cutover. — deferred to Section 18
- [ ] Raise TTL again after the deployment is stable. — deferred to Section 18
- [ ] Manage DNS records through IaC where practical. — deferred to Section 4 build phase (Terraform's `cloudflare` provider)

### TLS

- [ ] Request an AWS Certificate Manager certificate. — **likely N/A for this architecture** — TLS terminates at Cloudflare's edge, not an AWS ALB/CloudFront; needs Gabriel to confirm there's no AWS-facing TLS need I'm missing
- [ ] Include every required hostname. — N/A pending above
- [ ] Prefer DNS validation. — N/A pending above
- [ ] Retain ACM validation records for automatic renewal. — N/A pending above
- [x] Redirect all HTTP traffic to HTTPS. — Cloudflare/nginx setup already enforces HTTPS-only (port 80 not open at all)
- [x] Use modern TLS policies. — Cloudflare-managed edge TLS; verified live cert is current (Let's Encrypt, issued 2026-08-01)
- [x] Configure HSTS only after HTTPS and subdomain behavior are verified. — already live: `Strict-Transport-Security` with `includeSubDomains` confirmed in nginx config
- [ ] Confirm certificate-expiration monitoring. — **not done** — no alerting exists yet (Section 12); manual verification only (as done earlier this session)
- [x] Test the full certificate chain. — verified via `openssl s_client` this session; valid, correctly issued, correct CN
- [ ] Confirm cookies use:
  - [x] N/A — MuseHub uses header-based MSign auth, no cookies exist anywhere in the app

---

## 7. Secrets and Application Configuration

> Full detail: [`production-readiness/07-secrets-and-config.md`](production-readiness/07-secrets-and-config.md)

- [x] Inventory all secrets. — done, 6 secrets identified (DB password, webhook key, runner token, R2 credentials, worker key, worker URL)
- [x] Remove secrets from source control and repository history. — confirmed via grep, none found
- [ ] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate. — **the real work item**: `deploy/secrets.sh` implements this but only 3/7 required SSM params exist; not live yet
- [ ] Encrypt secrets with KMS. — partial; the 3 existing SSM params are SecureString/KMS-encrypted, rest aren't in SSM
- [x] Use separate staging and production secrets. — confirmed: separate `.env`, separate R2 buckets, separate SSM prefixes
- [ ] Give workloads access only to the secrets they require. — likely N/A for a single-process-per-instance app; not force-fit
- [x] Do not expose production secrets to pull-request workflows. — N/A, no PR-based CI exists
- [ ] Do not print secrets in logs. — not verified; no centralized logs exist yet to audit (Section 12)
- [ ] Configure secret rotation where supported. — not done
- [ ] Rotate credentials during the production launch. — deferred to Section 18
- [ ] Document emergency rotation procedures. — not done
- [x] Separate secrets from nonsecret configuration. — confirmed: `musehub/config.py`'s `Settings` class already separates these cleanly in code
- [x] Validate required configuration at application startup. — confirmed via `pydantic_settings.BaseSettings`
- [x] Fail safely when critical configuration is missing. — confirmed, same mechanism plus `secrets.sh`'s own preflight `die` checks
- [x] Remove default credentials and sample keys. — confirmed, no placeholder secrets in `config.py`; `secrets.sh` also actively rejects known-weak passwords
- [ ] Audit OAuth callback URLs and API allowlists for production. — N/A, no OAuth exists
- [ ] Audit every external webhook secret. — `WORKER_INTERNAL_KEY` identified; validation logic in `musehub/worker.py` not independently reviewed in this pass
- [ ] Document ownership and rotation frequency for every production secret. — not done, blocked on Section 1

---

## 8. Database Security, Migrations, and Backups

> Full detail: [`production-readiness/08-database-backups-migrations.md`](production-readiness/08-database-backups-migrations.md)
> ⚠️ **Verified live via SSM: there is currently no usable backup of the database, anywhere.**

### Database Security

- [ ] Enable encryption at rest. — not verified in this pass
- [ ] Enforce encryption in transit. — not verified in this pass
- [ ] Use a production-specific KMS key where appropriate. — not started
- [ ] Store credentials in Secrets Manager. — see Section 7
- [ ] Avoid administrator credentials for normal application traffic. — **not done** — single `musehub` role used for app traffic, migrations, and admin access
- [ ] Create separate roles for:
  - [ ] Application runtime — not done
  - [ ] Migrations — not done
  - [ ] Read-only operations — not done
  - [ ] Administration — not done
- [x] Keep the database off the public internet. — confirmed
- [x] Restrict security-group access to the application layer. — confirmed
- [ ] Enable deletion protection. — not applicable in the RDS sense (self-hosted); EBS/snapshot-level protection not configured
- [ ] Enable database-level audit logging where appropriate. — not enabled
- [ ] Monitor storage, CPU, memory, connections, locks, and slow queries. — partial: app-level slow-query logging exists (`slow_query_threshold_ms` in `musehub/config.py`); no infrastructure-level DB monitoring exists

### Backups and Recovery

- [ ] Enable automated backups. — **not done, verified live**: no crontab installed, `rclone` not present, one stale 20-byte manual backup from ~2 months ago is the only thing that exists
- [ ] Set an explicit retention period. — N/A, no backups running
- [ ] Enable point-in-time recovery where supported. — not available for self-hosted Postgres without WAL archiving, which isn't configured
- [ ] Configure scheduled longer-term snapshots. — not done
- [ ] Configure cross-account backup copies for critical production data. — not done
- [ ] Consider cross-region copies based on the agreed disaster model. — not done
- [ ] Protect backup vaults from routine production credentials. — N/A, no backups exist
- [ ] Encrypt backup recovery points. — N/A, no backups exist
- [ ] Enable backup-vault lock or equivalent immutability where justified. — N/A, no backups exist
- [ ] Monitor backup-job failures. — N/A, no backup job exists
- [ ] Define retention and deletion policies. — not done
- [ ] Document the restoration procedure. — not done
- [ ] Perform an actual restore test before launch. — **cannot happen until backups exist**
- [ ] Schedule recurring restore tests. — N/A yet
- [ ] Measure restoration time against the RTO. — N/A, RTO undefined (Section 0) and no backups exist
- [ ] Confirm restored data satisfies the RPO. — N/A, same reasons
- [ ] Record the latest successful restore test. — N/A
- [ ] Verify backups are restorable by both Gabriel and Aaron. — N/A
- [ ] Back up any other mission-critical state, including object storage and configuration data. — object storage is content-addressed R2 (durable by design, per Cloudflare); `.env` configuration data has no backup of its own beyond what's in this documentation

### Schema Migrations

- [x] Run migrations automatically but as an explicit deployment step. — confirmed, part of `deploy.sh`, before slot swap
- [ ] Ensure only one migration runner executes at a time. — not independently verified; no explicit lock mechanism found
- [ ] Back up before high-risk migrations. — moot until backups exist
- [ ] Prefer expand-and-contract migrations. — not reviewed across the 75 existing migrations in this pass
- [ ] Avoid combining destructive schema changes with dependent application changes in one irreversible step. — not reviewed
- [ ] Test migrations against a production-like dataset. — not verified
- [ ] Define rollback or forward-repair procedures. — not documented
- [x] Record migration state and duration. — implicitly via Alembic's `alembic_version` table and deploy logs; no structured/queryable record beyond that
- [ ] Alert on migration failure. — not done, no alerting exists (Section 12)
- [x] Ensure older and newer application versions can coexist during rolling or blue/green deployments where required. — pipeline ordering (migrate-then-swap) is the right shape; ⚠️ **found a real risk**: on `alembic upgrade head` failure, `deploy.sh` runs `alembic stamp --purge head` and retries, which could mask a genuinely broken migration as a "stale revision ID" — mitigated by a hard-failing schema-parity gate afterward, but worth explicit awareness since that gate is the only thing standing between this pattern and a silently-broken deploy

---

## 9. CI Pipeline

> Full detail: [`production-readiness/09-ci-pipeline.md`](production-readiness/09-ci-pipeline.md)
> ⚠️ **Bigger finding than a typical checklist item**: MuseHub-native equivalents of branch
> protection, required status checks, and CODEOWNERS **don't exist as product features yet** —
> this section requires building platform capability, not just configuring deployment. What does
> exist and is genuinely good: a concurrency-safe internal background job queue.

- [ ] Require proposals (MuseHub's PR-equivalent) for protected branches. — not started, feature doesn't exist
- [ ] Protect `main`/`dev`. — not started
- [ ] Require CI checks before merging. — not started; `merge_proposal` has no CI-status gate
- [ ] Require review from another maintainer for sensitive changes. — not started
- [ ] Define required-reviewer paths (`CODEOWNERS`-equivalent) for:
  - [ ] Infrastructure — not started
  - [ ] Deployment workflows — not started
  - [ ] Authentication and security — not started
  - [ ] Database migrations — not started
- [ ] Run:
  - [ ] Formatting — manual only, not automated on push
  - [ ] Linting — manual only
  - [ ] Type checking — manual only
  - [ ] Unit tests — manual only
  - [ ] Integration tests — manual only
  - [ ] Migration checks — runs at deploy time (`schema_gate`, see Section 8), not at CI/proposal time
  - [ ] Dependency vulnerability scanning — not automated
  - [ ] Secret scanning — not automated
  - [ ] Container or artifact scanning — not automated
  - [ ] IaC validation and security scanning — N/A, no IaC exists yet (Section 4)
- [ ] Generate immutable, versioned build artifacts. — partial: ECR image tags exist and are immutable once pushed, but built manually per-environment rather than automatically per-commit
- [x] Record the Git commit SHA in every artifact. — N/A concept (no git); MuseHub commit IDs are content-addressed (`sha256:...`) and could be embedded in image tags — not currently done, worth adding
- [ ] Produce the artifact once and promote the same artifact through environments. — **not done by default**; `IMAGE_TAG` redeploy support in `push.sh` makes promotion *possible* but isn't the default flow
- [ ] Generate an SBOM where practical. — not started
- [x] Pin third-party GitHub Actions to immutable commit SHAs for sensitive workflows. — N/A, no GitHub Actions used
- [x] Minimize GitHub Actions token permissions. — N/A, no GitHub Actions used
- [ ] Avoid executing untrusted pull-request code with production-capable credentials. — moot until proposal-triggered CI execution exists at all
- [ ] Configure dependency update automation. — not started
- [ ] Define an urgent-patch workflow. — not documented
- [ ] Retain CI and deployment logs for an explicit period. — N/A, no CI exists to log yet
- [ ] Prevent force-pushes and branch deletion on protected branches. — not started

---

## 10. Automated Staging Deployment

> Full detail: [`production-readiness/10-11-automated-deployment.md`](production-readiness/10-11-automated-deployment.md)
> Deploys today are 100% manual (`bash deploy/push.sh staging`, human-triggered, shared credential).

- [ ] Create a MuseHub-native staging-environment concept (GitHub environment equivalent). — doesn't exist
- [ ] Deploy automatically after successful merges. — not started, no trigger mechanism exists
- [ ] Authenticate to AWS through short-lived/OIDC-equivalent credentials. — not started
- [x] Do not store long-lived AWS keys in CI config. — moot today (no CI config exists); the actual problem is the shared laptop credential (Section 1/3), not a CI secret leak
- [ ] Restrict the staging role to the staging AWS account and resources. — not started (Section 3)
- [ ] Run infrastructure planning. — N/A, no IaC yet (Section 4)
- [ ] Apply approved or policy-compliant infrastructure changes. — N/A, same reason
- [x] Deploy the immutable application artifact. — already true today, manually
- [x] Run database migrations. — already true, automatic within the deploy
- [ ] Run smoke tests. — not automated as a gate; `deploy/smoke_muse.sh` exists but its coverage wasn't reviewed in this pass
- [x] Run health checks. — already true, gates the blue/green flip
- [ ] Publish the deployment URL and commit SHA. — not automated
- [ ] Notify both maintainers of success or failure. — not automated
- [ ] Preserve deployment logs. — not done; currently only in the deployer's terminal
- [ ] Automatically roll back application deployment when health checks fail, where safe. — not automated, but effectively moot: the health-gated blue/green means a failing new slot never goes live in the first place, rather than going live and needing rollback
- [ ] Keep database rollback separate and explicit. — not formalized
- [ ] Prevent concurrent staging deployments from colliding. — not enforced
- [ ] Make the deployment status visible in MuseHub. — not built
- [ ] Record artifact digest, migration version, deployer, and timestamp. — not recorded anywhere structured

---

## 11. Automated Production Deployment

> Full detail: [`production-readiness/10-11-automated-deployment.md`](production-readiness/10-11-automated-deployment.md)
> ⚠️ Reminder: prod deploy is currently broken regardless of automation — no IAM instance profile
> on the prod instance (Section 0), so `push.sh prod` fails outright even run manually today.

- [ ] Create a MuseHub-native production-environment concept. — doesn't exist
- [ ] Use a distinct production AWS role. — not started (Section 3)
- [ ] Restrict that role to the production account and resources. — not started
- [ ] Trigger production deployment through an explicit release action:
  - [ ] Version tag — MuseHub has `muse release add`/`muse hub release create` already; not wired to deploy
  - [ ] MuseHub release — same as above
  - [ ] Manual promotion of a verified artifact — possible via `IMAGE_TAG=<tag>`, not the default flow
- [ ] Do not rebuild the application for production. — **not done by default** today
- [ ] Promote the exact artifact tested in staging. — possible, not automatic
- [ ] Require successful staging deployment and tests first. — not enforced
- [ ] Require production approval. — not built
- [ ] Prevent self-approval when practical. — not built
- [ ] Permit both Gabriel and Aaron to approve. — not built; also blocked since Aaron has no independent deploy identity yet (Section 1/3)
- [ ] Require approval from someone other than the person initiating the deployment when practical. — not built
- [ ] Display the complete deployment plan before approval. — not built
- [ ] Back up stateful systems before risky changes. — **currently impossible to do meaningfully** — no backups exist (Section 8)
- [x] Run migrations as a controlled step. — already true
- [ ] Deploy using:
  - [ ] Rolling deployment — N/A, using blue/green instead
  - [x] Blue/green deployment — already true, this is the current strategy
  - [ ] Canary deployment — N/A, not the chosen strategy
- [ ] Run post-deployment smoke tests. — not automated
- [ ] Monitor error rate, latency, and health during rollout. — not possible yet, no monitoring exists (Section 12)
- [ ] Stop or roll back when thresholds are exceeded. — N/A, no thresholds/monitoring exist
- [ ] Record who approved and initiated each deployment. — not recorded
- [ ] Record commit SHA, artifact identifier, and migration version. — not recorded
- [ ] Generate release notes. — not automated
- [ ] Notify both operators after completion. — not automated
- [ ] Prevent concurrent production deployments. — not enforced
- [x] Add a documented manual deployment path for emergencies. — exists (it's the current default path)
- [x] Add a documented rollback command or workflow. — exists (`IMAGE_TAG=<previous-tag> bash deploy/push.sh prod`)
- [ ] Test rollback before launch. — **not yet done**, explicitly required

---

## 12. Observability

> Full detail: [`production-readiness/12-observability.md`](production-readiness/12-observability.md)
> ⚠️ **Correction to Section 0/5**: logging is further along than earlier stated in this sweep —
> structured JSON logs ARE centralized live in CloudWatch (`/musehub/staging`, ~1.7GB, verified via
> `aws logs tail`). It's specifically alarms/dashboards/metrics that don't exist, not logging itself.

- [x] Centralize application logs. — confirmed live, CloudWatch Logs `/musehub/staging`
- [x] Use structured logs. — confirmed, JSON with rich per-request fields (`musehub/logging_config.py`)
- [ ] Include:
  - [x] Timestamp — present
  - [ ] Environment — **missing** from the JSON schema
  - [ ] Service — implicit via `logger` field, not an explicit `service` field
  - [x] Request or correlation ID — present (`request_id`)
  - [ ] Release version — **missing** from the JSON schema
  - [x] Severity — present (`level`)
- [x] Prevent credentials, tokens, and sensitive personal data from entering logs. — confirmed, `PiiFilter` scrubs Bearer tokens/passwords/secrets already
- [ ] Set explicit log-retention periods. — **not done**, currently unlimited retention on the live log group
- [ ] Encrypt logs. — CloudWatch default at-rest encryption applies; customer-managed KMS key not confirmed
- [ ] Collect infrastructure and application metrics. — partial: app-level data flows into logs (could feed metric filters); true instance-level metrics (CPU/disk/memory via CloudWatch Agent) not confirmed installed
- [ ] Build dashboards for:
  - [ ] Request rate — data exists in logs, no dashboard
  - [ ] Error rate — data exists in logs, no dashboard
  - [ ] Latency — data exists in logs (`duration_ms`), no dashboard
  - [ ] Saturation — not collected
  - [ ] Task or instance health — `/healthz` exists but isn't dashboarded
  - [ ] Database capacity — not collected
  - [ ] Database connections — not collected
  - [ ] Queue depth — the internal job queue (Section 9) has no depth metric exposed
  - [ ] Background-job failures — job status is tracked in the DB (`attempt`, `status`) but not surfaced to any dashboard
- [ ] Add external uptime checks. — not done
- [ ] Configure alerts for:
  - [ ] Site unavailable — not done
  - [ ] Elevated 5xx responses — not done (data available, not wired)
  - [ ] Elevated latency — not done (data available, not wired)
  - [ ] Unhealthy deployment — not done
  - [ ] Database storage pressure — not done
  - [ ] CPU or memory exhaustion — partial signal exists (`[memory] HIGH RSS` app-level warnings already fire), not wired to an actual alert
  - [ ] Backup failure — N/A, no backup job exists yet (Section 8)
  - [ ] Certificate problems — not done
  - [ ] Queue backlog — not done
  - [ ] Security findings — not done
  - [ ] Unexpected cost growth — not done (Section 15)
- [ ] Route alerts to both Gabriel and Aaron. — N/A, no alerts exist
- [ ] Define warning versus page-worthy alert severity. — not done
- [ ] Test alert delivery. — N/A
- [ ] Ensure every urgent alert links to a runbook. — N/A
- [ ] Configure error tracking with release correlation. — not done, no tool exists
- [ ] Define service-level indicators and initial service objectives. — not done, blocked on Section 0's undefined availability targets
- [ ] Add deployment markers to dashboards. — N/A, no dashboards exist
- [ ] Verify logs and metrics remain available during an application outage. — not tested, but likely true by construction (CloudWatch Logs is a separate AWS service from the EC2 instance)

---

## 13. Security Monitoring and Auditability

> Full detail: [`production-readiness/13-security-monitoring.md`](production-readiness/13-security-monitoring.md)
> Most items **blocked** on the broader-read-access grant Gabriel agreed to earlier this session —
> re-run once that lands.

- [ ] Enable CloudTrail. — **blocked**, cannot confirm current state
- [ ] Use a multi-region trail for management activity. — **blocked**
- [ ] Include read and write management events. — **blocked**
- [ ] Encrypt CloudTrail logs. — **blocked**
- [ ] Enable log-file validation. — **blocked**
- [ ] Store logs in a nonpublic, access-restricted bucket. — **blocked**
- [ ] Send relevant CloudTrail events to CloudWatch Logs. — **blocked**
- [ ] Enable AWS Config where cost and scope permit. — **blocked**
- [ ] Enable GuardDuty. — **blocked**
- [ ] Enable Security Hub and AWS Foundational Security Best Practices controls. — **blocked**
- [ ] Enable IAM Access Analyzer. — **blocked**
- [ ] Review public and cross-account resource access. — **blocked**
- [ ] Enable S3 Block Public Access at the account level unless explicitly incompatible. — **checked one bucket directly**: `musehub-releases` has all 4 settings disabled — likely intentional (serves public `/install.sh`), needs Gabriel's explicit sign-off; account-level setting itself is blocked
- [x] Enable ECR or image vulnerability scanning if containers are used. — **confirmed NOT enabled** — `imageScanStatus` is `null` on the latest image; easy, low-risk fix for the build phase
- [ ] Monitor changes to IAM, KMS, CloudTrail, DNS, and security groups. — N/A until CloudTrail exists
- [ ] Alert on root-account activity. — N/A until CloudTrail/GuardDuty exist
- [ ] Alert on failed console logins and suspicious authentication. — N/A, same reason
- [ ] Define a vulnerability-remediation SLA. — not done; can't be meaningful before scanning exists
- [ ] Run a production security review before launch. — this checklist effort is a step toward it; a dedicated pass still needed closer to launch
- [ ] Record and assign every security finding. — no tracking process exists yet
- [ ] Review high-severity findings before every production launch. — N/A, no findings exist yet

---

## 14. Application Security

> Full detail: [`production-readiness/14-application-security.md`](production-readiness/14-application-security.md)
> ✅ **Fixed**: `/_debug/memory` was exposed unconditionally in production (missing the
> `if settings.debug:` guard `/docs`/`/redoc` correctly have) — patched this session.

- [ ] Document the authentication and authorization model. — MSign is well-implemented in code but not written up as a standalone doc yet
- [ ] Test authorization independently of UI visibility. — not verified in this pass
- [x] Enforce server-side access control. — confirmed via MSign fresh-signature requirement on privileged MCP tool calls
- [x] Use secure password hashing if MuseHub stores passwords. — N/A, no passwords exist (MSign is signature-based)
- [ ] Require email verification where relevant. — N/A, no email-based flows exist
- [x] Implement secure account recovery. — N/A in the traditional sense; equivalent is Ed25519 mnemonic recovery, already documented at the ecosystem level (`muse auth recover`)
- [x] Rate-limit login and recovery endpoints. — confirmed, `slowapi`-based limits + bot throttling (Section 5)
- [ ] Protect against CSRF where applicable. — not explicitly reviewed; likely lower risk given no cookie-based sessions, not confirmed for every state-changing endpoint
- [x] Validate and sanitize untrusted input. — Pydantic models validate at the API boundary throughout
- [x] Use parameterized database queries. — confirmed, no raw SQL string formatting found anywhere
- [x] Restrict CORS to deliberate origins. — confirmed, fails closed (`cors_origins: [] ` default) with a wildcard-in-production warning
- [x] Define a Content Security Policy. — confirmed, set in `musehub/main.py`
- [x] Add standard security headers. — confirmed: `X-Frame-Options`, `X-Content-Type-Options`, HSTS, CSP, Referrer-Policy
- [x] Validate uploaded file type, size, and storage behavior. — confirmed: `mpack_max_bytes`/`mpack_max_commits`/`mpack_max_objects` and per-user/per-repo/daily quotas
- [x] Protect against path traversal and unsafe archive extraction. — confirmed: traversal-char rejection in coordination module, `mpack_max_decompressed_bytes` zip-bomb quarantine
- [ ] Scan uploaded content if the risk model warrants it. — not done beyond structural/size limits
- [x] Set request-body and upload limits. — confirmed, see above
- [ ] Review SSRF exposure from URL-fetching features. — no SSRF-prone code found in main request handling; `musehub/worker.py`'s Cloudflare Worker calls specifically not yet reviewed
- [x] Ensure production debug mode is disabled. — confirmed, `debug: bool = False` default, and `/docs`/`/redoc` correctly gated behind it
- [ ] Ensure stack traces are not returned to users. — not independently verified in this pass
- [ ] Review WebSocket or SSE authentication if used. — MCP/SSE endpoints exist; auth model for the streaming case specifically not independently verified
- [ ] Document data classification and retention. — not done
- [ ] Add an account and data deletion path where applicable. — not verified whether one exists
- [ ] Run an OWASP-oriented review before launch. — this pass covers several categories but isn't a formal pass; deferred to Section 17
- [ ] Review session expiration and token revocation behavior. — N/A in the traditional sense (no sessions); MSign's per-request timestamp/replay-window logic not independently verified
- [x] Verify privileged actions require fresh authorization where appropriate. — confirmed, MCP tool calls require a fresh MSign signature, not just a valid prior session

---

## 15. Cost Controls

> Full detail: [`production-readiness/15-cost-controls.md`](production-readiness/15-cost-controls.md)

- [ ] Apply consistent cost-allocation tags. — **not done**; confirmed inconsistent (prod has only `Name`, staging has `Name`+`Environment`, S3 bucket has none)
- [ ] Create monthly AWS budgets. — **blocked** on broader access grant
- [ ] Alert Gabriel and Aaron at multiple thresholds. — N/A until budgets exist
- [ ] Enable Cost Anomaly Detection. — **blocked**
- [ ] Review NAT Gateway, data-transfer, and logging costs. — N/A for NAT (none exists); logging cost is real and growing (Section 12's unlimited-retention log group)
- [x] Set log-retention limits. — tracked under Section 12, cross-referenced here
- [ ] Set backup-retention limits consistent with recovery requirements. — N/A, no backups exist yet
- [ ] Right-size production after load testing. — not done, no load testing has occurred
- [ ] Review idle resources. — staging appears to run continuously despite `docs/infrastructure.md` suggesting it be stopped when idle — worth Gabriel deciding whether to actually do that
- [x] Document expected baseline monthly cost. — already done, `docs/infrastructure.md` (~$35/mo for both environments)
- [ ] Document the cost implications of autoscaling. — N/A, no autoscaling exists
- [ ] Configure service quotas and limits deliberately. — not done
- [ ] Review production cost weekly during the first month. — deferred to Section 19
- [ ] Add owner and purpose tags to every billable resource. — **not done**, see tagging findings
- [ ] Alert on untagged or unexpectedly expensive resources where practical. — not done

---

## 16. Documentation and Runbooks

> Full detail: [`production-readiness/16-19-docs-testing-cutover-postlaunch.md`](production-readiness/16-19-docs-testing-cutover-postlaunch.md)

- [ ] Create a system architecture diagram. — not created
- [ ] Create a data-flow diagram. — not created
- [x] Document every AWS account and region. — done, Section 0
- [x] Document DNS and certificate ownership. — done, Section 6
- [ ] Document environment differences. — scattered, not consolidated
- [ ] Document the IaC repository structure. — N/A, no IaC exists yet
- [ ] Document local development setup. — not reviewed in this pass
- [x] Document staging deployment. — done, `docs/infrastructure.md`, plus a team-onboarding runbook: [`production-readiness/runbook-deploy-onboarding.md`](production-readiness/runbook-deploy-onboarding.md)
- [x] Document production deployment. — done, `docs/infrastructure.md` (though prod is currently broken)
- [x] Document rollback. — documented, not yet tested
- [ ] Document database migration procedures. — partial; rollback/forward-repair not documented
- [ ] Document database restoration. — **cannot be meaningful yet**, no backups exist
- [ ] Document secret rotation. — not done
- [ ] Document certificate and DNS troubleshooting. — not done
- [ ] Document user-access provisioning and removal. — not done
- [ ] Document break-glass access. — not done
- [ ] Document incident response. — not done
- [ ] Document backup retention and restoration tests. — N/A, no backups exist
- [ ] Document common alarms and remediation. — N/A, no alarms exist
- [ ] Create a service inventory with owners. — partial (inventory yes, owners no)
- [x] Create a dependency inventory. — done, Section 0
- [x] Create an operational decision log. — done, this checklist's Decision Log
- [x] Record accepted risks and deferred work. — done, this checklist's Accepted Risks table
- [x] Keep documentation versioned beside the relevant code where practical. — done by construction
- [ ] Confirm Aaron can follow the documentation without Gabriel's help. — not tested
- [ ] Confirm Gabriel can follow it without relying on personal memory. — not tested
- [ ] Add a production launch checklist and rollback decision tree. — Section 18 is the launch checklist; decision tree specifically doesn't exist
- [ ] Add contact information and escalation paths. — not done
- [ ] Review documentation after every incident or major deployment. — process doesn't exist yet

---

## 17. Testing Before Launch

> Full detail: [`production-readiness/16-19-docs-testing-cutover-postlaunch.md`](production-readiness/16-19-docs-testing-cutover-postlaunch.md)

- [ ] Run the complete automated test suite. — **verified the suite exists and collects cleanly: 7,028 tests, zero collection errors**; actually running it pass/fail is real Section 17 work, not done in this doc-sweep pass
- [ ] Run production-like integration tests. — not verified
- [ ] Test authentication and authorization. — likely covered somewhere in the 7,028 tests; not confirmed
- [ ] Test a clean infrastructure deployment in a disposable environment. — N/A, no IaC exists yet
- [ ] Test migration from the current schema. — not done
- [ ] Test application rollback. — **not done**, explicitly required
- [ ] Test failed migration handling. — not done
- [ ] Test a database restore. — **cannot happen, no backups exist** (Section 8)
- [ ] Test backup integrity. — same blocker
- [ ] Test task or instance failure. — not done
- [ ] Test deployment during active traffic. — not done
- [ ] Test autoscaling if enabled. — N/A, none exists
- [ ] Run basic load and soak tests. — not done
- [ ] Verify timeout behavior. — deliberate timeouts confirmed by code review (Section 5), not load-tested
- [ ] Verify retry behavior. — not load-tested
- [ ] Verify idempotency for jobs and webhooks. — confirmed by code review for background jobs (Section 9); webhooks not reviewed
- [ ] Test DNS and TLS from external networks. — partially done this session (live cert verification); not a full external sweep
- [ ] Test both root and `www`. — both confirmed resolving; not a full test
- [ ] Verify canonical redirects. — **cannot pass**, no canonical redirect exists yet (Section 6)
- [ ] Test monitoring and alert delivery. — N/A, no alerts exist yet
- [ ] Run a tabletop incident exercise with Gabriel and Aaron. — not done
- [ ] Test the break-glass process. — N/A, doesn't exist yet
- [ ] Verify neither maintainer is a single point of operational failure. — **currently fails**, per Section 1 (shared credential, no Aaron AWS identity)
- [ ] Test production access from a clean workstation. — not done
- [ ] Test what happens when a secret, database, queue, or external API is unavailable. — not done

---

## 18. Cutover Plan

> Full detail: [`production-readiness/16-19-docs-testing-cutover-postlaunch.md`](production-readiness/16-19-docs-testing-cutover-postlaunch.md)
> Entirely future work — every item here depends on Sections 0–15 being substantially complete
> (backups existing, RPO/RTO defined, monitoring live, Aaron having independent access). Not
> walking each item individually since none are meaningfully startable yet. One prerequisite worth
> repeating: **prod deploy is currently broken** (no IAM instance profile, Section 0) — nothing in
> this section can happen until that's fixed, independent of everything else.

- [ ] Freeze unrelated infrastructure changes during cutover.
- [ ] Confirm the production environment is healthy.
- [ ] Confirm production secrets are installed.
- [ ] Confirm migrations have been rehearsed.
- [ ] Confirm current backups succeeded.
- [ ] Confirm a restoration has been tested.
- [ ] Confirm dashboards and alerts are active.
- [ ] Confirm Aaron and Gabriel are both available for the cutover.
- [ ] Lower DNS TTL in advance.
- [ ] Deploy the approved production artifact.
- [ ] Run migrations.
- [ ] Run internal smoke tests.
- [ ] Update DNS.
- [ ] Verify:
  - [ ] DNS resolution
  - [ ] TLS
  - [ ] Root domain
  - [ ] `www`
  - [ ] API
  - [ ] Authentication
  - [ ] Critical user paths
  - [ ] Background workers
  - [ ] Outbound email
  - [ ] Object uploads and downloads
- [ ] Monitor logs, error rate, and latency closely.
- [ ] Keep the previous environment available for rollback during the agreed window.
- [ ] Execute rollback if predetermined thresholds are crossed.
- [ ] Raise DNS TTL after stability is confirmed.
- [ ] Record the launch artifact and configuration versions.
- [ ] Announce launch status to stakeholders.
- [ ] Record all manual actions taken during cutover.
- [ ] Confirm production backups run successfully after cutover.

---

## 19. Post-Launch

> Full detail: [`production-readiness/16-19-docs-testing-cutover-postlaunch.md`](production-readiness/16-19-docs-testing-cutover-postlaunch.md)
> Entirely contingent on launch having happened — not walking each item individually. The
> recurring-schedule items are worth setting up as real calendar reminders once launch is real;
> several (e.g. quarterly DR exercise) are only meaningful once their underlying capability exists
> (backups + restore procedure, for that example).

- [ ] Review production metrics after the first hour.
- [ ] Review again after 24 hours.
- [ ] Review again after one week.
- [ ] Inspect AWS costs after 24 hours and one week.
- [ ] Review security findings.
- [ ] Confirm backups continue to succeed.
- [ ] Perform another restore test after production has real data.
- [ ] Remove temporary cutover access and resources.
- [ ] Remove obsolete staging-derived credentials.
- [ ] Resolve or formally record every launch exception.
- [ ] Conduct a blameless launch retrospective.
- [ ] Update documentation with lessons learned.
- [ ] Schedule:
  - [ ] Monthly dependency updates
  - [ ] Monthly access review during the early production period
  - [ ] Quarterly disaster-recovery exercise
  - [ ] Quarterly permissions review
  - [ ] Quarterly cost review
  - [ ] Annual domain and recovery-contact review

---

# Recommended Implementation Order

## P0 — Before Creating Production

- [ ] Inventory staging.
- [ ] Establish shared organizational ownership and MFA.
- [ ] Decide AWS account boundaries.
- [ ] Select and implement Infrastructure as Code.
- [ ] Define RPO, RTO, and domain topology.
- [ ] Decide the canonical hostname and DNS provider.

## P1 — Build Production

- [ ] Provision the production account and environment through IaC.
- [ ] Configure identity, networking, encryption, and secrets.
- [ ] Configure the database, automated backups, and restore testing.
- [ ] Configure GitHub OIDC and separate deployment roles.
- [ ] Build staging and production pipelines.
- [ ] Add logs, dashboards, alarms, and audit trails.
- [ ] Complete operational documentation.

## P2 — Launch

- [ ] Configure certificates and DNS.
- [ ] Rehearse migrations, deployment, and rollback.
- [ ] Run security and load tests.
- [ ] Complete the cutover.
- [ ] Validate production and document the outcome.

---

# Suggested Ownership Split

| Workstream | Primary | Reviewer / Backup | Status |
|---|---|---|---|
| Staging architecture inventory | Gabriel | Claude Code | Done — Section 0 |
| AWS Organizations and accounts | Gabriel |  | Decided (deferred as tech debt) — Section 2 |
| IAM Identity Center and permissions | Gabriel |  | Documented gap, not built — Section 3 |
| Infrastructure as Code | Gabriel |  | Tool recommended, not built — Section 4 |
| Networking and production architecture | Gabriel |  | Documented, SPOFs identified — Section 5 |
| Namecheap, DNS, and TLS | Gabriel |  | Documented, Cloudflare-hosting correction made — Section 6 |
| Database and backups | Gabriel |  | ⚠️ Documented — **no usable backup exists, highest-priority risk** — Section 8 |
| MuseHub-native CI/CD | Gabriel |  | Documented — platform features (branch protection, required checks) don't exist yet — Sections 9-11 |
| Logging, metrics, and alerts | Gabriel |  | Logging is live and solid; alerts/dashboards don't exist — Section 12 |
| Security review | Gabriel |  | Partial — one live finding fixed-pending (`/_debug/memory`); account-wide checks blocked on access grant — Sections 13-14 |
| Documentation and runbooks | Gabriel | Claude Code | This sweep is the start of it — Section 16 |
| Cutover coordination |  |  | Correctly blocked until earlier sections close — Sections 18-19 |

---

# Decision Log

| Date | Decision | Rationale | Owner |
|---|---|---|---|
| 2026-08-04 | Accept single AWS account (`992382692655`, shared with Stori) as technical debt rather than splitting into AWS Organizations before launch | Keeps focus on getting production launched; account split can happen later without blocking | Gabriel |
| 2026-08-04 | Architecture target (harden current EC2/Docker-Compose vs. redesign to ECS+ALB+RDS) left open | Insufficient information at Section 0 time; revisit once inventory and RPO/RTO are settled | Gabriel |
| 2026-08-04 | IaC tool selection deferred; Terraform/OpenTofu recommended given the actual footprint (2 EC2, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params) | Small, EC2-centric footprint doesn't need CDK's language ceremony or CloudFormation's verbosity | Gabriel |
| 2026-08-04 | Every "GitHub" checklist item is translated to its MuseHub equivalent (proposals for PRs, MuseHub branch protection, `musehub-runner` CI, MuseHub collaborator roles) | Absolute ecosystem policy — git/GitHub are never the VCS, CI, or collaboration platform for this project; GitHub is a passive backup mirror only | Gabriel (policy), applied throughout by Claude Code |
| 2026-08-08 | Section 6's "Namecheap vs. Route 53" question reframed as "Cloudflare vs. Route 53" | Live `dig NS musehub.ai` confirms DNS is already hosted at Cloudflare (registrar-only at Namecheap); `docs/infrastructure.md` was stale on this. Recommend staying on Cloudflare DNS given it also does edge proxy/TLS/possibly R2 | Claude Code (finding), decision pending Gabriel |
| 2026-08-08 | Deploy team expanded from 2 (Gabriel, Aaron) to 5 (+ interns Sol, Toby, Ben); interns get staging-only access; Section 3's IAM Identity Center rollout is scoped for all 5 from the start | Sharing one AWS credential 5 ways has no accountability and no environment separation; the current key should not be distributed to interns before individual identities exist | Gabriel |

---

# Accepted Risks and Deferred Work

> ⚠️ **Highest-priority item in this table**: there is currently no usable database backup,
> verified live on staging (no cron job, no rclone, one stale 20-byte manual dump). If the
> production database is lost today, there is no recovery path. This should not wait for the rest
> of the checklist sweep to be addressed.

| Item | Risk | Mitigation | Target Date | Owner |
|---|---|---|---|---|
| No usable database backup exists (verified live: no cron, no rclone, one stale 20-byte manual dump from ~2 months ago) | Total, unrecoverable data loss on any DB failure, corruption, or operator error | Wire `deploy/backup.sh` in for real: install rclone, configure R2 remote, install the cron job, verify a real non-empty backup lands, then perform an actual restore test | ASAP — before any other Section 8 work | Gabriel |
| Single Postgres role used for app runtime, migrations, and admin access | An app-level SQL injection or bug would have migration/admin-level DB privileges, not just runtime privileges | Split into separate roles (app runtime, migrations, read-only, admin) per the checklist's explicit ask | Before Section 8 sign-off | Gabriel |
| Migration failure-handling (`alembic stamp --purge head` on upgrade failure) could mask a genuinely broken migration as a stale-revision-ID issue | A broken migration could silently ship if the follow-up schema-parity gate ever has a false negative | Document the reliance on `schema_gate` explicitly; consider distinguishing failure causes rather than blanket stamp-and-retry | Before Section 8 sign-off | Gabriel |
| MuseHub-native branch protection, required status checks, and CODEOWNERS-equivalent don't exist as product features (Sections 9-11 assumed they just needed configuring, like GitHub Actions would) | Sections 9-11 require building MuseHub platform capability, not just deployment config — bigger scope than the checklist implies | Scope this explicitly as product work before estimating Sections 9-11's timeline; prioritize required-status-check gating on `merge_proposal` first since everything else depends on it | Before Section 9 build phase begins | Gabriel |
| ~~`/_debug/memory` endpoint exposed unconditionally in production~~ | ~~Minor info disclosure~~ | **Fixed this session** — guarded behind `if settings.debug:` like `/docs`/`/redoc`; ships on next deploy | Done 2026-08-08 | Claude Code |
| Single AWS account shared with Stori | Blast radius: a MuseHub compromise/misconfig can affect Stori resources and vice versa; billing and ownership entangled | Revisit AWS Organizations split post-launch; `musehub-infra` IAM user already scoped narrowly (verified via access-denied tests on IAM/CloudTrail/GuardDuty/Route53/etc.) | TBD | Gabriel |
| `docs/infrastructure.md` is stale — object storage is actually Cloudflare R2, not a local Docker volume as documented | Anyone reading the doc (including Aaron) gets the wrong mental model of where pushed content lives | Correct `docs/infrastructure.md` directly (separate from this checklist effort); resolved here via live proof this session | Soon | Gabriel |
| Secrets are hand-provisioned per instance, not via the drafted SSM pipeline (`deploy/secrets.sh`) — no rotation, no audit trail beyond manual memory | Rotation and auditability (Section 7's actual goals) aren't met by the current hand-provisioned `.env` approach | Gabriel to decide whether production adopts the SSM pipeline from day one rather than repeating staging's hand-provisioning | Before Section 7 sign-off | Gabriel |
| No RPO/RTO, traffic, or availability targets defined | Can't size backup cadence, DR posture, or capacity correctly without them | Define before Section 5 (architecture) and Section 8 (backups) are finalized | Before P1 build phase | Gabriel |
| Canonical hostname (root vs. `www`) and DNS host (Namecheap vs. Route 53) undecided | Blocks finalizing Section 6 and the cutover plan | Decide before Section 18 cutover | Before cutover | Gabriel |
| Deploys use a single shared AWS credential (`musehub-infra` in `~/.aws/credentials`), not individual identities | Violates "no shared AWS access keys"; no accountability for who ran what; single point of failure if Gabriel is unavailable | Resolve via Section 3 (IAM Identity Center + per-person roles) before Section 1's independent-operation test can pass | Before Section 1 sign-off | Gabriel |
| Aaron has no AWS identity or verified Namecheap access today | Aaron cannot operate independently — fails the checklist's explicit "test that Aaron can independently…" requirement | Create Aaron's AWS identity via Section 3; confirm/grant Namecheap access | Before Section 1 sign-off | Gabriel |