# MuseHub Production-Readiness Checklist

> Master checklist for promoting MuseHub from staging to production, automating deployments, sharing operational ownership between Gabriel and Aaron, and applying security, reliability, backup, documentation, and infrastructure best practices.

> **Working method:** sections are swept in order, once each. Unresolved items are flagged inline
> and rolled up into the Decision Log / Accepted Risks tables at the bottom rather than blocking
> progress on later sections. See `docs/production-readiness/` for the per-section companion docs
> with full verification detail.

## 0. Decisions and Inventory

> Full detail: [`production-readiness/00-inventory-and-decisions.md`](production-readiness/00-inventory-and-decisions.md)

- [ ] Document the current staging architecture.
  - [x] AWS account and region — `992382692655`, `us-east-1` (verified live)
  - [x] Compute platform: ECS, EC2, App Runner, Elastic Beanstalk, Lambda, EKS, etc. — raw EC2 (`t3.small`/`t3.medium`), Docker Compose, no managed compute service
  - [x] Load balancer, API Gateway, CloudFront, or other ingress — none; Cloudflare proxy → nginx on the instance
  - [x] Database engine and topology — self-hosted PostgreSQL 16, single instance, no RDS, no replica
  - [x] Redis, queues, object storage, search, background workers — background worker (`musehub-runner`) confirmed; object storage confirmed **live on Cloudflare R2** (bucket `musehub-staging`, proven by a successful push this session — `docs/infrastructure.md` corrected); no Redis/search identified
  - [x] VPC, subnets, route tables, NAT gateways, and security groups — default VPC, no custom subnets, no NAT gateway, `musehub-sg` verified live (443 from Cloudflare ranges only)
  - [x] Secrets and configuration sources — hand-provisioned `.env` per instance (not SSM-automated); confirmed it includes working R2 credentials `docs/infrastructure.md` didn't document. Open question for Gabriel: should production adopt the drafted SSM pipeline (`deploy/secrets.sh`) from day one instead of hand-provisioning like staging?
  - [x] Logging and monitoring — confirmed none live (no CloudWatch alarms, no SNS topics exist despite a script that would create them)
  - [x] Current deployment process — documented: manual, human-triggered `deploy/push.sh`, no CI/CD automation
- [x] Inventory every manually created AWS resource. — 2 EC2 + 2 EIP, 1 SG, 1 IAM user, 1 IAM instance role, 1 ECR repo, 1 S3 bucket, 3 SSM params (all listed in companion doc)
- [ ] Inventory every external dependency.
  - [x] GitHub — backup-mirror only, never VCS/CI/collaboration (ecosystem policy)
  - [x] Namecheap — registrar + DNS host for `musehub.ai`
  - [x] Email provider — confirmed none exists
  - [x] OAuth providers — confirmed none exist (MSign/Ed25519 only)
  - [x] Payment or billing providers — confirmed none exist
  - [x] Error tracking — confirmed none exists
  - [ ] Third-party APIs — Cloudflare R2/Worker usage inferred from scripts but not independently verified against the Cloudflare dashboard
- [x] Identify all stateful and mission-critical data. — Postgres DB, object store, `musehub-releases` S3 bucket, SSM params (all listed in companion doc)
- [ ] Define expected initial traffic and reasonable growth assumptions. — **not yet defined, needs Gabriel**
- [ ] Define availability expectations. — **not yet defined, needs Gabriel**
- [ ] Define acceptable downtime during deployment. — **not yet defined, needs Gabriel**
- [ ] Define recovery objectives.
  - [ ] **RPO:** maximum acceptable data loss — **not yet defined, needs Gabriel**
  - [ ] **RTO:** maximum acceptable restoration time — **not yet defined, needs Gabriel**
- [ ] Decide the canonical public URLs.
  - [x] Root (`musehub.ai`) — exists, live
  - [x] `www.musehub.ai` — exists, live
  - [ ] API hostname — none exists; MuseHub serves API + web from the same origin, no `api.` subdomain in use today
  - [x] Staging (`staging.musehub.ai`) — exists, live
- [ ] Choose either the root domain or `www` as canonical. — **not yet decided, needs Gabriel**
- [ ] Redirect the noncanonical hostname permanently to the canonical one. — no redirect logic found in nginx config; not implemented
- [ ] Decide whether DNS remains hosted at Namecheap or moves to Route 53. — **not yet decided, needs Gabriel**
- [x] Decide whether production and non-production will use separate AWS accounts. — **Decided:** single account accepted as technical debt for now (see Decision Log)

---

## 1. Ownership and Shared Access

> No shared human accounts, shared passwords, or shared AWS access keys.
> Full detail: [`production-readiness/01-ownership-and-access.md`](production-readiness/01-ownership-and-access.md).
> "GitHub organization" items below are the MuseHub-native equivalent per the Decision Log.

- [ ] Confirm the domain is held by the company rather than an individual wherever practical. — **needs Gabriel (Namecheap console)**
- [ ] Confirm AWS account ownership and billing contacts. — **needs Gabriel (AWS console)**
- [x] Confirm MuseHub repo ownership/collaboration model. — `gabriel` owns both repos; `aaronrene` has `write` collaborator access on `musehub` and `muse` (verified via `muse hub collaborator list`)
- [ ] Add both Gabriel and Aaron as appropriate organizational owners or administrators. — done for MuseHub repos (Aaron has `write`); **not done for AWS/Namecheap — needs Gabriel**
- [ ] Ensure neither person is the sole recovery path for:
  - [ ] Namecheap — **needs Gabriel**
  - [ ] AWS — **needs Gabriel**
  - [x] MuseHub — Aaron already has `write` access independent of Gabriel
  - [ ] DNS — **needs Gabriel**
  - [ ] Database — **needs Gabriel** (currently only reachable via `musehub-infra` shared AWS credential)
  - [ ] Production secrets — **needs Gabriel**
  - [ ] Billing — **needs Gabriel**
  - [ ] Backups — **needs Gabriel** (also blocked on resolving Section 0 Contradiction 2)
- [ ] Use company-controlled email addresses for infrastructure accounts. — **needs Gabriel**
- [ ] Configure at least two trusted recovery paths. — **needs Gabriel**
- [ ] Store emergency recovery documentation in a location accessible to both founders. — **needs Gabriel**
- [ ] Create a documented joiner–mover–leaver access procedure. — not started
- [ ] Create a quarterly access-review process. — not started
- [ ] Record ownership for every operational area:
  - [ ] AWS administration — currently Gabriel only, undocumented
  - [ ] DNS and certificates — currently Gabriel only, undocumented
  - [ ] Deployments — currently Gabriel only (shared `musehub-infra` credential), undocumented
  - [ ] Database administration — currently Gabriel only, undocumented
  - [ ] Incident response — no process exists
  - [ ] Billing — currently Gabriel only, undocumented
  - [ ] Backups — no automated backups exist yet (Section 0 Contradiction 2)
- [ ] Create an emergency break-glass procedure. — not started
- [ ] Test that Aaron can independently:
  - [ ] Sign in to AWS — **cannot test — no AWS identity exists for Aaron yet**
  - [ ] Inspect production — **cannot test**
  - [ ] Deploy staging — **cannot test as designed** — today's deploy uses a shared AWS credential (`musehub-infra`), not an individual one; this violates the "no shared AWS access keys" principle this section opens with
  - [ ] Approve or deploy production — **cannot test** — prod deploy is currently broken regardless (Section 0: no IAM instance profile on the prod instance)
  - [ ] Inspect logs and alarms — **cannot test — no logs/alarms exist yet** (Section 12)
  - [ ] Restore from backup — **cannot test — no backups exist yet** (Section 0 Contradiction 2)
  - [ ] Update DNS when authorized — **cannot test — Aaron's Namecheap access unverified**
- [ ] Test that Gabriel can perform the same tasks without relying on undocumented knowledge. — not started; current process is entirely undocumented outside `docs/infrastructure.md`

---

## 2. AWS Account Structure

> Full detail: [`production-readiness/02-aws-account-structure.md`](production-readiness/02-aws-account-structure.md)

### Recommended Target

- [ ] Create or verify an AWS Organization. — **deferred, see Decision Log** (single account accepted as technical debt)
- [ ] Use distinct AWS accounts for:
  - [ ] Management — deferred
  - [ ] Production — deferred
  - [ ] Non-production — deferred
- [ ] Consider separate security and log-archive accounts as the operation grows. — deferred
- [ ] Place production and non-production into separate organizational units. — deferred (no Organization exists)
- [ ] Apply Service Control Policies only after testing them carefully. — N/A while deferred
- [ ] Centralize billing. — N/A, single account
- [ ] Enable account-level budget alerts. — not done; tracked under Section 15 (Cost Controls) instead since it applies within the single account regardless of structure — **needs Gabriel (AWS console access required)**
- [ ] Set alternate security, operations, and billing contacts on every account. — **needs Gabriel**
- [ ] Restrict enabled AWS regions to the regions MuseHub actually uses, where feasible. — **needs Gabriel**; note the account is shared with Stori (region `us-east-2` bucket observed), so any region restriction must account for both projects
- [x] Document any temporary decision to keep staging and production in one AWS account as accepted technical debt. — done, see Decision Log and Accepted Risks in this checklist

---

## 3. Identity, Authentication, and Permissions

> Full detail: [`production-readiness/03-identity-and-permissions.md`](production-readiness/03-identity-and-permissions.md)
> Document-only pass — no IAM changes made. Real build happens after the full sweep.

### Root and Recovery Access

- [ ] Protect every AWS root account with phishing-resistant MFA where possible. — **needs Gabriel (AWS console)**
- [ ] Do not create root access keys. — **needs Gabriel to confirm**
- [ ] Do not use root for routine administration. — **needs Gabriel to confirm**
- [ ] Secure the root email account with MFA. — **needs Gabriel**
- [ ] Store root recovery information securely. — **needs Gabriel**
- [ ] Document the legitimate uses of root. — not started
- [ ] Configure multi-person approval for emergency root use if practical. — not started

### Human Access

- [ ] Configure AWS IAM Identity Center. — not started; visibility blocked (`sso-admin` denied) — **needs Gabriel to check/build**
- [ ] Create separate identities for Gabriel and Aaron. — not started (blocks Section 1's independent-operation test)
- [ ] Require MFA for both. — not started
- [ ] Prefer hardware security keys or passkeys. — not started
- [ ] Define permission sets, for example:
  - [ ] `Administrator` — not started
  - [ ] `ProductionOperator` — not started
  - [ ] `Developer` — not started
  - [ ] `ReadOnly` — not started
  - [ ] `Billing` — not started
- [ ] Give routine access through the least-privileged applicable role. — not started
- [ ] Require explicit elevation for administrative work. — not started
- [ ] Avoid permanent `AdministratorAccess` for everyday use where practical. — not started
- [ ] Use IAM Access Analyzer to inspect and refine permissions. — not started; needs broader read access
- [ ] Review unused roles, users, policies, and credentials. — not started; needs broader read access
- [ ] Document who can approve and perform production changes. — not started

### Workload Access

- [x] Give every application or service its own IAM role. — 3 separate IAM users confirmed live (`musehub-infra`, `musehub-app`, `stori-assets-app`), no shared workload credential across services
- [x] Do not place AWS access keys in source code. — confirmed via grep, none found
- [x] Do not place long-lived AWS access keys in GitHub secrets. — N/A, no GitHub Actions/secrets used (see Decision Log); equivalent check for MuseHub-runner CI config deferred to Section 9
- [ ] Restrict every deployment role to:
  - [ ] The correct repository — N/A, no deployment role exists yet (deploys use a human + shared credential)
  - [ ] The correct branch, tag, or GitHub environment — N/A, same reason
  - [ ] The correct AWS account — N/A, same reason
  - [ ] The required AWS resources only — partially true of `musehub-infra`'s scoping, but it's a human-shared credential, not a deployment role
- [ ] Separate staging and production deployment roles. — **not done** — both environments share the `musehub-infra` credential
- [ ] Validate trust policies and permission boundaries. — not started; needs broader read access
- [ ] Prefer temporary credentials everywhere. — **not done** — all current credentials are long-lived IAM user access keys

---

## 4. Infrastructure as Code

> Full detail: [`production-readiness/04-iac.md`](production-readiness/04-iac.md)
> Document-only pass. No IaC written yet — deferred to the build phase after the full sweep.

- [ ] Select one Infrastructure-as-Code system:
  - [ ] Terraform/OpenTofu — **recommended**, not yet formally chosen by Gabriel
  - [ ] AWS CDK
  - [ ] CloudFormation
- [ ] Capture the existing staging infrastructure in code. — not started; full resource list is in the [Section 0 doc](production-readiness/00-inventory-and-decisions.md)
- [ ] Stop making undocumented console-only changes. — behavioral change, not yet in effect
- [ ] Build reusable modules rather than duplicating entire configurations. — N/A until IaC exists
- [ ] Provide separate environment configuration for:
  - [ ] Staging — N/A until IaC exists
  - [ ] Production — N/A until IaC exists
- [ ] Keep production and staging state isolated. — N/A until IaC exists
- [ ] Encrypt remote IaC state. — N/A until IaC exists
- [ ] Enable versioning and locking for remote state. — N/A until IaC exists
- [ ] Restrict access to IaC state because it may contain sensitive metadata. — N/A until IaC exists
- [ ] Pin provider, module, and action versions. — N/A until IaC exists
- [ ] Add formatting and validation checks. — N/A until IaC exists
- [ ] Add static security scanning. — N/A until IaC exists
- [ ] Require a reviewed plan before production changes. — N/A until IaC exists
- [ ] Detect configuration drift. — N/A until IaC exists
- [ ] Document how to import existing resources into IaC. — not started; note this must be *import*, not recreate, since real resources already exist and are live
- [ ] Add environment-standard tags:
  - [ ] Application — not applied anywhere currently
  - [ ] Environment — not applied anywhere currently
  - [ ] Owner — not applied anywhere currently
  - [ ] Managed-by — not applied anywhere currently
  - [ ] Cost-center — not applied anywhere currently
  - [ ] Data-classification — not applied anywhere currently
- [ ] Make a fresh environment reproducible from source control and documented prerequisites. — not started
- [ ] Add a teardown guard so production cannot be accidentally destroyed. — N/A until IaC exists
- [ ] Add deletion protection to critical resources. — not started (also relevant to Section 8 — DB deletion protection specifically)
- [ ] Add policy checks that prevent production resources from being publicly exposed unintentionally. — N/A until IaC exists
- [ ] Ensure IaC changes are reviewed by someone other than the author when practical. — N/A until IaC exists

---

## 5. Production Architecture

> Full detail: [`production-readiness/05-production-architecture.md`](production-readiness/05-production-architecture.md)

- [ ] Reassess whether staging's sizing and topology are production-appropriate. — flagged: staging is `t3.medium`, prod is `t3.small` — inconsistent, no documented reason
- [x] Do not copy staging data into production unless explicitly sanitized and required. — not done, and correctly not done — no evidence of staging→prod data copying anywhere
- [ ] Use separate production:
  - [x] Database — separate instance entirely (different EC2 host)
  - [ ] Credentials — unresolved pending Section 0 Contradiction 2
  - [ ] Secrets — unresolved pending Section 0 Contradiction 2
  - [ ] Object-storage buckets — unresolved pending Section 0 Contradiction 1
  - [ ] Queues — none exist in either environment
  - [ ] Cache — none exist in either environment
  - [ ] Logs — not separated in any meaningful way; no centralized logging exists yet (Section 12)
  - [ ] Encryption keys where appropriate — not reviewed yet
- [ ] Deploy production across at least two Availability Zones where supported and justified. — **not done**; single instance, no ASG, documented as intentional SPOF below
- [x] Keep databases and internal services in private subnets. — partially true in spirit (DB only reachable via the app's Docker network, not the public internet) but not true architecturally (default VPC, no real private subnet); security group is the actual control here
- [x] Expose only the public ingress layer. — confirmed: only 443 open, restricted to Cloudflare IP ranges
- [x] Eliminate direct public database access. — confirmed, DB has no public exposure
- [x] Use security groups by service relationship rather than broad CIDR ranges. — reasonable substitution given the architecture: scoped to Cloudflare's published IP ranges, not `0.0.0.0/0`
- [x] Prohibit unrestricted administrative ports such as public SSH. — confirmed, no port 22 open
- [x] Use AWS Systems Manager Session Manager rather than public SSH where applicable. — confirmed, this is already the only access path
- [x] Configure health checks. — confirmed: `/healthz` probes both DB and object storage, gates blue/green flips
- [ ] Configure graceful application shutdown. — **not verified** — no SIGTERM handling found in `musehub/main.py`; uvicorn defaults apply
- [ ] Configure connection draining. — partial: blue/green waits for new slot health before flip, but old slot's in-flight requests during `docker rm` aren't verified graceful
- [x] Set sensible application and load-balancer timeouts. — confirmed: deliberate per-route nginx timeouts (5s health, 60s default, up to 3600s for streaming/MCP)
- [ ] Configure autoscaling or document why fixed capacity is appropriate. — neither done; fixed capacity with no written justification
- [ ] Set resource requests, limits, or instance sizes deliberately. — instance types chosen but not justified with load data
- [ ] Verify production can survive a routine instance or task replacement. — **not verified**; single-instance architecture means instance failure has no automatic failover (documented SPOF below)
- [ ] Confirm background jobs are idempotent or safely retryable. — not verified in this pass (`musehub-runner` job-claiming logic not reviewed)
- [ ] Confirm database migrations are compatible with rolling deployments. — pipeline runs migrations before slot flip (right shape); expand/contract discipline not reviewed — deferred to Section 8
- [x] Add rate limiting and abuse controls. — confirmed: `slowapi`-based per-route limits + global baseline + bot-throttle middleware, already implemented
- [ ] Consider AWS WAF for public endpoints. — N/A as literally stated (no ALB/CloudFront); Cloudflare is the substitute layer, but its WAF/rate-limit config is unverified — **needs Gabriel to check Cloudflare dashboard**
- [x] Configure DDoS protections appropriate to the architecture. — Cloudflare edge proxy already provides this; not independently verified beyond "Cloudflare is in front of everything"
- [ ] Establish AWS service quota monitoring for relevant resources. — not started
- [x] Document every intentional single point of failure. — done, see companion doc: single EC2 per environment, single Postgres instance, single AWS account (shared with Stori), single Cloudflare account, single human deployer with shared credential
- [ ] Create a plan to remove unacceptable single points of failure. — **needs Gabriel** — depends on RPO/RTO/availability targets that are still undefined (Section 0)

---

## 6. DNS, Domain, and TLS

> Full detail: [`production-readiness/06-dns-domain-tls.md`](production-readiness/06-dns-domain-tls.md)
> ⚠️ **Finding:** DNS is actually hosted at **Cloudflare**, not Namecheap — verified via live
> `dig NS musehub.ai`. `docs/infrastructure.md` is stale on this point. Namecheap is registrar
> only. This changes several items below from "Namecheap vs Route 53" to "Cloudflare vs Route 53."

### Namecheap Account

- [ ] Enable MFA on Namecheap for every authorized administrator. — **needs Gabriel**
- [ ] Use unique individual access rather than shared credentials where the account model permits. — **needs Gabriel**
- [ ] Enable domain lock. — **needs Gabriel**
- [ ] Verify registrant and recovery contact details. — **needs Gabriel**
- [ ] Enable automatic renewal. — **needs Gabriel**
- [ ] Verify the payment method will not unexpectedly expire. — **needs Gabriel**
- [ ] Document who may change nameservers or DNS. — **needs Gabriel**; note DNS itself is now on Cloudflare, so this also means documenting Cloudflare access, not just Namecheap
- [ ] Export or document all existing DNS records before making changes. — **needs Gabriel, from the Cloudflare dashboard**, not Namecheap's Advanced DNS tab
- [ ] Preserve mail-related MX, SPF, DKIM, and DMARC records. — **needs Gabriel to confirm these exist and are preserved in Cloudflare's DNS**
- [ ] Confirm both Gabriel and Aaron have an approved recovery path. — **needs Gabriel**, for both Namecheap and Cloudflare

### DNS Design

- [ ] Decide between:
  - [x] Namecheap-hosted DNS — **not the current state**; superseded by the finding above
  - [ ] Route 53-hosted DNS with Namecheap remaining registrar — open; recommendation below
- [x] ~~Prefer Route 53...~~ — **recommendation revised**: current DNS host is Cloudflare (not Namecheap), and Cloudflare is also doing edge proxy/TLS/possibly R2 — recommend staying on Cloudflare DNS rather than fragmenting the stack by moving to Route 53, unless Gabriel has a specific reason to want AWS-native alias records
- [ ] Create a Route 53 hosted zone if selected. — N/A per above recommendation, pending Gabriel's decision
- [x] Reproduce every required existing record before switching nameservers. — N/A, no switch planned per recommendation
- [ ] Lower DNS TTL ahead of cutover. — deferred to Section 18 (Cutover Plan)
- [x] Configure the root/apex record. — exists, live, resolves via Cloudflare
- [x] Configure `www`. — exists, live
- [ ] Configure `api` if needed. — no `api.` hostname exists; MuseHub serves API + web from one origin — confirm this is intentional
- [x] Keep staging on a clearly separate hostname. — confirmed, `staging.musehub.ai`
- [x] Do not expose internal AWS service hostnames unnecessarily. — confirmed, only the public hostnames resolve; no raw AWS DNS names exposed
- [ ] Verify IPv4 and IPv6 behavior. — not tested in this pass
- [ ] Configure a canonical-host redirect. — **not implemented** — no redirect logic found; root and `www` both resolve without a canonical redirect
- [ ] Test DNS from multiple resolvers after cutover. — deferred to Section 18
- [ ] Raise TTL again after the deployment is stable. — deferred to Section 18
- [ ] Manage DNS records through IaC where practical. — deferred to Section 4 build phase (Terraform's `cloudflare` provider)

### TLS

- [ ] Request an AWS Certificate Manager certificate. — **likely N/A for this architecture** — TLS terminates at Cloudflare's edge, not an AWS ALB/CloudFront; needs Gabriel to confirm there's no AWS-facing TLS need I'm missing
- [ ] Include every required hostname. — N/A pending above
- [ ] Prefer DNS validation. — N/A pending above
- [ ] Retain ACM validation records for automatic renewal. — N/A pending above
- [x] Redirect all HTTP traffic to HTTPS. — Cloudflare/nginx setup already enforces HTTPS-only (port 80 not open at all)
- [x] Use modern TLS policies. — Cloudflare-managed edge TLS; verified live cert is current (Let's Encrypt, issued 2026-08-01)
- [x] Configure HSTS only after HTTPS and subdomain behavior are verified. — already live: `Strict-Transport-Security` with `includeSubDomains` confirmed in nginx config
- [ ] Confirm certificate-expiration monitoring. — **not done** — no alerting exists yet (Section 12); manual verification only (as done earlier this session)
- [x] Test the full certificate chain. — verified via `openssl s_client` this session; valid, correctly issued, correct CN
- [ ] Confirm cookies use:
  - [x] N/A — MuseHub uses header-based MSign auth, no cookies exist anywhere in the app

---

## 7. Secrets and Application Configuration

> Full detail: [`production-readiness/07-secrets-and-config.md`](production-readiness/07-secrets-and-config.md)

- [x] Inventory all secrets. — done, 6 secrets identified (DB password, webhook key, runner token, R2 credentials, worker key, worker URL)
- [x] Remove secrets from source control and repository history. — confirmed via grep, none found
- [ ] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate. — **the real work item**: `deploy/secrets.sh` implements this but only 3/7 required SSM params exist; not live yet
- [ ] Encrypt secrets with KMS. — partial; the 3 existing SSM params are SecureString/KMS-encrypted, rest aren't in SSM
- [x] Use separate staging and production secrets. — confirmed: separate `.env`, separate R2 buckets, separate SSM prefixes
- [ ] Give workloads access only to the secrets they require. — likely N/A for a single-process-per-instance app; not force-fit
- [x] Do not expose production secrets to pull-request workflows. — N/A, no PR-based CI exists
- [ ] Do not print secrets in logs. — not verified; no centralized logs exist yet to audit (Section 12)
- [ ] Configure secret rotation where supported. — not done
- [ ] Rotate credentials during the production launch. — deferred to Section 18
- [ ] Document emergency rotation procedures. — not done
- [x] Separate secrets from nonsecret configuration. — confirmed: `musehub/config.py`'s `Settings` class already separates these cleanly in code
- [x] Validate required configuration at application startup. — confirmed via `pydantic_settings.BaseSettings`
- [x] Fail safely when critical configuration is missing. — confirmed, same mechanism plus `secrets.sh`'s own preflight `die` checks
- [x] Remove default credentials and sample keys. — confirmed, no placeholder secrets in `config.py`; `secrets.sh` also actively rejects known-weak passwords
- [ ] Audit OAuth callback URLs and API allowlists for production. — N/A, no OAuth exists
- [ ] Audit every external webhook secret. — `WORKER_INTERNAL_KEY` identified; validation logic in `musehub/worker.py` not independently reviewed in this pass
- [ ] Document ownership and rotation frequency for every production secret. — not done, blocked on Section 1

---

## 8. Database Security, Migrations, and Backups

> Full detail: [`production-readiness/08-database-backups-migrations.md`](production-readiness/08-database-backups-migrations.md)
> ⚠️ **Verified live via SSM: there is currently no usable backup of the database, anywhere.**

### Database Security

- [ ] Enable encryption at rest. — not verified in this pass
- [ ] Enforce encryption in transit. — not verified in this pass
- [ ] Use a production-specific KMS key where appropriate. — not started
- [ ] Store credentials in Secrets Manager. — see Section 7
- [ ] Avoid administrator credentials for normal application traffic. — **not done** — single `musehub` role used for app traffic, migrations, and admin access
- [ ] Create separate roles for:
  - [ ] Application runtime — not done
  - [ ] Migrations — not done
  - [ ] Read-only operations — not done
  - [ ] Administration — not done
- [x] Keep the database off the public internet. — confirmed
- [x] Restrict security-group access to the application layer. — confirmed
- [ ] Enable deletion protection. — not applicable in the RDS sense (self-hosted); EBS/snapshot-level protection not configured
- [ ] Enable database-level audit logging where appropriate. — not enabled
- [ ] Monitor storage, CPU, memory, connections, locks, and slow queries. — partial: app-level slow-query logging exists (`slow_query_threshold_ms` in `musehub/config.py`); no infrastructure-level DB monitoring exists

### Backups and Recovery

- [ ] Enable automated backups. — **not done, verified live**: no crontab installed, `rclone` not present, one stale 20-byte manual backup from ~2 months ago is the only thing that exists
- [ ] Set an explicit retention period. — N/A, no backups running
- [ ] Enable point-in-time recovery where supported. — not available for self-hosted Postgres without WAL archiving, which isn't configured
- [ ] Configure scheduled longer-term snapshots. — not done
- [ ] Configure cross-account backup copies for critical production data. — not done
- [ ] Consider cross-region copies based on the agreed disaster model. — not done
- [ ] Protect backup vaults from routine production credentials. — N/A, no backups exist
- [ ] Encrypt backup recovery points. — N/A, no backups exist
- [ ] Enable backup-vault lock or equivalent immutability where justified. — N/A, no backups exist
- [ ] Monitor backup-job failures. — N/A, no backup job exists
- [ ] Define retention and deletion policies. — not done
- [ ] Document the restoration procedure. — not done
- [ ] Perform an actual restore test before launch. — **cannot happen until backups exist**
- [ ] Schedule recurring restore tests. — N/A yet
- [ ] Measure restoration time against the RTO. — N/A, RTO undefined (Section 0) and no backups exist
- [ ] Confirm restored data satisfies the RPO. — N/A, same reasons
- [ ] Record the latest successful restore test. — N/A
- [ ] Verify backups are restorable by both Gabriel and Aaron. — N/A
- [ ] Back up any other mission-critical state, including object storage and configuration data. — object storage is content-addressed R2 (durable by design, per Cloudflare); `.env` configuration data has no backup of its own beyond what's in this documentation

### Schema Migrations

- [x] Run migrations automatically but as an explicit deployment step. — confirmed, part of `deploy.sh`, before slot swap
- [ ] Ensure only one migration runner executes at a time. — not independently verified; no explicit lock mechanism found
- [ ] Back up before high-risk migrations. — moot until backups exist
- [ ] Prefer expand-and-contract migrations. — not reviewed across the 75 existing migrations in this pass
- [ ] Avoid combining destructive schema changes with dependent application changes in one irreversible step. — not reviewed
- [ ] Test migrations against a production-like dataset. — not verified
- [ ] Define rollback or forward-repair procedures. — not documented
- [x] Record migration state and duration. — implicitly via Alembic's `alembic_version` table and deploy logs; no structured/queryable record beyond that
- [ ] Alert on migration failure. — not done, no alerting exists (Section 12)
- [x] Ensure older and newer application versions can coexist during rolling or blue/green deployments where required. — pipeline ordering (migrate-then-swap) is the right shape; ⚠️ **found a real risk**: on `alembic upgrade head` failure, `deploy.sh` runs `alembic stamp --purge head` and retries, which could mask a genuinely broken migration as a "stale revision ID" — mitigated by a hard-failing schema-parity gate afterward, but worth explicit awareness since that gate is the only thing standing between this pattern and a silently-broken deploy

---

## 9. CI Pipeline

- [ ] Require pull requests for protected branches.
- [ ] Protect `main`.
- [ ] Require CI checks before merging.
- [ ] Require review from another maintainer for sensitive changes.
- [ ] Define `CODEOWNERS` for:
  - [ ] Infrastructure
  - [ ] Deployment workflows
  - [ ] Authentication and security
  - [ ] Database migrations
- [ ] Run:
  - [ ] Formatting
  - [ ] Linting
  - [ ] Type checking
  - [ ] Unit tests
  - [ ] Integration tests
  - [ ] Migration checks
  - [ ] Dependency vulnerability scanning
  - [ ] Secret scanning
  - [ ] Container or artifact scanning
  - [ ] IaC validation and security scanning
- [ ] Generate immutable, versioned build artifacts.
- [ ] Record the Git commit SHA in every artifact.
- [ ] Produce the artifact once and promote the same artifact through environments.
- [ ] Generate an SBOM where practical.
- [ ] Pin third-party GitHub Actions to immutable commit SHAs for sensitive workflows.
- [ ] Minimize GitHub Actions token permissions.
- [ ] Avoid executing untrusted pull-request code with production-capable credentials.
- [ ] Configure dependency update automation.
- [ ] Define an urgent-patch workflow.
- [ ] Retain CI and deployment logs for an explicit period.
- [ ] Prevent force-pushes and branch deletion on protected branches.

---

## 10. Automated Staging Deployment

- [ ] Create a GitHub `staging` environment.
- [ ] Deploy automatically after successful merges to `main`, or adopt a deliberate release branch if preferred.
- [ ] Authenticate to AWS through GitHub OIDC.
- [ ] Do not store long-lived AWS keys in GitHub.
- [ ] Restrict the staging role to the staging AWS account and resources.
- [ ] Run infrastructure planning.
- [ ] Apply approved or policy-compliant infrastructure changes.
- [ ] Deploy the immutable application artifact.
- [ ] Run database migrations.
- [ ] Run smoke tests.
- [ ] Run health checks.
- [ ] Publish the deployment URL and commit SHA.
- [ ] Notify both maintainers of success or failure.
- [ ] Preserve deployment logs.
- [ ] Automatically roll back application deployment when health checks fail, where safe.
- [ ] Keep database rollback separate and explicit.
- [ ] Prevent concurrent staging deployments from colliding.
- [ ] Make the deployment status visible in GitHub.
- [ ] Record artifact digest, migration version, deployer, and timestamp.

---

## 11. Automated Production Deployment

- [ ] Create a GitHub `production` environment.
- [ ] Use a distinct production AWS role.
- [ ] Restrict that role to the production account and resources.
- [ ] Trigger production deployment through an explicit release action:
  - [ ] Version tag
  - [ ] GitHub release
  - [ ] Manual promotion of a verified artifact
- [ ] Do not rebuild the application for production.
- [ ] Promote the exact artifact tested in staging.
- [ ] Require successful staging deployment and tests first.
- [ ] Require production approval.
- [ ] Prevent self-approval when practical.
- [ ] Permit both Gabriel and Aaron to approve.
- [ ] Require approval from someone other than the person initiating the deployment when practical.
- [ ] Display the complete deployment plan before approval.
- [ ] Back up stateful systems before risky changes.
- [ ] Run migrations as a controlled step.
- [ ] Deploy using:
  - [ ] Rolling deployment
  - [ ] Blue/green deployment
  - [ ] Canary deployment
- [ ] Run post-deployment smoke tests.
- [ ] Monitor error rate, latency, and health during rollout.
- [ ] Stop or roll back when thresholds are exceeded.
- [ ] Record who approved and initiated each deployment.
- [ ] Record commit SHA, artifact identifier, and migration version.
- [ ] Generate release notes.
- [ ] Notify both operators after completion.
- [ ] Prevent concurrent production deployments.
- [ ] Add a documented manual deployment path for emergencies.
- [ ] Add a documented rollback command or workflow.
- [ ] Test rollback before launch.

---

## 12. Observability

- [ ] Centralize application logs.
- [ ] Use structured logs.
- [ ] Include:
  - [ ] Timestamp
  - [ ] Environment
  - [ ] Service
  - [ ] Request or correlation ID
  - [ ] Release version
  - [ ] Severity
- [ ] Prevent credentials, tokens, and sensitive personal data from entering logs.
- [ ] Set explicit log-retention periods.
- [ ] Encrypt logs.
- [ ] Collect infrastructure and application metrics.
- [ ] Build dashboards for:
  - [ ] Request rate
  - [ ] Error rate
  - [ ] Latency
  - [ ] Saturation
  - [ ] Task or instance health
  - [ ] Database capacity
  - [ ] Database connections
  - [ ] Queue depth
  - [ ] Background-job failures
- [ ] Add external uptime checks.
- [ ] Configure alerts for:
  - [ ] Site unavailable
  - [ ] Elevated 5xx responses
  - [ ] Elevated latency
  - [ ] Unhealthy deployment
  - [ ] Database storage pressure
  - [ ] CPU or memory exhaustion
  - [ ] Backup failure
  - [ ] Certificate problems
  - [ ] Queue backlog
  - [ ] Security findings
  - [ ] Unexpected cost growth
- [ ] Route alerts to both Gabriel and Aaron.
- [ ] Define warning versus page-worthy alert severity.
- [ ] Test alert delivery.
- [ ] Ensure every urgent alert links to a runbook.
- [ ] Configure error tracking with release correlation.
- [ ] Define service-level indicators and initial service objectives.
- [ ] Add deployment markers to dashboards.
- [ ] Verify logs and metrics remain available during an application outage.

---

## 13. Security Monitoring and Auditability

- [ ] Enable CloudTrail.
- [ ] Use a multi-region trail for management activity.
- [ ] Include read and write management events.
- [ ] Encrypt CloudTrail logs.
- [ ] Enable log-file validation.
- [ ] Store logs in a nonpublic, access-restricted bucket.
- [ ] Send relevant CloudTrail events to CloudWatch Logs.
- [ ] Enable AWS Config where cost and scope permit.
- [ ] Enable GuardDuty.
- [ ] Enable Security Hub and AWS Foundational Security Best Practices controls.
- [ ] Enable IAM Access Analyzer.
- [ ] Review public and cross-account resource access.
- [ ] Enable S3 Block Public Access at the account level unless explicitly incompatible.
- [ ] Enable ECR or image vulnerability scanning if containers are used.
- [ ] Monitor changes to IAM, KMS, CloudTrail, DNS, and security groups.
- [ ] Alert on root-account activity.
- [ ] Alert on failed console logins and suspicious authentication.
- [ ] Define a vulnerability-remediation SLA.
- [ ] Run a production security review before launch.
- [ ] Record and assign every security finding.
- [ ] Review high-severity findings before every production launch.

---

## 14. Application Security

- [ ] Document the authentication and authorization model.
- [ ] Test authorization independently of UI visibility.
- [ ] Enforce server-side access control.
- [ ] Use secure password hashing if MuseHub stores passwords.
- [ ] Require email verification where relevant.
- [ ] Implement secure account recovery.
- [ ] Rate-limit login and recovery endpoints.
- [ ] Protect against CSRF where applicable.
- [ ] Validate and sanitize untrusted input.
- [ ] Use parameterized database queries.
- [ ] Restrict CORS to deliberate origins.
- [ ] Define a Content Security Policy.
- [ ] Add standard security headers.
- [ ] Validate uploaded file type, size, and storage behavior.
- [ ] Protect against path traversal and unsafe archive extraction.
- [ ] Scan uploaded content if the risk model warrants it.
- [ ] Set request-body and upload limits.
- [ ] Review SSRF exposure from URL-fetching features.
- [ ] Ensure production debug mode is disabled.
- [ ] Ensure stack traces are not returned to users.
- [ ] Review WebSocket or SSE authentication if used.
- [ ] Document data classification and retention.
- [ ] Add an account and data deletion path where applicable.
- [ ] Run an OWASP-oriented review before launch.
- [ ] Review session expiration and token revocation behavior.
- [ ] Verify privileged actions require fresh authorization where appropriate.

---

## 15. Cost Controls

- [ ] Apply consistent cost-allocation tags.
- [ ] Create monthly AWS budgets.
- [ ] Alert Gabriel and Aaron at multiple thresholds.
- [ ] Enable Cost Anomaly Detection.
- [ ] Review NAT Gateway, data-transfer, and logging costs.
- [ ] Set log-retention limits.
- [ ] Set backup-retention limits consistent with recovery requirements.
- [ ] Right-size production after load testing.
- [ ] Review idle resources.
- [ ] Document expected baseline monthly cost.
- [ ] Document the cost implications of autoscaling.
- [ ] Configure service quotas and limits deliberately.
- [ ] Review production cost weekly during the first month.
- [ ] Add owner and purpose tags to every billable resource.
- [ ] Alert on untagged or unexpectedly expensive resources where practical.

---

## 16. Documentation and Runbooks

- [ ] Create a system architecture diagram.
- [ ] Create a data-flow diagram.
- [ ] Document every AWS account and region.
- [ ] Document DNS and certificate ownership.
- [ ] Document environment differences.
- [ ] Document the IaC repository structure.
- [ ] Document local development setup.
- [ ] Document staging deployment.
- [ ] Document production deployment.
- [ ] Document rollback.
- [ ] Document database migration procedures.
- [ ] Document database restoration.
- [ ] Document secret rotation.
- [ ] Document certificate and DNS troubleshooting.
- [ ] Document user-access provisioning and removal.
- [ ] Document break-glass access.
- [ ] Document incident response.
- [ ] Document backup retention and restoration tests.
- [ ] Document common alarms and remediation.
- [ ] Create a service inventory with owners.
- [ ] Create a dependency inventory.
- [ ] Create an operational decision log.
- [ ] Record accepted risks and deferred work.
- [ ] Keep documentation versioned beside the relevant code where practical.
- [ ] Confirm Aaron can follow the documentation without Gabriel’s help.
- [ ] Confirm Gabriel can follow it without relying on personal memory.
- [ ] Add a production launch checklist and rollback decision tree.
- [ ] Add contact information and escalation paths.
- [ ] Review documentation after every incident or major deployment.

---

## 17. Testing Before Launch

- [ ] Run the complete automated test suite.
- [ ] Run production-like integration tests.
- [ ] Test authentication and authorization.
- [ ] Test a clean infrastructure deployment in a disposable environment.
- [ ] Test migration from the current schema.
- [ ] Test application rollback.
- [ ] Test failed migration handling.
- [ ] Test a database restore.
- [ ] Test backup integrity.
- [ ] Test task or instance failure.
- [ ] Test deployment during active traffic.
- [ ] Test autoscaling if enabled.
- [ ] Run basic load and soak tests.
- [ ] Verify timeout behavior.
- [ ] Verify retry behavior.
- [ ] Verify idempotency for jobs and webhooks.
- [ ] Test DNS and TLS from external networks.
- [ ] Test both root and `www`.
- [ ] Verify canonical redirects.
- [ ] Test monitoring and alert delivery.
- [ ] Run a tabletop incident exercise with Gabriel and Aaron.
- [ ] Test the break-glass process.
- [ ] Verify neither maintainer is a single point of operational failure.
- [ ] Test production access from a clean workstation.
- [ ] Test what happens when a secret, database, queue, or external API is unavailable.

---

## 18. Cutover Plan

- [ ] Freeze unrelated infrastructure changes during cutover.
- [ ] Confirm the production environment is healthy.
- [ ] Confirm production secrets are installed.
- [ ] Confirm migrations have been rehearsed.
- [ ] Confirm current backups succeeded.
- [ ] Confirm a restoration has been tested.
- [ ] Confirm dashboards and alerts are active.
- [ ] Confirm Aaron and Gabriel are both available for the cutover.
- [ ] Lower DNS TTL in advance.
- [ ] Deploy the approved production artifact.
- [ ] Run migrations.
- [ ] Run internal smoke tests.
- [ ] Update DNS.
- [ ] Verify:
  - [ ] DNS resolution
  - [ ] TLS
  - [ ] Root domain
  - [ ] `www`
  - [ ] API
  - [ ] Authentication
  - [ ] Critical user paths
  - [ ] Background workers
  - [ ] Outbound email
  - [ ] Object uploads and downloads
- [ ] Monitor logs, error rate, and latency closely.
- [ ] Keep the previous environment available for rollback during the agreed window.
- [ ] Execute rollback if predetermined thresholds are crossed.
- [ ] Raise DNS TTL after stability is confirmed.
- [ ] Record the launch artifact and configuration versions.
- [ ] Announce launch status to stakeholders.
- [ ] Record all manual actions taken during cutover.
- [ ] Confirm production backups run successfully after cutover.

---

## 19. Post-Launch

- [ ] Review production metrics after the first hour.
- [ ] Review again after 24 hours.
- [ ] Review again after one week.
- [ ] Inspect AWS costs after 24 hours and one week.
- [ ] Review security findings.
- [ ] Confirm backups continue to succeed.
- [ ] Perform another restore test after production has real data.
- [ ] Remove temporary cutover access and resources.
- [ ] Remove obsolete staging-derived credentials.
- [ ] Resolve or formally record every launch exception.
- [ ] Conduct a blameless launch retrospective.
- [ ] Update documentation with lessons learned.
- [ ] Schedule:
  - [ ] Monthly dependency updates
  - [ ] Monthly access review during the early production period
  - [ ] Quarterly disaster-recovery exercise
  - [ ] Quarterly permissions review
  - [ ] Quarterly cost review
  - [ ] Annual domain and recovery-contact review

---

# Recommended Implementation Order

## P0 — Before Creating Production

- [ ] Inventory staging.
- [ ] Establish shared organizational ownership and MFA.
- [ ] Decide AWS account boundaries.
- [ ] Select and implement Infrastructure as Code.
- [ ] Define RPO, RTO, and domain topology.
- [ ] Decide the canonical hostname and DNS provider.

## P1 — Build Production

- [ ] Provision the production account and environment through IaC.
- [ ] Configure identity, networking, encryption, and secrets.
- [ ] Configure the database, automated backups, and restore testing.
- [ ] Configure GitHub OIDC and separate deployment roles.
- [ ] Build staging and production pipelines.
- [ ] Add logs, dashboards, alarms, and audit trails.
- [ ] Complete operational documentation.

## P2 — Launch

- [ ] Configure certificates and DNS.
- [ ] Rehearse migrations, deployment, and rollback.
- [ ] Run security and load tests.
- [ ] Complete the cutover.
- [ ] Validate production and document the outcome.

---

# Suggested Ownership Split

| Workstream | Primary | Reviewer / Backup | Status |
|---|---|---|---|
| Staging architecture inventory | Gabriel | Claude Code | In progress — see Section 0 |
| AWS Organizations and accounts |  |  | Not started |
| IAM Identity Center and permissions |  |  | Not started |
| Infrastructure as Code |  |  | Not started |
| Networking and production architecture |  |  | Not started |
| Namecheap, DNS, and TLS |  |  | Not started |
| Database and backups |  |  | Not started |
| GitHub CI/CD |  |  | Not started |
| Logging, metrics, and alerts |  |  | Not started |
| Security review |  |  | Not started |
| Documentation and runbooks |  |  | Not started |
| Cutover coordination |  |  | Not started |

---

# Decision Log

| Date | Decision | Rationale | Owner |
|---|---|---|---|
| 2026-08-04 | Accept single AWS account (`992382692655`, shared with Stori) as technical debt rather than splitting into AWS Organizations before launch | Keeps focus on getting production launched; account split can happen later without blocking | Gabriel |
| 2026-08-04 | Architecture target (harden current EC2/Docker-Compose vs. redesign to ECS+ALB+RDS) left open | Insufficient information at Section 0 time; revisit once inventory and RPO/RTO are settled | Gabriel |
| 2026-08-04 | IaC tool selection deferred; Terraform/OpenTofu recommended given the actual footprint (2 EC2, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params) | Small, EC2-centric footprint doesn't need CDK's language ceremony or CloudFormation's verbosity | Gabriel |
| 2026-08-04 | Every "GitHub" checklist item is translated to its MuseHub equivalent (proposals for PRs, MuseHub branch protection, `musehub-runner` CI, MuseHub collaborator roles) | Absolute ecosystem policy — git/GitHub are never the VCS, CI, or collaboration platform for this project; GitHub is a passive backup mirror only | Gabriel (policy), applied throughout by Claude Code |
| 2026-08-08 | Section 6's "Namecheap vs. Route 53" question reframed as "Cloudflare vs. Route 53" | Live `dig NS musehub.ai` confirms DNS is already hosted at Cloudflare (registrar-only at Namecheap); `docs/infrastructure.md` was stale on this. Recommend staying on Cloudflare DNS given it also does edge proxy/TLS/possibly R2 | Claude Code (finding), decision pending Gabriel |

---

# Accepted Risks and Deferred Work

> ⚠️ **Highest-priority item in this table**: there is currently no usable database backup,
> verified live on staging (no cron job, no rclone, one stale 20-byte manual dump). If the
> production database is lost today, there is no recovery path. This should not wait for the rest
> of the checklist sweep to be addressed.

| Item | Risk | Mitigation | Target Date | Owner |
|---|---|---|---|---|
| No usable database backup exists (verified live: no cron, no rclone, one stale 20-byte manual dump from ~2 months ago) | Total, unrecoverable data loss on any DB failure, corruption, or operator error | Wire `deploy/backup.sh` in for real: install rclone, configure R2 remote, install the cron job, verify a real non-empty backup lands, then perform an actual restore test | ASAP — before any other Section 8 work | Gabriel |
| Single Postgres role used for app runtime, migrations, and admin access | An app-level SQL injection or bug would have migration/admin-level DB privileges, not just runtime privileges | Split into separate roles (app runtime, migrations, read-only, admin) per the checklist's explicit ask | Before Section 8 sign-off | Gabriel |
| Migration failure-handling (`alembic stamp --purge head` on upgrade failure) could mask a genuinely broken migration as a stale-revision-ID issue | A broken migration could silently ship if the follow-up schema-parity gate ever has a false negative | Document the reliance on `schema_gate` explicitly; consider distinguishing failure causes rather than blanket stamp-and-retry | Before Section 8 sign-off | Gabriel |
| Single AWS account shared with Stori | Blast radius: a MuseHub compromise/misconfig can affect Stori resources and vice versa; billing and ownership entangled | Revisit AWS Organizations split post-launch; `musehub-infra` IAM user already scoped narrowly (verified via access-denied tests on IAM/CloudTrail/GuardDuty/Route53/etc.) | TBD | Gabriel |
| `docs/infrastructure.md` is stale — object storage is actually Cloudflare R2, not a local Docker volume as documented | Anyone reading the doc (including Aaron) gets the wrong mental model of where pushed content lives | Correct `docs/infrastructure.md` directly (separate from this checklist effort); resolved here via live proof this session | Soon | Gabriel |
| Secrets are hand-provisioned per instance, not via the drafted SSM pipeline (`deploy/secrets.sh`) — no rotation, no audit trail beyond manual memory | Rotation and auditability (Section 7's actual goals) aren't met by the current hand-provisioned `.env` approach | Gabriel to decide whether production adopts the SSM pipeline from day one rather than repeating staging's hand-provisioning | Before Section 7 sign-off | Gabriel |
| No RPO/RTO, traffic, or availability targets defined | Can't size backup cadence, DR posture, or capacity correctly without them | Define before Section 5 (architecture) and Section 8 (backups) are finalized | Before P1 build phase | Gabriel |
| Canonical hostname (root vs. `www`) and DNS host (Namecheap vs. Route 53) undecided | Blocks finalizing Section 6 and the cutover plan | Decide before Section 18 cutover | Before cutover | Gabriel |
| Deploys use a single shared AWS credential (`musehub-infra` in `~/.aws/credentials`), not individual identities | Violates "no shared AWS access keys"; no accountability for who ran what; single point of failure if Gabriel is unavailable | Resolve via Section 3 (IAM Identity Center + per-person roles) before Section 1's independent-operation test can pass | Before Section 1 sign-off | Gabriel |
| Aaron has no AWS identity or verified Namecheap access today | Aaron cannot operate independently — fails the checklist's explicit "test that Aaron can independently…" requirement | Create Aaron's AWS identity via Section 3; confirm/grant Namecheap access | Before Section 1 sign-off | Gabriel |