MuseHub Production-Readiness Checklist

Master checklist for promoting MuseHub from staging to production, automating deployments, sharing operational ownership between Gabriel and Aaron, and applying security, reliability, backup, documentation, and infrastructure best practices.

Working method: sections are swept in order, once each. Unresolved items are flagged inline and rolled up into the Decision Log / Accepted Risks tables at the bottom rather than blocking progress on later sections. See docs/production-readiness/ for the per-section companion docs with full verification detail.

0. Decisions and Inventory

Full detail: production-readiness/00-inventory-and-decisions.md

• [ ] Document the current staging architecture.
  • [x] AWS account and region — 992382692655, us-east-1 (verified live)
  • [x] Compute platform: ECS, EC2, App Runner, Elastic Beanstalk, Lambda, EKS, etc. — raw EC2 (t3.small/t3.medium), Docker Compose, no managed compute service
  • [x] Load balancer, API Gateway, CloudFront, or other ingress — none; Cloudflare proxy → nginx on the instance
  • [x] Database engine and topology — self-hosted PostgreSQL 16, single instance, no RDS, no replica
  • [x] Redis, queues, object storage, search, background workers — background worker (musehub-runner) confirmed; object storage confirmed live on Cloudflare R2 (bucket musehub-staging, proven by a successful push this session — docs/infrastructure.md corrected); no Redis/search identified
  • [x] VPC, subnets, route tables, NAT gateways, and security groups — default VPC, no custom subnets, no NAT gateway, musehub-sg verified live (443 from Cloudflare ranges only)
  • [x] Secrets and configuration sources — hand-provisioned .env per instance (not SSM-automated); confirmed it includes working R2 credentials docs/infrastructure.md didn't document. Open question for Gabriel: should production adopt the drafted SSM pipeline (deploy/secrets.sh) from day one instead of hand-provisioning like staging?
  • [x] Logging and monitoring — confirmed none live (no CloudWatch alarms, no SNS topics exist despite a script that would create them)
  • [x] Current deployment process — documented: manual, human-triggered deploy/push.sh, no CI/CD automation
• [x] Inventory every manually created AWS resource. — 2 EC2 + 2 EIP, 1 SG, 1 IAM user, 1 IAM instance role, 1 ECR repo, 1 S3 bucket, 3 SSM params (all listed in companion doc)
• [ ] Inventory every external dependency.
  • [x] GitHub — backup-mirror only, never VCS/CI/collaboration (ecosystem policy)
  • [x] Namecheap — registrar + DNS host for musehub.ai
  • [x] Email provider — confirmed none exists
  • [x] OAuth providers — confirmed none exist (MSign/Ed25519 only)
  • [x] Payment or billing providers — confirmed none exist
  • [x] Error tracking — confirmed none exists
  • [ ] Third-party APIs — Cloudflare R2/Worker usage inferred from scripts but not independently verified against the Cloudflare dashboard
• [x] Identify all stateful and mission-critical data. — Postgres DB, object store, musehub-releases S3 bucket, SSM params (all listed in companion doc)
• [ ] Define expected initial traffic and reasonable growth assumptions. — not yet defined, needs Gabriel
• [ ] Define availability expectations. — not yet defined, needs Gabriel
• [ ] Define acceptable downtime during deployment. — not yet defined, needs Gabriel
• [ ] Define recovery objectives.
  • [ ] RPO: maximum acceptable data loss — not yet defined, needs Gabriel
  • [ ] RTO: maximum acceptable restoration time — not yet defined, needs Gabriel
• [ ] Decide the canonical public URLs.
  • [x] Root (musehub.ai) — exists, live
  • [x] www.musehub.ai — exists, live
  • [ ] API hostname — none exists; MuseHub serves API + web from the same origin, no api. subdomain in use today
  • [x] Staging (staging.musehub.ai) — exists, live
• [ ] Choose either the root domain or www as canonical. — not yet decided, needs Gabriel
• [ ] Redirect the noncanonical hostname permanently to the canonical one. — no redirect logic found in nginx config; not implemented
• [ ] Decide whether DNS remains hosted at Namecheap or moves to Route 53. — not yet decided, needs Gabriel
• [x] Decide whether production and non-production will use separate AWS accounts. — Decided: single account accepted as technical debt for now (see Decision Log)

---

1. Ownership and Shared Access

No shared human accounts, shared passwords, or shared AWS access keys. Full detail: production-readiness/01-ownership-and-access.md. "GitHub organization" items below are the MuseHub-native equivalent per the Decision Log.

• [ ] Confirm the domain is held by the company rather than an individual wherever practical. — needs Gabriel (Namecheap console)
• [ ] Confirm AWS account ownership and billing contacts. — needs Gabriel (AWS console)
• [x] Confirm MuseHub repo ownership/collaboration model. — gabriel owns both repos; aaronrene has write collaborator access on musehub and muse (verified via muse hub collaborator list)
• [ ] Add both Gabriel and Aaron as appropriate organizational owners or administrators. — done for MuseHub repos (Aaron has write); not done for AWS/Namecheap — needs Gabriel
• [ ] Ensure neither person is the sole recovery path for:
  • [ ] Namecheap — needs Gabriel
  • [ ] AWS — needs Gabriel
  • [x] MuseHub — Aaron already has write access independent of Gabriel
  • [ ] DNS — needs Gabriel
  • [ ] Database — needs Gabriel (currently only reachable via musehub-infra shared AWS credential)
  • [ ] Production secrets — needs Gabriel
  • [ ] Billing — needs Gabriel
  • [ ] Backups — needs Gabriel (also blocked on resolving Section 0 Contradiction 2)
• [ ] Use company-controlled email addresses for infrastructure accounts. — needs Gabriel
• [ ] Configure at least two trusted recovery paths. — needs Gabriel
• [ ] Store emergency recovery documentation in a location accessible to both founders. — needs Gabriel
• [ ] Create a documented joiner–mover–leaver access procedure. — not started
• [ ] Create a quarterly access-review process. — not started
• [ ] Record ownership for every operational area:
  • [ ] AWS administration — currently Gabriel only, undocumented
  • [ ] DNS and certificates — currently Gabriel only, undocumented
  • [ ] Deployments — currently Gabriel only (shared musehub-infra credential), undocumented
  • [ ] Database administration — currently Gabriel only, undocumented
  • [ ] Incident response — no process exists
  • [ ] Billing — currently Gabriel only, undocumented
  • [ ] Backups — no automated backups exist yet (Section 0 Contradiction 2)
• [ ] Create an emergency break-glass procedure. — not started
• [ ] Test that Aaron can independently:
  • [ ] Sign in to AWS — cannot test — no AWS identity exists for Aaron yet
  • [ ] Inspect production — cannot test
  • [ ] Deploy staging — cannot test as designed — today's deploy uses a shared AWS credential (musehub-infra), not an individual one; this violates the "no shared AWS access keys" principle this section opens with
  • [ ] Approve or deploy production — cannot test — prod deploy is currently broken regardless (Section 0: no IAM instance profile on the prod instance)
  • [ ] Inspect logs and alarms — cannot test — no logs/alarms exist yet (Section 12)
  • [ ] Restore from backup — cannot test — no backups exist yet (Section 0 Contradiction 2)
  • [ ] Update DNS when authorized — cannot test — Aaron's Namecheap access unverified
• [ ] Test that Gabriel can perform the same tasks without relying on undocumented knowledge. — not started; current process is entirely undocumented outside docs/infrastructure.md

---

2. AWS Account Structure

Full detail: production-readiness/02-aws-account-structure.md

Recommended Target

• [ ] Create or verify an AWS Organization. — deferred, see Decision Log (single account accepted as technical debt)
• [ ] Use distinct AWS accounts for:
  • [ ] Management — deferred
  • [ ] Production — deferred
  • [ ] Non-production — deferred
• [ ] Consider separate security and log-archive accounts as the operation grows. — deferred
• [ ] Place production and non-production into separate organizational units. — deferred (no Organization exists)
• [ ] Apply Service Control Policies only after testing them carefully. — N/A while deferred
• [ ] Centralize billing. — N/A, single account
• [ ] Enable account-level budget alerts. — not done; tracked under Section 15 (Cost Controls) instead since it applies within the single account regardless of structure — needs Gabriel (AWS console access required)
• [ ] Set alternate security, operations, and billing contacts on every account. — needs Gabriel
• [ ] Restrict enabled AWS regions to the regions MuseHub actually uses, where feasible. — needs Gabriel; note the account is shared with Stori (region us-east-2 bucket observed), so any region restriction must account for both projects
• [x] Document any temporary decision to keep staging and production in one AWS account as accepted technical debt. — done, see Decision Log and Accepted Risks in this checklist

---

3. Identity, Authentication, and Permissions

Full detail: production-readiness/03-identity-and-permissions.md Document-only pass — no IAM changes made. Real build happens after the full sweep.

Root and Recovery Access

• [ ] Protect every AWS root account with phishing-resistant MFA where possible. — needs Gabriel (AWS console)
• [ ] Do not create root access keys. — needs Gabriel to confirm
• [ ] Do not use root for routine administration. — needs Gabriel to confirm
• [ ] Secure the root email account with MFA. — needs Gabriel
• [ ] Store root recovery information securely. — needs Gabriel
• [ ] Document the legitimate uses of root. — not started
• [ ] Configure multi-person approval for emergency root use if practical. — not started

Human Access

• [ ] Configure AWS IAM Identity Center. — not started; visibility blocked (sso-admin denied) — needs Gabriel to check/build
• [ ] Create separate identities for Gabriel and Aaron. — not started (blocks Section 1's independent-operation test)
• [ ] Require MFA for both. — not started
• [ ] Prefer hardware security keys or passkeys. — not started
• [ ] Define permission sets, for example:
  • [ ] Administrator — not started
  • [ ] ProductionOperator — not started
  • [ ] Developer — not started
  • [ ] ReadOnly — not started
  • [ ] Billing — not started
• [ ] Give routine access through the least-privileged applicable role. — not started
• [ ] Require explicit elevation for administrative work. — not started
• [ ] Avoid permanent AdministratorAccess for everyday use where practical. — not started
• [ ] Use IAM Access Analyzer to inspect and refine permissions. — not started; needs broader read access
• [ ] Review unused roles, users, policies, and credentials. — not started; needs broader read access
• [ ] Document who can approve and perform production changes. — not started

Workload Access

• [x] Give every application or service its own IAM role. — 3 separate IAM users confirmed live (musehub-infra, musehub-app, stori-assets-app), no shared workload credential across services
• [x] Do not place AWS access keys in source code. — confirmed via grep, none found
• [x] Do not place long-lived AWS access keys in GitHub secrets. — N/A, no GitHub Actions/secrets used (see Decision Log); equivalent check for MuseHub-runner CI config deferred to Section 9
• [ ] Restrict every deployment role to:
  • [ ] The correct repository — N/A, no deployment role exists yet (deploys use a human + shared credential)
  • [ ] The correct branch, tag, or GitHub environment — N/A, same reason
  • [ ] The correct AWS account — N/A, same reason
  • [ ] The required AWS resources only — partially true of musehub-infra's scoping, but it's a human-shared credential, not a deployment role
• [ ] Separate staging and production deployment roles. — not done — both environments share the musehub-infra credential
• [ ] Validate trust policies and permission boundaries. — not started; needs broader read access
• [ ] Prefer temporary credentials everywhere. — not done — all current credentials are long-lived IAM user access keys

---

4. Infrastructure as Code

Full detail: production-readiness/04-iac.md Document-only pass. No IaC written yet — deferred to the build phase after the full sweep.

• [ ] Select one Infrastructure-as-Code system:
  • [ ] Terraform/OpenTofu — recommended, not yet formally chosen by Gabriel
  • [ ] AWS CDK
  • [ ] CloudFormation
• [ ] Capture the existing staging infrastructure in code. — not started; full resource list is in the Section 0 doc
• [ ] Stop making undocumented console-only changes. — behavioral change, not yet in effect
• [ ] Build reusable modules rather than duplicating entire configurations. — N/A until IaC exists
• [ ] Provide separate environment configuration for:
  • [ ] Staging — N/A until IaC exists
  • [ ] Production — N/A until IaC exists
• [ ] Keep production and staging state isolated. — N/A until IaC exists
• [ ] Encrypt remote IaC state. — N/A until IaC exists
• [ ] Enable versioning and locking for remote state. — N/A until IaC exists
• [ ] Restrict access to IaC state because it may contain sensitive metadata. — N/A until IaC exists
• [ ] Pin provider, module, and action versions. — N/A until IaC exists
• [ ] Add formatting and validation checks. — N/A until IaC exists
• [ ] Add static security scanning. — N/A until IaC exists
• [ ] Require a reviewed plan before production changes. — N/A until IaC exists
• [ ] Detect configuration drift. — N/A until IaC exists
• [ ] Document how to import existing resources into IaC. — not started; note this must be import, not recreate, since real resources already exist and are live
• [ ] Add environment-standard tags:
  • [ ] Application — not applied anywhere currently
  • [ ] Environment — not applied anywhere currently
  • [ ] Owner — not applied anywhere currently
  • [ ] Managed-by — not applied anywhere currently
  • [ ] Cost-center — not applied anywhere currently
  • [ ] Data-classification — not applied anywhere currently
• [ ] Make a fresh environment reproducible from source control and documented prerequisites. — not started
• [ ] Add a teardown guard so production cannot be accidentally destroyed. — N/A until IaC exists
• [ ] Add deletion protection to critical resources. — not started (also relevant to Section 8 — DB deletion protection specifically)
• [ ] Add policy checks that prevent production resources from being publicly exposed unintentionally. — N/A until IaC exists
• [ ] Ensure IaC changes are reviewed by someone other than the author when practical. — N/A until IaC exists

---

5. Production Architecture

Full detail: production-readiness/05-production-architecture.md

• [ ] Reassess whether staging's sizing and topology are production-appropriate. — flagged: staging is t3.medium, prod is t3.small — inconsistent, no documented reason
• [x] Do not copy staging data into production unless explicitly sanitized and required. — not done, and correctly not done — no evidence of staging→prod data copying anywhere
• [ ] Use separate production:
  • [x] Database — separate instance entirely (different EC2 host)
  • [ ] Credentials — unresolved pending Section 0 Contradiction 2
  • [ ] Secrets — unresolved pending Section 0 Contradiction 2
  • [ ] Object-storage buckets — unresolved pending Section 0 Contradiction 1
  • [ ] Queues — none exist in either environment
  • [ ] Cache — none exist in either environment
  • [ ] Logs — not separated in any meaningful way; no centralized logging exists yet (Section 12)
  • [ ] Encryption keys where appropriate — not reviewed yet
• [ ] Deploy production across at least two Availability Zones where supported and justified. — not done; single instance, no ASG, documented as intentional SPOF below
• [x] Keep databases and internal services in private subnets. — partially true in spirit (DB only reachable via the app's Docker network, not the public internet) but not true architecturally (default VPC, no real private subnet); security group is the actual control here
• [x] Expose only the public ingress layer. — confirmed: only 443 open, restricted to Cloudflare IP ranges
• [x] Eliminate direct public database access. — confirmed, DB has no public exposure
• [x] Use security groups by service relationship rather than broad CIDR ranges. — reasonable substitution given the architecture: scoped to Cloudflare's published IP ranges, not 0.0.0.0/0
• [x] Prohibit unrestricted administrative ports such as public SSH. — confirmed, no port 22 open
• [x] Use AWS Systems Manager Session Manager rather than public SSH where applicable. — confirmed, this is already the only access path
• [x] Configure health checks. — confirmed: /healthz probes both DB and object storage, gates blue/green flips
• [ ] Configure graceful application shutdown. — not verified — no SIGTERM handling found in musehub/main.py; uvicorn defaults apply
• [ ] Configure connection draining. — partial: blue/green waits for new slot health before flip, but old slot's in-flight requests during docker rm aren't verified graceful
• [x] Set sensible application and load-balancer timeouts. — confirmed: deliberate per-route nginx timeouts (5s health, 60s default, up to 3600s for streaming/MCP)
• [ ] Configure autoscaling or document why fixed capacity is appropriate. — neither done; fixed capacity with no written justification
• [ ] Set resource requests, limits, or instance sizes deliberately. — instance types chosen but not justified with load data
• [ ] Verify production can survive a routine instance or task replacement. — not verified; single-instance architecture means instance failure has no automatic failover (documented SPOF below)
• [x] Confirm background jobs are idempotent or safely retryable. — confirmed: claim_next_job uses SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent claiming, reclaim_stale_jobs resets crashed-worker jobs back to pending (see Section 9 doc)
• [ ] Confirm database migrations are compatible with rolling deployments. — pipeline runs migrations before slot flip (right shape); expand/contract discipline not reviewed — deferred to Section 8
• [x] Add rate limiting and abuse controls. — confirmed: slowapi-based per-route limits + global baseline + bot-throttle middleware, already implemented
• [ ] Consider AWS WAF for public endpoints. — N/A as literally stated (no ALB/CloudFront); Cloudflare is the substitute layer, but its WAF/rate-limit config is unverified — needs Gabriel to check Cloudflare dashboard
• [x] Configure DDoS protections appropriate to the architecture. — Cloudflare edge proxy already provides this; not independently verified beyond "Cloudflare is in front of everything"
• [ ] Establish AWS service quota monitoring for relevant resources. — not started
• [x] Document every intentional single point of failure. — done, see companion doc: single EC2 per environment, single Postgres instance, single AWS account (shared with Stori), single Cloudflare account, single human deployer with shared credential
• [ ] Create a plan to remove unacceptable single points of failure. — needs Gabriel — depends on RPO/RTO/availability targets that are still undefined (Section 0)

---

6. DNS, Domain, and TLS

Full detail: production-readiness/06-dns-domain-tls.md ⚠️ Finding: DNS is actually hosted at Cloudflare, not Namecheap — verified via live dig NS musehub.ai. docs/infrastructure.md is stale on this point. Namecheap is registrar only. This changes several items below from "Namecheap vs Route 53" to "Cloudflare vs Route 53."

Namecheap Account

• [ ] Enable MFA on Namecheap for every authorized administrator. — needs Gabriel
• [ ] Use unique individual access rather than shared credentials where the account model permits. — needs Gabriel
• [ ] Enable domain lock. — needs Gabriel
• [ ] Verify registrant and recovery contact details. — needs Gabriel
• [ ] Enable automatic renewal. — needs Gabriel
• [ ] Verify the payment method will not unexpectedly expire. — needs Gabriel
• [ ] Document who may change nameservers or DNS. — needs Gabriel; note DNS itself is now on Cloudflare, so this also means documenting Cloudflare access, not just Namecheap
• [ ] Export or document all existing DNS records before making changes. — needs Gabriel, from the Cloudflare dashboard, not Namecheap's Advanced DNS tab
• [ ] Preserve mail-related MX, SPF, DKIM, and DMARC records. — needs Gabriel to confirm these exist and are preserved in Cloudflare's DNS
• [ ] Confirm both Gabriel and Aaron have an approved recovery path. — needs Gabriel, for both Namecheap and Cloudflare

DNS Design

• [ ] Decide between:
  • [x] Namecheap-hosted DNS — not the current state; superseded by the finding above
  • [ ] Route 53-hosted DNS with Namecheap remaining registrar — open; recommendation below
• [x] Prefer Route 53... — recommendation revised: current DNS host is Cloudflare (not Namecheap), and Cloudflare is also doing edge proxy/TLS/possibly R2 — recommend staying on Cloudflare DNS rather than fragmenting the stack by moving to Route 53, unless Gabriel has a specific reason to want AWS-native alias records
• [ ] Create a Route 53 hosted zone if selected. — N/A per above recommendation, pending Gabriel's decision
• [x] Reproduce every required existing record before switching nameservers. — N/A, no switch planned per recommendation
• [ ] Lower DNS TTL ahead of cutover. — deferred to Section 18 (Cutover Plan)
• [x] Configure the root/apex record. — exists, live, resolves via Cloudflare
• [x] Configure www. — exists, live
• [ ] Configure api if needed. — no api. hostname exists; MuseHub serves API + web from one origin — confirm this is intentional
• [x] Keep staging on a clearly separate hostname. — confirmed, staging.musehub.ai
• [x] Do not expose internal AWS service hostnames unnecessarily. — confirmed, only the public hostnames resolve; no raw AWS DNS names exposed
• [ ] Verify IPv4 and IPv6 behavior. — not tested in this pass
• [ ] Configure a canonical-host redirect. — not implemented — no redirect logic found; root and www both resolve without a canonical redirect
• [ ] Test DNS from multiple resolvers after cutover. — deferred to Section 18
• [ ] Raise TTL again after the deployment is stable. — deferred to Section 18
• [ ] Manage DNS records through IaC where practical. — deferred to Section 4 build phase (Terraform's cloudflare provider)

TLS

• [ ] Request an AWS Certificate Manager certificate. — likely N/A for this architecture — TLS terminates at Cloudflare's edge, not an AWS ALB/CloudFront; needs Gabriel to confirm there's no AWS-facing TLS need I'm missing
• [ ] Include every required hostname. — N/A pending above
• [ ] Prefer DNS validation. — N/A pending above
• [ ] Retain ACM validation records for automatic renewal. — N/A pending above
• [x] Redirect all HTTP traffic to HTTPS. — Cloudflare/nginx setup already enforces HTTPS-only (port 80 not open at all)
• [x] Use modern TLS policies. — Cloudflare-managed edge TLS; verified live cert is current (Let's Encrypt, issued 2026-08-01)
• [x] Configure HSTS only after HTTPS and subdomain behavior are verified. — already live: Strict-Transport-Security with includeSubDomains confirmed in nginx config
• [ ] Confirm certificate-expiration monitoring. — not done — no alerting exists yet (Section 12); manual verification only (as done earlier this session)
• [x] Test the full certificate chain. — verified via openssl s_client this session; valid, correctly issued, correct CN
• [ ] Confirm cookies use:
  • [x] N/A — MuseHub uses header-based MSign auth, no cookies exist anywhere in the app

---

7. Secrets and Application Configuration

Full detail: production-readiness/07-secrets-and-config.md

• [x] Inventory all secrets. — done, 6 secrets identified (DB password, webhook key, runner token, R2 credentials, worker key, worker URL)
• [x] Remove secrets from source control and repository history. — confirmed via grep, none found
• [ ] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate. — the real work item: deploy/secrets.sh implements this but only 3/7 required SSM params exist; not live yet
• [ ] Encrypt secrets with KMS. — partial; the 3 existing SSM params are SecureString/KMS-encrypted, rest aren't in SSM
• [x] Use separate staging and production secrets. — confirmed: separate .env, separate R2 buckets, separate SSM prefixes
• [ ] Give workloads access only to the secrets they require. — likely N/A for a single-process-per-instance app; not force-fit
• [x] Do not expose production secrets to pull-request workflows. — N/A, no PR-based CI exists
• [ ] Do not print secrets in logs. — not verified; no centralized logs exist yet to audit (Section 12)
• [ ] Configure secret rotation where supported. — not done
• [ ] Rotate credentials during the production launch. — deferred to Section 18
• [ ] Document emergency rotation procedures. — not done
• [x] Separate secrets from nonsecret configuration. — confirmed: musehub/config.py's Settings class already separates these cleanly in code
• [x] Validate required configuration at application startup. — confirmed via pydantic_settings.BaseSettings
• [x] Fail safely when critical configuration is missing. — confirmed, same mechanism plus secrets.sh's own preflight die checks
• [x] Remove default credentials and sample keys. — confirmed, no placeholder secrets in config.py; secrets.sh also actively rejects known-weak passwords
• [ ] Audit OAuth callback URLs and API allowlists for production. — N/A, no OAuth exists
• [ ] Audit every external webhook secret. — WORKER_INTERNAL_KEY identified; validation logic in musehub/worker.py not independently reviewed in this pass
• [ ] Document ownership and rotation frequency for every production secret. — not done, blocked on Section 1

---

8. Database Security, Migrations, and Backups

Full detail: production-readiness/08-database-backups-migrations.md ⚠️ Verified live via SSM: there is currently no usable backup of the database, anywhere.

Database Security

• [ ] Enable encryption at rest. — not verified in this pass
• [ ] Enforce encryption in transit. — not verified in this pass
• [ ] Use a production-specific KMS key where appropriate. — not started
• [ ] Store credentials in Secrets Manager. — see Section 7
• [ ] Avoid administrator credentials for normal application traffic. — not done — single musehub role used for app traffic, migrations, and admin access
• [ ] Create separate roles for:
  • [ ] Application runtime — not done
  • [ ] Migrations — not done
  • [ ] Read-only operations — not done
  • [ ] Administration — not done
• [x] Keep the database off the public internet. — confirmed
• [x] Restrict security-group access to the application layer. — confirmed
• [ ] Enable deletion protection. — not applicable in the RDS sense (self-hosted); EBS/snapshot-level protection not configured
• [ ] Enable database-level audit logging where appropriate. — not enabled
• [ ] Monitor storage, CPU, memory, connections, locks, and slow queries. — partial: app-level slow-query logging exists (slow_query_threshold_ms in musehub/config.py); no infrastructure-level DB monitoring exists

Backups and Recovery

• [ ] Enable automated backups. — not done, verified live: no crontab installed, rclone not present, one stale 20-byte manual backup from ~2 months ago is the only thing that exists
• [ ] Set an explicit retention period. — N/A, no backups running
• [ ] Enable point-in-time recovery where supported. — not available for self-hosted Postgres without WAL archiving, which isn't configured
• [ ] Configure scheduled longer-term snapshots. — not done
• [ ] Configure cross-account backup copies for critical production data. — not done
• [ ] Consider cross-region copies based on the agreed disaster model. — not done
• [ ] Protect backup vaults from routine production credentials. — N/A, no backups exist
• [ ] Encrypt backup recovery points. — N/A, no backups exist
• [ ] Enable backup-vault lock or equivalent immutability where justified. — N/A, no backups exist
• [ ] Monitor backup-job failures. — N/A, no backup job exists
• [ ] Define retention and deletion policies. — not done
• [ ] Document the restoration procedure. — not done
• [ ] Perform an actual restore test before launch. — cannot happen until backups exist
• [ ] Schedule recurring restore tests. — N/A yet
• [ ] Measure restoration time against the RTO. — N/A, RTO undefined (Section 0) and no backups exist
• [ ] Confirm restored data satisfies the RPO. — N/A, same reasons
• [ ] Record the latest successful restore test. — N/A
• [ ] Verify backups are restorable by both Gabriel and Aaron. — N/A
• [ ] Back up any other mission-critical state, including object storage and configuration data. — object storage is content-addressed R2 (durable by design, per Cloudflare); .env configuration data has no backup of its own beyond what's in this documentation

Schema Migrations

• [x] Run migrations automatically but as an explicit deployment step. — confirmed, part of deploy.sh, before slot swap
• [ ] Ensure only one migration runner executes at a time. — not independently verified; no explicit lock mechanism found
• [ ] Back up before high-risk migrations. — moot until backups exist
• [ ] Prefer expand-and-contract migrations. — not reviewed across the 75 existing migrations in this pass
• [ ] Avoid combining destructive schema changes with dependent application changes in one irreversible step. — not reviewed
• [ ] Test migrations against a production-like dataset. — not verified
• [ ] Define rollback or forward-repair procedures. — not documented
• [x] Record migration state and duration. — implicitly via Alembic's alembic_version table and deploy logs; no structured/queryable record beyond that
• [ ] Alert on migration failure. — not done, no alerting exists (Section 12)
• [x] Ensure older and newer application versions can coexist during rolling or blue/green deployments where required. — pipeline ordering (migrate-then-swap) is the right shape; ⚠️ found a real risk: on alembic upgrade head failure, deploy.sh runs alembic stamp --purge head and retries, which could mask a genuinely broken migration as a "stale revision ID" — mitigated by a hard-failing schema-parity gate afterward, but worth explicit awareness since that gate is the only thing standing between this pattern and a silently-broken deploy

---

9. CI Pipeline

Full detail: production-readiness/09-ci-pipeline.md ⚠️ Bigger finding than a typical checklist item: MuseHub-native equivalents of branch protection, required status checks, and CODEOWNERS don't exist as product features yet — this section requires building platform capability, not just configuring deployment. What does exist and is genuinely good: a concurrency-safe internal background job queue.

• [ ] Require proposals (MuseHub's PR-equivalent) for protected branches. — not started, feature doesn't exist
• [ ] Protect main/dev. — not started
• [ ] Require CI checks before merging. — not started; merge_proposal has no CI-status gate
• [ ] Require review from another maintainer for sensitive changes. — not started
• [ ] Define required-reviewer paths (CODEOWNERS-equivalent) for:
  • [ ] Infrastructure — not started
  • [ ] Deployment workflows — not started
  • [ ] Authentication and security — not started
  • [ ] Database migrations — not started
• [ ] Run:
  • [ ] Formatting — manual only, not automated on push
  • [ ] Linting — manual only
  • [ ] Type checking — manual only
  • [ ] Unit tests — manual only
  • [ ] Integration tests — manual only
  • [ ] Migration checks — runs at deploy time (schema_gate, see Section 8), not at CI/proposal time
  • [ ] Dependency vulnerability scanning — not automated
  • [ ] Secret scanning — not automated
  • [ ] Container or artifact scanning — not automated
  • [ ] IaC validation and security scanning — N/A, no IaC exists yet (Section 4)
• [ ] Generate immutable, versioned build artifacts. — partial: ECR image tags exist and are immutable once pushed, but built manually per-environment rather than automatically per-commit
• [x] Record the Git commit SHA in every artifact. — N/A concept (no git); MuseHub commit IDs are content-addressed (sha256:...) and could be embedded in image tags — not currently done, worth adding
• [ ] Produce the artifact once and promote the same artifact through environments. — not done by default; IMAGE_TAG redeploy support in push.sh makes promotion possible but isn't the default flow
• [ ] Generate an SBOM where practical. — not started
• [x] Pin third-party GitHub Actions to immutable commit SHAs for sensitive workflows. — N/A, no GitHub Actions used
• [x] Minimize GitHub Actions token permissions. — N/A, no GitHub Actions used
• [ ] Avoid executing untrusted pull-request code with production-capable credentials. — moot until proposal-triggered CI execution exists at all
• [ ] Configure dependency update automation. — not started
• [ ] Define an urgent-patch workflow. — not documented
• [ ] Retain CI and deployment logs for an explicit period. — N/A, no CI exists to log yet
• [ ] Prevent force-pushes and branch deletion on protected branches. — not started

---

10. Automated Staging Deployment

• [ ] Create a GitHub staging environment.
• [ ] Deploy automatically after successful merges to main, or adopt a deliberate release branch if preferred.
• [ ] Authenticate to AWS through GitHub OIDC.
• [ ] Do not store long-lived AWS keys in GitHub.
• [ ] Restrict the staging role to the staging AWS account and resources.
• [ ] Run infrastructure planning.
• [ ] Apply approved or policy-compliant infrastructure changes.
• [ ] Deploy the immutable application artifact.
• [ ] Run database migrations.
• [ ] Run smoke tests.
• [ ] Run health checks.
• [ ] Publish the deployment URL and commit SHA.
• [ ] Notify both maintainers of success or failure.
• [ ] Preserve deployment logs.
• [ ] Automatically roll back application deployment when health checks fail, where safe.
• [ ] Keep database rollback separate and explicit.
• [ ] Prevent concurrent staging deployments from colliding.
• [ ] Make the deployment status visible in GitHub.
• [ ] Record artifact digest, migration version, deployer, and timestamp.

---

11. Automated Production Deployment

• [ ] Create a GitHub production environment.
• [ ] Use a distinct production AWS role.
• [ ] Restrict that role to the production account and resources.
• [ ] Trigger production deployment through an explicit release action:
  • [ ] Version tag
  • [ ] GitHub release
  • [ ] Manual promotion of a verified artifact
• [ ] Do not rebuild the application for production.
• [ ] Promote the exact artifact tested in staging.
• [ ] Require successful staging deployment and tests first.
• [ ] Require production approval.
• [ ] Prevent self-approval when practical.
• [ ] Permit both Gabriel and Aaron to approve.
• [ ] Require approval from someone other than the person initiating the deployment when practical.
• [ ] Display the complete deployment plan before approval.
• [ ] Back up stateful systems before risky changes.
• [ ] Run migrations as a controlled step.
• [ ] Deploy using:
  • [ ] Rolling deployment
  • [ ] Blue/green deployment
  • [ ] Canary deployment
• [ ] Run post-deployment smoke tests.
• [ ] Monitor error rate, latency, and health during rollout.
• [ ] Stop or roll back when thresholds are exceeded.
• [ ] Record who approved and initiated each deployment.
• [ ] Record commit SHA, artifact identifier, and migration version.
• [ ] Generate release notes.
• [ ] Notify both operators after completion.
• [ ] Prevent concurrent production deployments.
• [ ] Add a documented manual deployment path for emergencies.
• [ ] Add a documented rollback command or workflow.
• [ ] Test rollback before launch.

---

12. Observability

• [ ] Centralize application logs.
• [ ] Use structured logs.
• [ ] Include:
  • [ ] Timestamp
  • [ ] Environment
  • [ ] Service
  • [ ] Request or correlation ID
  • [ ] Release version
  • [ ] Severity
• [ ] Prevent credentials, tokens, and sensitive personal data from entering logs.
• [ ] Set explicit log-retention periods.
• [ ] Encrypt logs.
• [ ] Collect infrastructure and application metrics.
• [ ] Build dashboards for:
  • [ ] Request rate
  • [ ] Error rate
  • [ ] Latency
  • [ ] Saturation
  • [ ] Task or instance health
  • [ ] Database capacity
  • [ ] Database connections
  • [ ] Queue depth
  • [ ] Background-job failures
• [ ] Add external uptime checks.
• [ ] Configure alerts for:
  • [ ] Site unavailable
  • [ ] Elevated 5xx responses
  • [ ] Elevated latency
  • [ ] Unhealthy deployment
  • [ ] Database storage pressure
  • [ ] CPU or memory exhaustion
  • [ ] Backup failure
  • [ ] Certificate problems
  • [ ] Queue backlog
  • [ ] Security findings
  • [ ] Unexpected cost growth
• [ ] Route alerts to both Gabriel and Aaron.
• [ ] Define warning versus page-worthy alert severity.
• [ ] Test alert delivery.
• [ ] Ensure every urgent alert links to a runbook.
• [ ] Configure error tracking with release correlation.
• [ ] Define service-level indicators and initial service objectives.
• [ ] Add deployment markers to dashboards.
• [ ] Verify logs and metrics remain available during an application outage.

---

13. Security Monitoring and Auditability

• [ ] Enable CloudTrail.
• [ ] Use a multi-region trail for management activity.
• [ ] Include read and write management events.
• [ ] Encrypt CloudTrail logs.
• [ ] Enable log-file validation.
• [ ] Store logs in a nonpublic, access-restricted bucket.
• [ ] Send relevant CloudTrail events to CloudWatch Logs.
• [ ] Enable AWS Config where cost and scope permit.
• [ ] Enable GuardDuty.
• [ ] Enable Security Hub and AWS Foundational Security Best Practices controls.
• [ ] Enable IAM Access Analyzer.
• [ ] Review public and cross-account resource access.
• [ ] Enable S3 Block Public Access at the account level unless explicitly incompatible.
• [ ] Enable ECR or image vulnerability scanning if containers are used.
• [ ] Monitor changes to IAM, KMS, CloudTrail, DNS, and security groups.
• [ ] Alert on root-account activity.
• [ ] Alert on failed console logins and suspicious authentication.
• [ ] Define a vulnerability-remediation SLA.
• [ ] Run a production security review before launch.
• [ ] Record and assign every security finding.
• [ ] Review high-severity findings before every production launch.

---

14. Application Security

• [ ] Document the authentication and authorization model.
• [ ] Test authorization independently of UI visibility.
• [ ] Enforce server-side access control.
• [ ] Use secure password hashing if MuseHub stores passwords.
• [ ] Require email verification where relevant.
• [ ] Implement secure account recovery.
• [ ] Rate-limit login and recovery endpoints.
• [ ] Protect against CSRF where applicable.
• [ ] Validate and sanitize untrusted input.
• [ ] Use parameterized database queries.
• [ ] Restrict CORS to deliberate origins.
• [ ] Define a Content Security Policy.
• [ ] Add standard security headers.
• [ ] Validate uploaded file type, size, and storage behavior.
• [ ] Protect against path traversal and unsafe archive extraction.
• [ ] Scan uploaded content if the risk model warrants it.
• [ ] Set request-body and upload limits.
• [ ] Review SSRF exposure from URL-fetching features.
• [ ] Ensure production debug mode is disabled.
• [ ] Ensure stack traces are not returned to users.
• [ ] Review WebSocket or SSE authentication if used.
• [ ] Document data classification and retention.
• [ ] Add an account and data deletion path where applicable.
• [ ] Run an OWASP-oriented review before launch.
• [ ] Review session expiration and token revocation behavior.
• [ ] Verify privileged actions require fresh authorization where appropriate.

---

15. Cost Controls

• [ ] Apply consistent cost-allocation tags.
• [ ] Create monthly AWS budgets.
• [ ] Alert Gabriel and Aaron at multiple thresholds.
• [ ] Enable Cost Anomaly Detection.
• [ ] Review NAT Gateway, data-transfer, and logging costs.
• [ ] Set log-retention limits.
• [ ] Set backup-retention limits consistent with recovery requirements.
• [ ] Right-size production after load testing.
• [ ] Review idle resources.
• [ ] Document expected baseline monthly cost.
• [ ] Document the cost implications of autoscaling.
• [ ] Configure service quotas and limits deliberately.
• [ ] Review production cost weekly during the first month.
• [ ] Add owner and purpose tags to every billable resource.
• [ ] Alert on untagged or unexpectedly expensive resources where practical.

---

16. Documentation and Runbooks

• [ ] Create a system architecture diagram.
• [ ] Create a data-flow diagram.
• [ ] Document every AWS account and region.
• [ ] Document DNS and certificate ownership.
• [ ] Document environment differences.
• [ ] Document the IaC repository structure.
• [ ] Document local development setup.
• [ ] Document staging deployment.
• [ ] Document production deployment.
• [ ] Document rollback.
• [ ] Document database migration procedures.
• [ ] Document database restoration.
• [ ] Document secret rotation.
• [ ] Document certificate and DNS troubleshooting.
• [ ] Document user-access provisioning and removal.
• [ ] Document break-glass access.
• [ ] Document incident response.
• [ ] Document backup retention and restoration tests.
• [ ] Document common alarms and remediation.
• [ ] Create a service inventory with owners.
• [ ] Create a dependency inventory.
• [ ] Create an operational decision log.
• [ ] Record accepted risks and deferred work.
• [ ] Keep documentation versioned beside the relevant code where practical.
• [ ] Confirm Aaron can follow the documentation without Gabriel’s help.
• [ ] Confirm Gabriel can follow it without relying on personal memory.
• [ ] Add a production launch checklist and rollback decision tree.
• [ ] Add contact information and escalation paths.
• [ ] Review documentation after every incident or major deployment.

---

17. Testing Before Launch

• [ ] Run the complete automated test suite.
• [ ] Run production-like integration tests.
• [ ] Test authentication and authorization.
• [ ] Test a clean infrastructure deployment in a disposable environment.
• [ ] Test migration from the current schema.
• [ ] Test application rollback.
• [ ] Test failed migration handling.
• [ ] Test a database restore.
• [ ] Test backup integrity.
• [ ] Test task or instance failure.
• [ ] Test deployment during active traffic.
• [ ] Test autoscaling if enabled.
• [ ] Run basic load and soak tests.
• [ ] Verify timeout behavior.
• [ ] Verify retry behavior.
• [ ] Verify idempotency for jobs and webhooks.
• [ ] Test DNS and TLS from external networks.
• [ ] Test both root and www.
• [ ] Verify canonical redirects.
• [ ] Test monitoring and alert delivery.
• [ ] Run a tabletop incident exercise with Gabriel and Aaron.
• [ ] Test the break-glass process.
• [ ] Verify neither maintainer is a single point of operational failure.
• [ ] Test production access from a clean workstation.
• [ ] Test what happens when a secret, database, queue, or external API is unavailable.

---

18. Cutover Plan

• [ ] Freeze unrelated infrastructure changes during cutover.
• [ ] Confirm the production environment is healthy.
• [ ] Confirm production secrets are installed.
• [ ] Confirm migrations have been rehearsed.
• [ ] Confirm current backups succeeded.
• [ ] Confirm a restoration has been tested.
• [ ] Confirm dashboards and alerts are active.
• [ ] Confirm Aaron and Gabriel are both available for the cutover.
• [ ] Lower DNS TTL in advance.
• [ ] Deploy the approved production artifact.
• [ ] Run migrations.
• [ ] Run internal smoke tests.
• [ ] Update DNS.
• [ ] Verify:
  • [ ] DNS resolution
  • [ ] TLS
  • [ ] Root domain
  • [ ] www
  • [ ] API
  • [ ] Authentication
  • [ ] Critical user paths
  • [ ] Background workers
  • [ ] Outbound email
  • [ ] Object uploads and downloads
• [ ] Monitor logs, error rate, and latency closely.
• [ ] Keep the previous environment available for rollback during the agreed window.
• [ ] Execute rollback if predetermined thresholds are crossed.
• [ ] Raise DNS TTL after stability is confirmed.
• [ ] Record the launch artifact and configuration versions.
• [ ] Announce launch status to stakeholders.
• [ ] Record all manual actions taken during cutover.
• [ ] Confirm production backups run successfully after cutover.

---

19. Post-Launch

• [ ] Review production metrics after the first hour.
• [ ] Review again after 24 hours.
• [ ] Review again after one week.
• [ ] Inspect AWS costs after 24 hours and one week.
• [ ] Review security findings.
• [ ] Confirm backups continue to succeed.
• [ ] Perform another restore test after production has real data.
• [ ] Remove temporary cutover access and resources.
• [ ] Remove obsolete staging-derived credentials.
• [ ] Resolve or formally record every launch exception.
• [ ] Conduct a blameless launch retrospective.
• [ ] Update documentation with lessons learned.
• [ ] Schedule:
  • [ ] Monthly dependency updates
  • [ ] Monthly access review during the early production period
  • [ ] Quarterly disaster-recovery exercise
  • [ ] Quarterly permissions review
  • [ ] Quarterly cost review
  • [ ] Annual domain and recovery-contact review

---

Recommended Implementation Order

P0 — Before Creating Production

• [ ] Inventory staging.
• [ ] Establish shared organizational ownership and MFA.
• [ ] Decide AWS account boundaries.
• [ ] Select and implement Infrastructure as Code.
• [ ] Define RPO, RTO, and domain topology.
• [ ] Decide the canonical hostname and DNS provider.

P1 — Build Production

• [ ] Provision the production account and environment through IaC.
• [ ] Configure identity, networking, encryption, and secrets.
• [ ] Configure the database, automated backups, and restore testing.
• [ ] Configure GitHub OIDC and separate deployment roles.
• [ ] Build staging and production pipelines.
• [ ] Add logs, dashboards, alarms, and audit trails.
• [ ] Complete operational documentation.

P2 — Launch

• [ ] Configure certificates and DNS.
• [ ] Rehearse migrations, deployment, and rollback.
• [ ] Run security and load tests.
• [ ] Complete the cutover.
• [ ] Validate production and document the outcome.

---

Suggested Ownership Split

Workstream	Primary	Reviewer / Backup	Status
Staging architecture inventory	Gabriel	Claude Code	In progress — see Section 0
AWS Organizations and accounts			Not started
IAM Identity Center and permissions			Not started
Infrastructure as Code			Not started
Networking and production architecture			Not started
Namecheap, DNS, and TLS			Not started
Database and backups			Not started
GitHub CI/CD			Not started
Logging, metrics, and alerts			Not started
Security review			Not started
Documentation and runbooks			Not started
Cutover coordination			Not started

---

Decision Log

Date	Decision	Rationale	Owner
2026-08-04	Accept single AWS account (992382692655, shared with Stori) as technical debt rather than splitting into AWS Organizations before launch	Keeps focus on getting production launched; account split can happen later without blocking	Gabriel
2026-08-04	Architecture target (harden current EC2/Docker-Compose vs. redesign to ECS+ALB+RDS) left open	Insufficient information at Section 0 time; revisit once inventory and RPO/RTO are settled	Gabriel
2026-08-04	IaC tool selection deferred; Terraform/OpenTofu recommended given the actual footprint (2 EC2, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params)	Small, EC2-centric footprint doesn't need CDK's language ceremony or CloudFormation's verbosity	Gabriel
2026-08-04	Every "GitHub" checklist item is translated to its MuseHub equivalent (proposals for PRs, MuseHub branch protection, musehub-runner CI, MuseHub collaborator roles)	Absolute ecosystem policy — git/GitHub are never the VCS, CI, or collaboration platform for this project; GitHub is a passive backup mirror only	Gabriel (policy), applied throughout by Claude Code
2026-08-08	Section 6's "Namecheap vs. Route 53" question reframed as "Cloudflare vs. Route 53"	Live dig NS musehub.ai confirms DNS is already hosted at Cloudflare (registrar-only at Namecheap); docs/infrastructure.md was stale on this. Recommend staying on Cloudflare DNS given it also does edge proxy/TLS/possibly R2	Claude Code (finding), decision pending Gabriel

---

Accepted Risks and Deferred Work

⚠️ Highest-priority item in this table: there is currently no usable database backup, verified live on staging (no cron job, no rclone, one stale 20-byte manual dump). If the production database is lost today, there is no recovery path. This should not wait for the rest of the checklist sweep to be addressed.

Item	Risk	Mitigation	Target Date	Owner
No usable database backup exists (verified live: no cron, no rclone, one stale 20-byte manual dump from ~2 months ago)	Total, unrecoverable data loss on any DB failure, corruption, or operator error	Wire deploy/backup.sh in for real: install rclone, configure R2 remote, install the cron job, verify a real non-empty backup lands, then perform an actual restore test	ASAP — before any other Section 8 work	Gabriel
Single Postgres role used for app runtime, migrations, and admin access	An app-level SQL injection or bug would have migration/admin-level DB privileges, not just runtime privileges	Split into separate roles (app runtime, migrations, read-only, admin) per the checklist's explicit ask	Before Section 8 sign-off	Gabriel
Migration failure-handling (alembic stamp --purge head on upgrade failure) could mask a genuinely broken migration as a stale-revision-ID issue	A broken migration could silently ship if the follow-up schema-parity gate ever has a false negative	Document the reliance on schema_gate explicitly; consider distinguishing failure causes rather than blanket stamp-and-retry	Before Section 8 sign-off	Gabriel
MuseHub-native branch protection, required status checks, and CODEOWNERS-equivalent don't exist as product features (Sections 9-11 assumed they just needed configuring, like GitHub Actions would)	Sections 9-11 require building MuseHub platform capability, not just deployment config — bigger scope than the checklist implies	Scope this explicitly as product work before estimating Sections 9-11's timeline; prioritize required-status-check gating on merge_proposal first since everything else depends on it	Before Section 9 build phase begins	Gabriel
Single AWS account shared with Stori	Blast radius: a MuseHub compromise/misconfig can affect Stori resources and vice versa; billing and ownership entangled	Revisit AWS Organizations split post-launch; musehub-infra IAM user already scoped narrowly (verified via access-denied tests on IAM/CloudTrail/GuardDuty/Route53/etc.)	TBD	Gabriel
docs/infrastructure.md is stale — object storage is actually Cloudflare R2, not a local Docker volume as documented	Anyone reading the doc (including Aaron) gets the wrong mental model of where pushed content lives	Correct docs/infrastructure.md directly (separate from this checklist effort); resolved here via live proof this session	Soon	Gabriel
Secrets are hand-provisioned per instance, not via the drafted SSM pipeline (deploy/secrets.sh) — no rotation, no audit trail beyond manual memory	Rotation and auditability (Section 7's actual goals) aren't met by the current hand-provisioned .env approach	Gabriel to decide whether production adopts the SSM pipeline from day one rather than repeating staging's hand-provisioning	Before Section 7 sign-off	Gabriel
No RPO/RTO, traffic, or availability targets defined	Can't size backup cadence, DR posture, or capacity correctly without them	Define before Section 5 (architecture) and Section 8 (backups) are finalized	Before P1 build phase	Gabriel
Canonical hostname (root vs. www) and DNS host (Namecheap vs. Route 53) undecided	Blocks finalizing Section 6 and the cutover plan	Decide before Section 18 cutover	Before cutover	Gabriel
Deploys use a single shared AWS credential (musehub-infra in ~/.aws/credentials), not individual identities	Violates "no shared AWS access keys"; no accountability for who ran what; single point of failure if Gabriel is unavailable	Resolve via Section 3 (IAM Identity Center + per-person roles) before Section 1's independent-operation test can pass	Before Section 1 sign-off	Gabriel
Aaron has no AWS identity or verified Namecheap access today	Aaron cannot operate independently — fails the checklist's explicit "test that Aaron can independently…" requirement	Create Aaron's AWS identity via Section 3; confirm/grant Namecheap access	Before Section 1 sign-off	Gabriel