gabriel / musehub public
musehub-production-readiness-checklist.md markdown
872 lines 68.0 KB
Raw
sha256:7d5985ef251de9f0154f9b185a75cf36174bf73e0c34f5eca133e7bd20224bf4 Merge branch 'feat/opengraph-repo-cards' into dev Human 23 days ago

MuseHub Production-Readiness Checklist

Master checklist for promoting MuseHub from staging to production, automating deployments, sharing operational ownership across the team (Gabriel, Aaron, and interns Sol/Toby/Ben), and applying security, reliability, backup, documentation, and infrastructure best practices.

Team access tiers (decided 2026-08-08, see production-readiness/runbook-deploy-onboarding.md): Gabriel and Aaron have full staging + production access; interns (Sol, Toby, Ben) are staging-only by design. Interim: the current shared AWS credential is not distributed to interns — real staging access for them waits on Section 3's individual-identity rollout, which is now scoped for all 5 people from the start.

Working method: sections are swept in order, once each. Unresolved items are flagged inline and rolled up into the Decision Log / Accepted Risks tables at the bottom rather than blocking progress on later sections. See docs/production-readiness/ for the per-section companion docs with full verification detail.

0. Decisions and Inventory

Full detail: production-readiness/00-inventory-and-decisions.md

  • [ ] Document the current staging architecture.
    • [x] AWS account and region — 992382692655, us-east-1 (verified live)
    • [x] Compute platform: ECS, EC2, App Runner, Elastic Beanstalk, Lambda, EKS, etc. — raw EC2 (t3.small/t3.medium), Docker Compose, no managed compute service
    • [x] Load balancer, API Gateway, CloudFront, or other ingress — none; Cloudflare proxy → nginx on the instance
    • [x] Database engine and topology — self-hosted PostgreSQL 16, single instance, no RDS, no replica
    • [x] Redis, queues, object storage, search, background workers — background worker (musehub-runner) confirmed; object storage confirmed live on Cloudflare R2 (bucket musehub-staging, proven by a successful push this session — docs/infrastructure.md corrected); no Redis/search identified
    • [x] VPC, subnets, route tables, NAT gateways, and security groups — default VPC, no custom subnets, no NAT gateway, musehub-sg verified live (443 from Cloudflare ranges only)
    • [x] Secrets and configuration sources — hand-provisioned .env per instance (not SSM-automated); confirmed it includes working R2 credentials docs/infrastructure.md didn't document. Open question for Gabriel: should production adopt the drafted SSM pipeline (deploy/secrets.sh) from day one instead of hand-provisioning like staging?
    • [x] Logging and monitoring — corrected (see Section 12): structured JSON logs are centralized live in CloudWatch (/musehub/staging, ~1.7GB, no retention limit set); alarms/SNS/dashboards on top of them do not exist
    • [x] Current deployment process — documented: manual, human-triggered deploy/push.sh, no CI/CD automation
  • [x] Inventory every manually created AWS resource. — 2 EC2 + 2 EIP, 1 SG, 1 IAM user, 1 IAM instance role, 1 ECR repo, 1 S3 bucket, 3 SSM params (all listed in companion doc)
  • [ ] Inventory every external dependency.
    • [x] GitHub — backup-mirror only, never VCS/CI/collaboration (ecosystem policy)
    • [x] Namecheap — registrar + DNS host for musehub.ai
    • [x] Email provider — confirmed none exists
    • [x] OAuth providers — confirmed none exist (MSign/Ed25519 only)
    • [x] Payment or billing providers — confirmed none exist
    • [x] Error tracking — confirmed none exists
    • [ ] Third-party APIs — Cloudflare R2/Worker usage inferred from scripts but not independently verified against the Cloudflare dashboard
  • [x] Identify all stateful and mission-critical data. — Postgres DB, object store, musehub-releases S3 bucket, SSM params (all listed in companion doc)
  • [ ] Define expected initial traffic and reasonable growth assumptions. — not yet defined, needs Gabriel
  • [ ] Define availability expectations. — not yet defined, needs Gabriel
  • [ ] Define acceptable downtime during deployment. — not yet defined, needs Gabriel
  • [ ] Define recovery objectives.
    • [ ] RPO: maximum acceptable data loss — not yet defined, needs Gabriel
    • [ ] RTO: maximum acceptable restoration time — not yet defined, needs Gabriel
  • [ ] Decide the canonical public URLs.
    • [x] Root (musehub.ai) — exists, live
    • [x] www.musehub.ai — exists, live
    • [ ] API hostname — none exists; MuseHub serves API + web from the same origin, no api. subdomain in use today
    • [x] Staging (staging.musehub.ai) — exists, live
  • [ ] Choose either the root domain or www as canonical. — not yet decided, needs Gabriel
  • [ ] Redirect the noncanonical hostname permanently to the canonical one. — no redirect logic found in nginx config; not implemented
  • [ ] Decide whether DNS remains hosted at Namecheap or moves to Route 53. — not yet decided, needs Gabriel
  • [x] Decide whether production and non-production will use separate AWS accounts. — Decided: single account accepted as technical debt for now (see Decision Log)

1. Ownership and Shared Access

No shared human accounts, shared passwords, or shared AWS access keys. Full detail: production-readiness/01-ownership-and-access.md. "GitHub organization" items below are the MuseHub-native equivalent per the Decision Log.

  • [ ] Confirm the domain is held by the company rather than an individual wherever practical. — needs Gabriel (Namecheap console)
  • [ ] Confirm AWS account ownership and billing contacts. — needs Gabriel (AWS console)
  • [x] Confirm MuseHub repo ownership/collaboration model. — gabriel owns both repos; aaronrene has write collaborator access on musehub and muse (verified via muse hub collaborator list)
  • [ ] Add both Gabriel and Aaron as appropriate organizational owners or administrators. — done for MuseHub repos (Aaron has write); not done for AWS/Namecheap — needs Gabriel
  • [ ] Ensure neither person is the sole recovery path for:
    • [ ] Namecheap — needs Gabriel
    • [ ] AWS — needs Gabriel
    • [x] MuseHub — Aaron already has write access independent of Gabriel
    • [ ] DNS — needs Gabriel
    • [ ] Database — needs Gabriel (currently only reachable via musehub-infra shared AWS credential)
    • [ ] Production secrets — needs Gabriel
    • [ ] Billing — needs Gabriel
    • [ ] Backups — needs Gabriel (also blocked on resolving Section 0 Contradiction 2)
  • [ ] Use company-controlled email addresses for infrastructure accounts. — needs Gabriel
  • [ ] Configure at least two trusted recovery paths. — needs Gabriel
  • [ ] Store emergency recovery documentation in a location accessible to both founders. — needs Gabriel
  • [ ] Create a documented joiner–mover–leaver access procedure. — not started
  • [ ] Create a quarterly access-review process. — not started
  • [ ] Record ownership for every operational area:
    • [ ] AWS administration — currently Gabriel only, undocumented
    • [ ] DNS and certificates — currently Gabriel only, undocumented
    • [ ] Deployments — currently Gabriel only (shared musehub-infra credential), undocumented
    • [ ] Database administration — currently Gabriel only, undocumented
    • [ ] Incident response — no process exists
    • [ ] Billing — currently Gabriel only, undocumented
    • [ ] Backups — no automated backups exist yet (Section 0 Contradiction 2)
  • [ ] Create an emergency break-glass procedure. — not started
  • [ ] Test that Aaron can independently:
    • [ ] Sign in to AWS — cannot test — no AWS identity exists for Aaron yet
    • [ ] Inspect production — cannot test
    • [ ] Deploy staging — cannot test as designed — today's deploy uses a shared AWS credential (musehub-infra), not an individual one; this violates the "no shared AWS access keys" principle this section opens with
    • [ ] Approve or deploy production — cannot test — prod deploy is currently broken regardless (Section 0: no IAM instance profile on the prod instance)
    • [ ] Inspect logs and alarms — cannot test — no logs/alarms exist yet (Section 12)
    • [ ] Restore from backup — cannot test — no backups exist yet (Section 0 Contradiction 2)
    • [ ] Update DNS when authorized — cannot test — Aaron's Namecheap access unverified
  • [ ] Test that Gabriel can perform the same tasks without relying on undocumented knowledge. — not started; current process is entirely undocumented outside docs/infrastructure.md

2. AWS Account Structure

Full detail: production-readiness/02-aws-account-structure.md

  • [ ] Create or verify an AWS Organization. — deferred, see Decision Log (single account accepted as technical debt)
  • [ ] Use distinct AWS accounts for:
    • [ ] Management — deferred
    • [ ] Production — deferred
    • [ ] Non-production — deferred
  • [ ] Consider separate security and log-archive accounts as the operation grows. — deferred
  • [ ] Place production and non-production into separate organizational units. — deferred (no Organization exists)
  • [ ] Apply Service Control Policies only after testing them carefully. — N/A while deferred
  • [ ] Centralize billing. — N/A, single account
  • [ ] Enable account-level budget alerts. — not done; tracked under Section 15 (Cost Controls) instead since it applies within the single account regardless of structure — needs Gabriel (AWS console access required)
  • [ ] Set alternate security, operations, and billing contacts on every account. — needs Gabriel
  • [ ] Restrict enabled AWS regions to the regions MuseHub actually uses, where feasible. — needs Gabriel; note the account is shared with Stori (region us-east-2 bucket observed), so any region restriction must account for both projects
  • [x] Document any temporary decision to keep staging and production in one AWS account as accepted technical debt. — done, see Decision Log and Accepted Risks in this checklist

3. Identity, Authentication, and Permissions

Full detail: production-readiness/03-identity-and-permissions.md Document-only pass — no IAM changes made. Real build happens after the full sweep.

Root and Recovery Access

  • [ ] Protect every AWS root account with phishing-resistant MFA where possible. — needs Gabriel (AWS console)
  • [ ] Do not create root access keys. — needs Gabriel to confirm
  • [ ] Do not use root for routine administration. — needs Gabriel to confirm
  • [ ] Secure the root email account with MFA. — needs Gabriel
  • [ ] Store root recovery information securely. — needs Gabriel
  • [ ] Document the legitimate uses of root. — not started
  • [ ] Configure multi-person approval for emergency root use if practical. — not started

Human Access

  • [ ] Configure AWS IAM Identity Center. — not started; visibility blocked (sso-admin denied) — needs Gabriel to check/build
  • [ ] Create separate identities for Gabriel and Aaron. — not started (blocks Section 1's independent-operation test)
  • [ ] Require MFA for both. — not started
  • [ ] Prefer hardware security keys or passkeys. — not started
  • [x] Define permission sets, for example:
    • [x] Administrator — decided: Gabriel
    • [x] ProductionOperator — decided: Aaron (staging + production)
    • [x] StagingOperator — decided: Sol, Toby, Ben (interns — staging-only, no production access; see runbook-deploy-onboarding.md)
    • [ ] ReadOnly — not started
    • [ ] Billing — not started
  • [ ] Give routine access through the least-privileged applicable role. — not started
  • [ ] Require explicit elevation for administrative work. — not started
  • [ ] Avoid permanent AdministratorAccess for everyday use where practical. — not started
  • [ ] Use IAM Access Analyzer to inspect and refine permissions. — not started; needs broader read access
  • [ ] Review unused roles, users, policies, and credentials. — not started; needs broader read access
  • [ ] Document who can approve and perform production changes. — not started

Workload Access

  • [x] Give every application or service its own IAM role. — 3 separate IAM users confirmed live (musehub-infra, musehub-app, stori-assets-app), no shared workload credential across services
  • [x] Do not place AWS access keys in source code. — confirmed via grep, none found
  • [x] Do not place long-lived AWS access keys in GitHub secrets. — N/A, no GitHub Actions/secrets used (see Decision Log); equivalent check for MuseHub-runner CI config deferred to Section 9
  • [ ] Restrict every deployment role to:
    • [ ] The correct repository — N/A, no deployment role exists yet (deploys use a human + shared credential)
    • [ ] The correct branch, tag, or GitHub environment — N/A, same reason
    • [ ] The correct AWS account — N/A, same reason
    • [ ] The required AWS resources only — partially true of musehub-infra's scoping, but it's a human-shared credential, not a deployment role
  • [ ] Separate staging and production deployment roles. — not done — both environments share the musehub-infra credential
  • [ ] Validate trust policies and permission boundaries. — not started; needs broader read access
  • [ ] Prefer temporary credentials everywhere. — not done — all current credentials are long-lived IAM user access keys

4. Infrastructure as Code

Full detail: production-readiness/04-iac.md Document-only pass. No IaC written yet — deferred to the build phase after the full sweep.

  • [ ] Select one Infrastructure-as-Code system:
    • [ ] Terraform/OpenTofu — recommended, not yet formally chosen by Gabriel
    • [ ] AWS CDK
    • [ ] CloudFormation
  • [ ] Capture the existing staging infrastructure in code. — not started; full resource list is in the Section 0 doc
  • [ ] Stop making undocumented console-only changes. — behavioral change, not yet in effect
  • [ ] Build reusable modules rather than duplicating entire configurations. — N/A until IaC exists
  • [ ] Provide separate environment configuration for:
    • [ ] Staging — N/A until IaC exists
    • [ ] Production — N/A until IaC exists
  • [ ] Keep production and staging state isolated. — N/A until IaC exists
  • [ ] Encrypt remote IaC state. — N/A until IaC exists
  • [ ] Enable versioning and locking for remote state. — N/A until IaC exists
  • [ ] Restrict access to IaC state because it may contain sensitive metadata. — N/A until IaC exists
  • [ ] Pin provider, module, and action versions. — N/A until IaC exists
  • [ ] Add formatting and validation checks. — N/A until IaC exists
  • [ ] Add static security scanning. — N/A until IaC exists
  • [ ] Require a reviewed plan before production changes. — N/A until IaC exists
  • [ ] Detect configuration drift. — N/A until IaC exists
  • [ ] Document how to import existing resources into IaC. — not started; note this must be import, not recreate, since real resources already exist and are live
  • [ ] Add environment-standard tags:
    • [ ] Application — not applied anywhere currently
    • [ ] Environment — not applied anywhere currently
    • [ ] Owner — not applied anywhere currently
    • [ ] Managed-by — not applied anywhere currently
    • [ ] Cost-center — not applied anywhere currently
    • [ ] Data-classification — not applied anywhere currently
  • [ ] Make a fresh environment reproducible from source control and documented prerequisites. — not started
  • [ ] Add a teardown guard so production cannot be accidentally destroyed. — N/A until IaC exists
  • [ ] Add deletion protection to critical resources. — not started (also relevant to Section 8 — DB deletion protection specifically)
  • [ ] Add policy checks that prevent production resources from being publicly exposed unintentionally. — N/A until IaC exists
  • [ ] Ensure IaC changes are reviewed by someone other than the author when practical. — N/A until IaC exists

5. Production Architecture

Full detail: production-readiness/05-production-architecture.md

  • [ ] Reassess whether staging's sizing and topology are production-appropriate. — flagged: staging is t3.medium, prod is t3.small — inconsistent, no documented reason
  • [x] Do not copy staging data into production unless explicitly sanitized and required. — not done, and correctly not done — no evidence of staging→prod data copying anywhere
  • [ ] Use separate production:
    • [x] Database — separate instance entirely (different EC2 host)
    • [ ] Credentials — unresolved pending Section 0 Contradiction 2
    • [ ] Secrets — unresolved pending Section 0 Contradiction 2
    • [ ] Object-storage buckets — unresolved pending Section 0 Contradiction 1
    • [ ] Queues — none exist in either environment
    • [ ] Cache — none exist in either environment
    • [ ] Logs — not separated in any meaningful way; no centralized logging exists yet (Section 12)
    • [ ] Encryption keys where appropriate — not reviewed yet
  • [ ] Deploy production across at least two Availability Zones where supported and justified. — not done; single instance, no ASG, documented as intentional SPOF below
  • [x] Keep databases and internal services in private subnets. — partially true in spirit (DB only reachable via the app's Docker network, not the public internet) but not true architecturally (default VPC, no real private subnet); security group is the actual control here
  • [x] Expose only the public ingress layer. — confirmed: only 443 open, restricted to Cloudflare IP ranges
  • [x] Eliminate direct public database access. — confirmed, DB has no public exposure
  • [x] Use security groups by service relationship rather than broad CIDR ranges. — reasonable substitution given the architecture: scoped to Cloudflare's published IP ranges, not 0.0.0.0/0
  • [x] Prohibit unrestricted administrative ports such as public SSH. — confirmed, no port 22 open
  • [x] Use AWS Systems Manager Session Manager rather than public SSH where applicable. — confirmed, this is already the only access path
  • [x] Configure health checks. — confirmed: /healthz probes both DB and object storage, gates blue/green flips
  • [ ] Configure graceful application shutdown. — not verified — no SIGTERM handling found in musehub/main.py; uvicorn defaults apply
  • [ ] Configure connection draining. — partial: blue/green waits for new slot health before flip, but old slot's in-flight requests during docker rm aren't verified graceful
  • [x] Set sensible application and load-balancer timeouts. — confirmed: deliberate per-route nginx timeouts (5s health, 60s default, up to 3600s for streaming/MCP)
  • [ ] Configure autoscaling or document why fixed capacity is appropriate. — neither done; fixed capacity with no written justification
  • [ ] Set resource requests, limits, or instance sizes deliberately. — instance types chosen but not justified with load data
  • [ ] Verify production can survive a routine instance or task replacement. — not verified; single-instance architecture means instance failure has no automatic failover (documented SPOF below)
  • [x] Confirm background jobs are idempotent or safely retryable. — confirmed: claim_next_job uses SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent claiming, reclaim_stale_jobs resets crashed-worker jobs back to pending (see Section 9 doc)
  • [ ] Confirm database migrations are compatible with rolling deployments. — pipeline runs migrations before slot flip (right shape); expand/contract discipline not reviewed — deferred to Section 8
  • [x] Add rate limiting and abuse controls. — confirmed: slowapi-based per-route limits + global baseline + bot-throttle middleware, already implemented
  • [ ] Consider AWS WAF for public endpoints. — N/A as literally stated (no ALB/CloudFront); Cloudflare is the substitute layer, but its WAF/rate-limit config is unverified — needs Gabriel to check Cloudflare dashboard
  • [x] Configure DDoS protections appropriate to the architecture. — Cloudflare edge proxy already provides this; not independently verified beyond "Cloudflare is in front of everything"
  • [ ] Establish AWS service quota monitoring for relevant resources. — not started
  • [x] Document every intentional single point of failure. — done, see companion doc: single EC2 per environment, single Postgres instance, single AWS account (shared with Stori), single Cloudflare account, single human deployer with shared credential
  • [ ] Create a plan to remove unacceptable single points of failure. — needs Gabriel — depends on RPO/RTO/availability targets that are still undefined (Section 0)

6. DNS, Domain, and TLS

Full detail: production-readiness/06-dns-domain-tls.md ⚠️ Finding: DNS is actually hosted at Cloudflare, not Namecheap — verified via live dig NS musehub.ai. docs/infrastructure.md is stale on this point. Namecheap is registrar only. This changes several items below from "Namecheap vs Route 53" to "Cloudflare vs Route 53."

Namecheap Account

  • [ ] Enable MFA on Namecheap for every authorized administrator. — needs Gabriel
  • [ ] Use unique individual access rather than shared credentials where the account model permits. — needs Gabriel
  • [ ] Enable domain lock. — needs Gabriel
  • [ ] Verify registrant and recovery contact details. — needs Gabriel
  • [ ] Enable automatic renewal. — needs Gabriel
  • [ ] Verify the payment method will not unexpectedly expire. — needs Gabriel
  • [ ] Document who may change nameservers or DNS. — needs Gabriel; note DNS itself is now on Cloudflare, so this also means documenting Cloudflare access, not just Namecheap
  • [ ] Export or document all existing DNS records before making changes. — needs Gabriel, from the Cloudflare dashboard, not Namecheap's Advanced DNS tab
  • [ ] Preserve mail-related MX, SPF, DKIM, and DMARC records. — needs Gabriel to confirm these exist and are preserved in Cloudflare's DNS
  • [ ] Confirm both Gabriel and Aaron have an approved recovery path. — needs Gabriel, for both Namecheap and Cloudflare

DNS Design

  • [ ] Decide between:
    • [x] Namecheap-hosted DNS — not the current state; superseded by the finding above
    • [ ] Route 53-hosted DNS with Namecheap remaining registrar — open; recommendation below
  • [x] Prefer Route 53... — recommendation revised: current DNS host is Cloudflare (not Namecheap), and Cloudflare is also doing edge proxy/TLS/possibly R2 — recommend staying on Cloudflare DNS rather than fragmenting the stack by moving to Route 53, unless Gabriel has a specific reason to want AWS-native alias records
  • [ ] Create a Route 53 hosted zone if selected. — N/A per above recommendation, pending Gabriel's decision
  • [x] Reproduce every required existing record before switching nameservers. — N/A, no switch planned per recommendation
  • [ ] Lower DNS TTL ahead of cutover. — deferred to Section 18 (Cutover Plan)
  • [x] Configure the root/apex record. — exists, live, resolves via Cloudflare
  • [x] Configure www. — exists, live
  • [ ] Configure api if needed. — no api. hostname exists; MuseHub serves API + web from one origin — confirm this is intentional
  • [x] Keep staging on a clearly separate hostname. — confirmed, staging.musehub.ai
  • [x] Do not expose internal AWS service hostnames unnecessarily. — confirmed, only the public hostnames resolve; no raw AWS DNS names exposed
  • [ ] Verify IPv4 and IPv6 behavior. — not tested in this pass
  • [ ] Configure a canonical-host redirect. — not implemented — no redirect logic found; root and www both resolve without a canonical redirect
  • [ ] Test DNS from multiple resolvers after cutover. — deferred to Section 18
  • [ ] Raise TTL again after the deployment is stable. — deferred to Section 18
  • [ ] Manage DNS records through IaC where practical. — deferred to Section 4 build phase (Terraform's cloudflare provider)

TLS

  • [ ] Request an AWS Certificate Manager certificate. — likely N/A for this architecture — TLS terminates at Cloudflare's edge, not an AWS ALB/CloudFront; needs Gabriel to confirm there's no AWS-facing TLS need I'm missing
  • [ ] Include every required hostname. — N/A pending above
  • [ ] Prefer DNS validation. — N/A pending above
  • [ ] Retain ACM validation records for automatic renewal. — N/A pending above
  • [x] Redirect all HTTP traffic to HTTPS. — Cloudflare/nginx setup already enforces HTTPS-only (port 80 not open at all)
  • [x] Use modern TLS policies. — Cloudflare-managed edge TLS; verified live cert is current (Let's Encrypt, issued 2026-08-01)
  • [x] Configure HSTS only after HTTPS and subdomain behavior are verified. — already live: Strict-Transport-Security with includeSubDomains confirmed in nginx config
  • [ ] Confirm certificate-expiration monitoring. — not done — no alerting exists yet (Section 12); manual verification only (as done earlier this session)
  • [x] Test the full certificate chain. — verified via openssl s_client this session; valid, correctly issued, correct CN
  • [ ] Confirm cookies use:
    • [x] N/A — MuseHub uses header-based MSign auth, no cookies exist anywhere in the app

7. Secrets and Application Configuration

Full detail: production-readiness/07-secrets-and-config.md

  • [x] Inventory all secrets. — done, 6 secrets identified (DB password, webhook key, runner token, R2 credentials, worker key, worker URL)
  • [x] Remove secrets from source control and repository history. — confirmed via grep, none found
  • [ ] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate. — the real work item: deploy/secrets.sh implements this but only 3/7 required SSM params exist; not live yet
  • [ ] Encrypt secrets with KMS. — partial; the 3 existing SSM params are SecureString/KMS-encrypted, rest aren't in SSM
  • [x] Use separate staging and production secrets. — confirmed: separate .env, separate R2 buckets, separate SSM prefixes
  • [ ] Give workloads access only to the secrets they require. — likely N/A for a single-process-per-instance app; not force-fit
  • [x] Do not expose production secrets to pull-request workflows. — N/A, no PR-based CI exists
  • [ ] Do not print secrets in logs. — not verified; no centralized logs exist yet to audit (Section 12)
  • [ ] Configure secret rotation where supported. — not done
  • [ ] Rotate credentials during the production launch. — deferred to Section 18
  • [ ] Document emergency rotation procedures. — not done
  • [x] Separate secrets from nonsecret configuration. — confirmed: musehub/config.py's Settings class already separates these cleanly in code
  • [x] Validate required configuration at application startup. — confirmed via pydantic_settings.BaseSettings
  • [x] Fail safely when critical configuration is missing. — confirmed, same mechanism plus secrets.sh's own preflight die checks
  • [x] Remove default credentials and sample keys. — confirmed, no placeholder secrets in config.py; secrets.sh also actively rejects known-weak passwords
  • [ ] Audit OAuth callback URLs and API allowlists for production. — N/A, no OAuth exists
  • [ ] Audit every external webhook secret. — WORKER_INTERNAL_KEY identified; validation logic in musehub/worker.py not independently reviewed in this pass
  • [ ] Document ownership and rotation frequency for every production secret. — not done, blocked on Section 1

8. Database Security, Migrations, and Backups

Full detail: production-readiness/08-database-backups-migrations.md ⚠️ Verified live via SSM: there is currently no usable backup of the database, anywhere.

Database Security

  • [ ] Enable encryption at rest. — not verified in this pass
  • [ ] Enforce encryption in transit. — not verified in this pass
  • [ ] Use a production-specific KMS key where appropriate. — not started
  • [ ] Store credentials in Secrets Manager. — see Section 7
  • [ ] Avoid administrator credentials for normal application traffic. — not done — single musehub role used for app traffic, migrations, and admin access
  • [ ] Create separate roles for:
    • [ ] Application runtime — not done
    • [ ] Migrations — not done
    • [ ] Read-only operations — not done
    • [ ] Administration — not done
  • [x] Keep the database off the public internet. — confirmed
  • [x] Restrict security-group access to the application layer. — confirmed
  • [ ] Enable deletion protection. — not applicable in the RDS sense (self-hosted); EBS/snapshot-level protection not configured
  • [ ] Enable database-level audit logging where appropriate. — not enabled
  • [ ] Monitor storage, CPU, memory, connections, locks, and slow queries. — partial: app-level slow-query logging exists (slow_query_threshold_ms in musehub/config.py); no infrastructure-level DB monitoring exists

Backups and Recovery

  • [ ] Enable automated backups. — not done, verified live: no crontab installed, rclone not present, one stale 20-byte manual backup from ~2 months ago is the only thing that exists
  • [ ] Set an explicit retention period. — N/A, no backups running
  • [ ] Enable point-in-time recovery where supported. — not available for self-hosted Postgres without WAL archiving, which isn't configured
  • [ ] Configure scheduled longer-term snapshots. — not done
  • [ ] Configure cross-account backup copies for critical production data. — not done
  • [ ] Consider cross-region copies based on the agreed disaster model. — not done
  • [ ] Protect backup vaults from routine production credentials. — N/A, no backups exist
  • [ ] Encrypt backup recovery points. — N/A, no backups exist
  • [ ] Enable backup-vault lock or equivalent immutability where justified. — N/A, no backups exist
  • [ ] Monitor backup-job failures. — N/A, no backup job exists
  • [ ] Define retention and deletion policies. — not done
  • [ ] Document the restoration procedure. — not done
  • [ ] Perform an actual restore test before launch. — cannot happen until backups exist
  • [ ] Schedule recurring restore tests. — N/A yet
  • [ ] Measure restoration time against the RTO. — N/A, RTO undefined (Section 0) and no backups exist
  • [ ] Confirm restored data satisfies the RPO. — N/A, same reasons
  • [ ] Record the latest successful restore test. — N/A
  • [ ] Verify backups are restorable by both Gabriel and Aaron. — N/A
  • [ ] Back up any other mission-critical state, including object storage and configuration data. — object storage is content-addressed R2 (durable by design, per Cloudflare); .env configuration data has no backup of its own beyond what's in this documentation

Schema Migrations

  • [x] Run migrations automatically but as an explicit deployment step. — confirmed, part of deploy.sh, before slot swap
  • [ ] Ensure only one migration runner executes at a time. — not independently verified; no explicit lock mechanism found
  • [ ] Back up before high-risk migrations. — moot until backups exist
  • [ ] Prefer expand-and-contract migrations. — not reviewed across the 75 existing migrations in this pass
  • [ ] Avoid combining destructive schema changes with dependent application changes in one irreversible step. — not reviewed
  • [ ] Test migrations against a production-like dataset. — not verified
  • [ ] Define rollback or forward-repair procedures. — not documented
  • [x] Record migration state and duration. — implicitly via Alembic's alembic_version table and deploy logs; no structured/queryable record beyond that
  • [ ] Alert on migration failure. — not done, no alerting exists (Section 12)
  • [x] Ensure older and newer application versions can coexist during rolling or blue/green deployments where required. — pipeline ordering (migrate-then-swap) is the right shape; ⚠️ found a real risk: on alembic upgrade head failure, deploy.sh runs alembic stamp --purge head and retries, which could mask a genuinely broken migration as a "stale revision ID" — mitigated by a hard-failing schema-parity gate afterward, but worth explicit awareness since that gate is the only thing standing between this pattern and a silently-broken deploy

9. CI Pipeline

Full detail: production-readiness/09-ci-pipeline.md ⚠️ Bigger finding than a typical checklist item: MuseHub-native equivalents of branch protection, required status checks, and CODEOWNERS don't exist as product features yet — this section requires building platform capability, not just configuring deployment. What does exist and is genuinely good: a concurrency-safe internal background job queue.

  • [ ] Require proposals (MuseHub's PR-equivalent) for protected branches. — not started, feature doesn't exist
  • [ ] Protect main/dev. — not started
  • [ ] Require CI checks before merging. — not started; merge_proposal has no CI-status gate
  • [ ] Require review from another maintainer for sensitive changes. — not started
  • [ ] Define required-reviewer paths (CODEOWNERS-equivalent) for:
    • [ ] Infrastructure — not started
    • [ ] Deployment workflows — not started
    • [ ] Authentication and security — not started
    • [ ] Database migrations — not started
  • [ ] Run:
    • [ ] Formatting — manual only, not automated on push
    • [ ] Linting — manual only
    • [ ] Type checking — manual only
    • [ ] Unit tests — manual only
    • [ ] Integration tests — manual only
    • [ ] Migration checks — runs at deploy time (schema_gate, see Section 8), not at CI/proposal time
    • [ ] Dependency vulnerability scanning — not automated
    • [ ] Secret scanning — not automated
    • [ ] Container or artifact scanning — not automated
    • [ ] IaC validation and security scanning — N/A, no IaC exists yet (Section 4)
  • [ ] Generate immutable, versioned build artifacts. — partial: ECR image tags exist and are immutable once pushed, but built manually per-environment rather than automatically per-commit
  • [x] Record the Git commit SHA in every artifact. — N/A concept (no git); MuseHub commit IDs are content-addressed (sha256:...) and could be embedded in image tags — not currently done, worth adding
  • [ ] Produce the artifact once and promote the same artifact through environments. — not done by default; IMAGE_TAG redeploy support in push.sh makes promotion possible but isn't the default flow
  • [ ] Generate an SBOM where practical. — not started
  • [x] Pin third-party GitHub Actions to immutable commit SHAs for sensitive workflows. — N/A, no GitHub Actions used
  • [x] Minimize GitHub Actions token permissions. — N/A, no GitHub Actions used
  • [ ] Avoid executing untrusted pull-request code with production-capable credentials. — moot until proposal-triggered CI execution exists at all
  • [ ] Configure dependency update automation. — not started
  • [ ] Define an urgent-patch workflow. — not documented
  • [ ] Retain CI and deployment logs for an explicit period. — N/A, no CI exists to log yet
  • [ ] Prevent force-pushes and branch deletion on protected branches. — not started

10. Automated Staging Deployment

Full detail: production-readiness/10-11-automated-deployment.md Deploys today are 100% manual (bash deploy/push.sh staging, human-triggered, shared credential).

  • [ ] Create a MuseHub-native staging-environment concept (GitHub environment equivalent). — doesn't exist
  • [ ] Deploy automatically after successful merges. — not started, no trigger mechanism exists
  • [ ] Authenticate to AWS through short-lived/OIDC-equivalent credentials. — not started
  • [x] Do not store long-lived AWS keys in CI config. — moot today (no CI config exists); the actual problem is the shared laptop credential (Section 1/3), not a CI secret leak
  • [ ] Restrict the staging role to the staging AWS account and resources. — not started (Section 3)
  • [ ] Run infrastructure planning. — N/A, no IaC yet (Section 4)
  • [ ] Apply approved or policy-compliant infrastructure changes. — N/A, same reason
  • [x] Deploy the immutable application artifact. — already true today, manually
  • [x] Run database migrations. — already true, automatic within the deploy
  • [ ] Run smoke tests. — not automated as a gate; deploy/smoke_muse.sh exists but its coverage wasn't reviewed in this pass
  • [x] Run health checks. — already true, gates the blue/green flip
  • [ ] Publish the deployment URL and commit SHA. — not automated
  • [ ] Notify both maintainers of success or failure. — not automated
  • [ ] Preserve deployment logs. — not done; currently only in the deployer's terminal
  • [ ] Automatically roll back application deployment when health checks fail, where safe. — not automated, but effectively moot: the health-gated blue/green means a failing new slot never goes live in the first place, rather than going live and needing rollback
  • [ ] Keep database rollback separate and explicit. — not formalized
  • [ ] Prevent concurrent staging deployments from colliding. — not enforced
  • [ ] Make the deployment status visible in MuseHub. — not built
  • [ ] Record artifact digest, migration version, deployer, and timestamp. — not recorded anywhere structured

11. Automated Production Deployment

Full detail: production-readiness/10-11-automated-deployment.md ⚠️ Reminder: prod deploy is currently broken regardless of automation — no IAM instance profile on the prod instance (Section 0), so push.sh prod fails outright even run manually today.

  • [ ] Create a MuseHub-native production-environment concept. — doesn't exist
  • [ ] Use a distinct production AWS role. — not started (Section 3)
  • [ ] Restrict that role to the production account and resources. — not started
  • [ ] Trigger production deployment through an explicit release action:
    • [ ] Version tag — MuseHub has muse release add/muse hub release create already; not wired to deploy
    • [ ] MuseHub release — same as above
    • [ ] Manual promotion of a verified artifact — possible via IMAGE_TAG=<tag>, not the default flow
  • [ ] Do not rebuild the application for production. — not done by default today
  • [ ] Promote the exact artifact tested in staging. — possible, not automatic
  • [ ] Require successful staging deployment and tests first. — not enforced
  • [ ] Require production approval. — not built
  • [ ] Prevent self-approval when practical. — not built
  • [ ] Permit both Gabriel and Aaron to approve. — not built; also blocked since Aaron has no independent deploy identity yet (Section 1/3)
  • [ ] Require approval from someone other than the person initiating the deployment when practical. — not built
  • [ ] Display the complete deployment plan before approval. — not built
  • [ ] Back up stateful systems before risky changes. — currently impossible to do meaningfully — no backups exist (Section 8)
  • [x] Run migrations as a controlled step. — already true
  • [ ] Deploy using:
    • [ ] Rolling deployment — N/A, using blue/green instead
    • [x] Blue/green deployment — already true, this is the current strategy
    • [ ] Canary deployment — N/A, not the chosen strategy
  • [ ] Run post-deployment smoke tests. — not automated
  • [ ] Monitor error rate, latency, and health during rollout. — not possible yet, no monitoring exists (Section 12)
  • [ ] Stop or roll back when thresholds are exceeded. — N/A, no thresholds/monitoring exist
  • [ ] Record who approved and initiated each deployment. — not recorded
  • [ ] Record commit SHA, artifact identifier, and migration version. — not recorded
  • [ ] Generate release notes. — not automated
  • [ ] Notify both operators after completion. — not automated
  • [ ] Prevent concurrent production deployments. — not enforced
  • [x] Add a documented manual deployment path for emergencies. — exists (it's the current default path)
  • [x] Add a documented rollback command or workflow. — exists (IMAGE_TAG=<previous-tag> bash deploy/push.sh prod)
  • [ ] Test rollback before launch. — not yet done, explicitly required

12. Observability

Full detail: production-readiness/12-observability.md ⚠️ Correction to Section 0/5: logging is further along than earlier stated in this sweep — structured JSON logs ARE centralized live in CloudWatch (/musehub/staging, ~1.7GB, verified via aws logs tail). It's specifically alarms/dashboards/metrics that don't exist, not logging itself.

  • [x] Centralize application logs. — confirmed live, CloudWatch Logs /musehub/staging
  • [x] Use structured logs. — confirmed, JSON with rich per-request fields (musehub/logging_config.py)
  • [ ] Include:
    • [x] Timestamp — present
    • [ ] Environment — missing from the JSON schema
    • [ ] Service — implicit via logger field, not an explicit service field
    • [x] Request or correlation ID — present (request_id)
    • [ ] Release version — missing from the JSON schema
    • [x] Severity — present (level)
  • [x] Prevent credentials, tokens, and sensitive personal data from entering logs. — confirmed, PiiFilter scrubs Bearer tokens/passwords/secrets already
  • [ ] Set explicit log-retention periods. — not done, currently unlimited retention on the live log group
  • [ ] Encrypt logs. — CloudWatch default at-rest encryption applies; customer-managed KMS key not confirmed
  • [ ] Collect infrastructure and application metrics. — partial: app-level data flows into logs (could feed metric filters); true instance-level metrics (CPU/disk/memory via CloudWatch Agent) not confirmed installed
  • [ ] Build dashboards for:
    • [ ] Request rate — data exists in logs, no dashboard
    • [ ] Error rate — data exists in logs, no dashboard
    • [ ] Latency — data exists in logs (duration_ms), no dashboard
    • [ ] Saturation — not collected
    • [ ] Task or instance health — /healthz exists but isn't dashboarded
    • [ ] Database capacity — not collected
    • [ ] Database connections — not collected
    • [ ] Queue depth — the internal job queue (Section 9) has no depth metric exposed
    • [ ] Background-job failures — job status is tracked in the DB (attempt, status) but not surfaced to any dashboard
  • [ ] Add external uptime checks. — not done
  • [ ] Configure alerts for:
    • [ ] Site unavailable — not done
    • [ ] Elevated 5xx responses — not done (data available, not wired)
    • [ ] Elevated latency — not done (data available, not wired)
    • [ ] Unhealthy deployment — not done
    • [ ] Database storage pressure — not done
    • [ ] CPU or memory exhaustion — partial signal exists ([memory] HIGH RSS app-level warnings already fire), not wired to an actual alert
    • [ ] Backup failure — N/A, no backup job exists yet (Section 8)
    • [ ] Certificate problems — not done
    • [ ] Queue backlog — not done
    • [ ] Security findings — not done
    • [ ] Unexpected cost growth — not done (Section 15)
  • [ ] Route alerts to both Gabriel and Aaron. — N/A, no alerts exist
  • [ ] Define warning versus page-worthy alert severity. — not done
  • [ ] Test alert delivery. — N/A
  • [ ] Ensure every urgent alert links to a runbook. — N/A
  • [ ] Configure error tracking with release correlation. — not done, no tool exists
  • [ ] Define service-level indicators and initial service objectives. — not done, blocked on Section 0's undefined availability targets
  • [ ] Add deployment markers to dashboards. — N/A, no dashboards exist
  • [ ] Verify logs and metrics remain available during an application outage. — not tested, but likely true by construction (CloudWatch Logs is a separate AWS service from the EC2 instance)

13. Security Monitoring and Auditability

Full detail: production-readiness/13-security-monitoring.md Most items blocked on the broader-read-access grant Gabriel agreed to earlier this session — re-run once that lands.

  • [ ] Enable CloudTrail. — blocked, cannot confirm current state
  • [ ] Use a multi-region trail for management activity. — blocked
  • [ ] Include read and write management events. — blocked
  • [ ] Encrypt CloudTrail logs. — blocked
  • [ ] Enable log-file validation. — blocked
  • [ ] Store logs in a nonpublic, access-restricted bucket. — blocked
  • [ ] Send relevant CloudTrail events to CloudWatch Logs. — blocked
  • [ ] Enable AWS Config where cost and scope permit. — blocked
  • [ ] Enable GuardDuty. — blocked
  • [ ] Enable Security Hub and AWS Foundational Security Best Practices controls. — blocked
  • [ ] Enable IAM Access Analyzer. — blocked
  • [ ] Review public and cross-account resource access. — blocked
  • [ ] Enable S3 Block Public Access at the account level unless explicitly incompatible. — checked one bucket directly: musehub-releases has all 4 settings disabled — likely intentional (serves public /install.sh), needs Gabriel's explicit sign-off; account-level setting itself is blocked
  • [x] Enable ECR or image vulnerability scanning if containers are used. — confirmed NOT enabled — imageScanStatus is null on the latest image; easy, low-risk fix for the build phase
  • [ ] Monitor changes to IAM, KMS, CloudTrail, DNS, and security groups. — N/A until CloudTrail exists
  • [ ] Alert on root-account activity. — N/A until CloudTrail/GuardDuty exist
  • [ ] Alert on failed console logins and suspicious authentication. — N/A, same reason
  • [ ] Define a vulnerability-remediation SLA. — not done; can't be meaningful before scanning exists
  • [ ] Run a production security review before launch. — this checklist effort is a step toward it; a dedicated pass still needed closer to launch
  • [ ] Record and assign every security finding. — no tracking process exists yet
  • [ ] Review high-severity findings before every production launch. — N/A, no findings exist yet

14. Application Security

Full detail: production-readiness/14-application-security.md ✅ Fixed: /_debug/memory was exposed unconditionally in production (missing the if settings.debug: guard /docs//redoc correctly have) — patched this session.

  • [ ] Document the authentication and authorization model. — MSign is well-implemented in code but not written up as a standalone doc yet
  • [ ] Test authorization independently of UI visibility. — not verified in this pass
  • [x] Enforce server-side access control. — confirmed via MSign fresh-signature requirement on privileged MCP tool calls
  • [x] Use secure password hashing if MuseHub stores passwords. — N/A, no passwords exist (MSign is signature-based)
  • [ ] Require email verification where relevant. — N/A, no email-based flows exist
  • [x] Implement secure account recovery. — N/A in the traditional sense; equivalent is Ed25519 mnemonic recovery, already documented at the ecosystem level (muse auth recover)
  • [x] Rate-limit login and recovery endpoints. — confirmed, slowapi-based limits + bot throttling (Section 5)
  • [ ] Protect against CSRF where applicable. — not explicitly reviewed; likely lower risk given no cookie-based sessions, not confirmed for every state-changing endpoint
  • [x] Validate and sanitize untrusted input. — Pydantic models validate at the API boundary throughout
  • [x] Use parameterized database queries. — confirmed, no raw SQL string formatting found anywhere
  • [x] Restrict CORS to deliberate origins. — confirmed, fails closed (cors_origins: [] default) with a wildcard-in-production warning
  • [x] Define a Content Security Policy. — confirmed, set in musehub/main.py
  • [x] Add standard security headers. — confirmed: X-Frame-Options, X-Content-Type-Options, HSTS, CSP, Referrer-Policy
  • [x] Validate uploaded file type, size, and storage behavior. — confirmed: mpack_max_bytes/mpack_max_commits/mpack_max_objects and per-user/per-repo/daily quotas
  • [x] Protect against path traversal and unsafe archive extraction. — confirmed: traversal-char rejection in coordination module, mpack_max_decompressed_bytes zip-bomb quarantine
  • [ ] Scan uploaded content if the risk model warrants it. — not done beyond structural/size limits
  • [x] Set request-body and upload limits. — confirmed, see above
  • [ ] Review SSRF exposure from URL-fetching features. — no SSRF-prone code found in main request handling; musehub/worker.py's Cloudflare Worker calls specifically not yet reviewed
  • [x] Ensure production debug mode is disabled. — confirmed, debug: bool = False default, and /docs//redoc correctly gated behind it
  • [ ] Ensure stack traces are not returned to users. — not independently verified in this pass
  • [ ] Review WebSocket or SSE authentication if used. — MCP/SSE endpoints exist; auth model for the streaming case specifically not independently verified
  • [ ] Document data classification and retention. — not done
  • [ ] Add an account and data deletion path where applicable. — not verified whether one exists
  • [ ] Run an OWASP-oriented review before launch. — this pass covers several categories but isn't a formal pass; deferred to Section 17
  • [ ] Review session expiration and token revocation behavior. — N/A in the traditional sense (no sessions); MSign's per-request timestamp/replay-window logic not independently verified
  • [x] Verify privileged actions require fresh authorization where appropriate. — confirmed, MCP tool calls require a fresh MSign signature, not just a valid prior session

15. Cost Controls

Full detail: production-readiness/15-cost-controls.md

  • [ ] Apply consistent cost-allocation tags. — not done; confirmed inconsistent (prod has only Name, staging has Name+Environment, S3 bucket has none)
  • [ ] Create monthly AWS budgets. — blocked on broader access grant
  • [ ] Alert Gabriel and Aaron at multiple thresholds. — N/A until budgets exist
  • [ ] Enable Cost Anomaly Detection. — blocked
  • [ ] Review NAT Gateway, data-transfer, and logging costs. — N/A for NAT (none exists); logging cost is real and growing (Section 12's unlimited-retention log group)
  • [x] Set log-retention limits. — tracked under Section 12, cross-referenced here
  • [ ] Set backup-retention limits consistent with recovery requirements. — N/A, no backups exist yet
  • [ ] Right-size production after load testing. — not done, no load testing has occurred
  • [ ] Review idle resources. — staging appears to run continuously despite docs/infrastructure.md suggesting it be stopped when idle — worth Gabriel deciding whether to actually do that
  • [x] Document expected baseline monthly cost. — already done, docs/infrastructure.md (~$35/mo for both environments)
  • [ ] Document the cost implications of autoscaling. — N/A, no autoscaling exists
  • [ ] Configure service quotas and limits deliberately. — not done
  • [ ] Review production cost weekly during the first month. — deferred to Section 19
  • [ ] Add owner and purpose tags to every billable resource. — not done, see tagging findings
  • [ ] Alert on untagged or unexpectedly expensive resources where practical. — not done

16. Documentation and Runbooks

Full detail: production-readiness/16-19-docs-testing-cutover-postlaunch.md

  • [ ] Create a system architecture diagram. — not created
  • [ ] Create a data-flow diagram. — not created
  • [x] Document every AWS account and region. — done, Section 0
  • [x] Document DNS and certificate ownership. — done, Section 6
  • [ ] Document environment differences. — scattered, not consolidated
  • [ ] Document the IaC repository structure. — N/A, no IaC exists yet
  • [ ] Document local development setup. — not reviewed in this pass
  • [x] Document staging deployment. — done, docs/infrastructure.md, plus a team-onboarding runbook: production-readiness/runbook-deploy-onboarding.md
  • [x] Document production deployment. — done, docs/infrastructure.md (though prod is currently broken)
  • [x] Document rollback. — documented, not yet tested
  • [ ] Document database migration procedures. — partial; rollback/forward-repair not documented
  • [ ] Document database restoration. — cannot be meaningful yet, no backups exist
  • [ ] Document secret rotation. — not done
  • [ ] Document certificate and DNS troubleshooting. — not done
  • [ ] Document user-access provisioning and removal. — not done
  • [ ] Document break-glass access. — not done
  • [ ] Document incident response. — not done
  • [ ] Document backup retention and restoration tests. — N/A, no backups exist
  • [ ] Document common alarms and remediation. — N/A, no alarms exist
  • [ ] Create a service inventory with owners. — partial (inventory yes, owners no)
  • [x] Create a dependency inventory. — done, Section 0
  • [x] Create an operational decision log. — done, this checklist's Decision Log
  • [x] Record accepted risks and deferred work. — done, this checklist's Accepted Risks table
  • [x] Keep documentation versioned beside the relevant code where practical. — done by construction
  • [ ] Confirm Aaron can follow the documentation without Gabriel's help. — not tested
  • [ ] Confirm Gabriel can follow it without relying on personal memory. — not tested
  • [ ] Add a production launch checklist and rollback decision tree. — Section 18 is the launch checklist; decision tree specifically doesn't exist
  • [ ] Add contact information and escalation paths. — not done
  • [ ] Review documentation after every incident or major deployment. — process doesn't exist yet

17. Testing Before Launch

Full detail: production-readiness/16-19-docs-testing-cutover-postlaunch.md

  • [ ] Run the complete automated test suite. — verified the suite exists and collects cleanly: 7,028 tests, zero collection errors; actually running it pass/fail is real Section 17 work, not done in this doc-sweep pass
  • [ ] Run production-like integration tests. — not verified
  • [ ] Test authentication and authorization. — likely covered somewhere in the 7,028 tests; not confirmed
  • [ ] Test a clean infrastructure deployment in a disposable environment. — N/A, no IaC exists yet
  • [ ] Test migration from the current schema. — not done
  • [ ] Test application rollback. — not done, explicitly required
  • [ ] Test failed migration handling. — not done
  • [ ] Test a database restore. — cannot happen, no backups exist (Section 8)
  • [ ] Test backup integrity. — same blocker
  • [ ] Test task or instance failure. — not done
  • [ ] Test deployment during active traffic. — not done
  • [ ] Test autoscaling if enabled. — N/A, none exists
  • [ ] Run basic load and soak tests. — not done
  • [ ] Verify timeout behavior. — deliberate timeouts confirmed by code review (Section 5), not load-tested
  • [ ] Verify retry behavior. — not load-tested
  • [ ] Verify idempotency for jobs and webhooks. — confirmed by code review for background jobs (Section 9); webhooks not reviewed
  • [ ] Test DNS and TLS from external networks. — partially done this session (live cert verification); not a full external sweep
  • [ ] Test both root and www. — both confirmed resolving; not a full test
  • [ ] Verify canonical redirects. — cannot pass, no canonical redirect exists yet (Section 6)
  • [ ] Test monitoring and alert delivery. — N/A, no alerts exist yet
  • [ ] Run a tabletop incident exercise with Gabriel and Aaron. — not done
  • [ ] Test the break-glass process. — N/A, doesn't exist yet
  • [ ] Verify neither maintainer is a single point of operational failure. — currently fails, per Section 1 (shared credential, no Aaron AWS identity)
  • [ ] Test production access from a clean workstation. — not done
  • [ ] Test what happens when a secret, database, queue, or external API is unavailable. — not done

18. Cutover Plan

Full detail: production-readiness/16-19-docs-testing-cutover-postlaunch.md Entirely future work — every item here depends on Sections 0–15 being substantially complete (backups existing, RPO/RTO defined, monitoring live, Aaron having independent access). Not walking each item individually since none are meaningfully startable yet. One prerequisite worth repeating: prod deploy is currently broken (no IAM instance profile, Section 0) — nothing in this section can happen until that's fixed, independent of everything else.

  • [ ] Freeze unrelated infrastructure changes during cutover.
  • [ ] Confirm the production environment is healthy.
  • [ ] Confirm production secrets are installed.
  • [ ] Confirm migrations have been rehearsed.
  • [ ] Confirm current backups succeeded.
  • [ ] Confirm a restoration has been tested.
  • [ ] Confirm dashboards and alerts are active.
  • [ ] Confirm Aaron and Gabriel are both available for the cutover.
  • [ ] Lower DNS TTL in advance.
  • [ ] Deploy the approved production artifact.
  • [ ] Run migrations.
  • [ ] Run internal smoke tests.
  • [ ] Update DNS.
  • [ ] Verify:
    • [ ] DNS resolution
    • [ ] TLS
    • [ ] Root domain
    • [ ] www
    • [ ] API
    • [ ] Authentication
    • [ ] Critical user paths
    • [ ] Background workers
    • [ ] Outbound email
    • [ ] Object uploads and downloads
  • [ ] Monitor logs, error rate, and latency closely.
  • [ ] Keep the previous environment available for rollback during the agreed window.
  • [ ] Execute rollback if predetermined thresholds are crossed.
  • [ ] Raise DNS TTL after stability is confirmed.
  • [ ] Record the launch artifact and configuration versions.
  • [ ] Announce launch status to stakeholders.
  • [ ] Record all manual actions taken during cutover.
  • [ ] Confirm production backups run successfully after cutover.

19. Post-Launch

Full detail: production-readiness/16-19-docs-testing-cutover-postlaunch.md Entirely contingent on launch having happened — not walking each item individually. The recurring-schedule items are worth setting up as real calendar reminders once launch is real; several (e.g. quarterly DR exercise) are only meaningful once their underlying capability exists (backups + restore procedure, for that example).

  • [ ] Review production metrics after the first hour.
  • [ ] Review again after 24 hours.
  • [ ] Review again after one week.
  • [ ] Inspect AWS costs after 24 hours and one week.
  • [ ] Review security findings.
  • [ ] Confirm backups continue to succeed.
  • [ ] Perform another restore test after production has real data.
  • [ ] Remove temporary cutover access and resources.
  • [ ] Remove obsolete staging-derived credentials.
  • [ ] Resolve or formally record every launch exception.
  • [ ] Conduct a blameless launch retrospective.
  • [ ] Update documentation with lessons learned.
  • [ ] Schedule:
    • [ ] Monthly dependency updates
    • [ ] Monthly access review during the early production period
    • [ ] Quarterly disaster-recovery exercise
    • [ ] Quarterly permissions review
    • [ ] Quarterly cost review
    • [ ] Annual domain and recovery-contact review

Recommended Implementation Order

P0 — Before Creating Production

  • [ ] Inventory staging.
  • [ ] Establish shared organizational ownership and MFA.
  • [ ] Decide AWS account boundaries.
  • [ ] Select and implement Infrastructure as Code.
  • [ ] Define RPO, RTO, and domain topology.
  • [ ] Decide the canonical hostname and DNS provider.

P1 — Build Production

  • [ ] Provision the production account and environment through IaC.
  • [ ] Configure identity, networking, encryption, and secrets.
  • [ ] Configure the database, automated backups, and restore testing.
  • [ ] Configure GitHub OIDC and separate deployment roles.
  • [ ] Build staging and production pipelines.
  • [ ] Add logs, dashboards, alarms, and audit trails.
  • [ ] Complete operational documentation.

P2 — Launch

  • [ ] Configure certificates and DNS.
  • [ ] Rehearse migrations, deployment, and rollback.
  • [ ] Run security and load tests.
  • [ ] Complete the cutover.
  • [ ] Validate production and document the outcome.

Suggested Ownership Split

Workstream Primary Reviewer / Backup Status
Staging architecture inventory Gabriel Claude Code Done — Section 0
AWS Organizations and accounts Gabriel Decided (deferred as tech debt) — Section 2
IAM Identity Center and permissions Gabriel Documented gap, not built — Section 3
Infrastructure as Code Gabriel Tool recommended, not built — Section 4
Networking and production architecture Gabriel Documented, SPOFs identified — Section 5
Namecheap, DNS, and TLS Gabriel Documented, Cloudflare-hosting correction made — Section 6
Database and backups Gabriel ⚠️ Documented — no usable backup exists, highest-priority risk — Section 8
MuseHub-native CI/CD Gabriel Documented — platform features (branch protection, required checks) don't exist yet — Sections 9-11
Logging, metrics, and alerts Gabriel Logging is live and solid; alerts/dashboards don't exist — Section 12
Security review Gabriel Partial — one live finding fixed-pending (/_debug/memory); account-wide checks blocked on access grant — Sections 13-14
Documentation and runbooks Gabriel Claude Code This sweep is the start of it — Section 16
Cutover coordination Correctly blocked until earlier sections close — Sections 18-19

Decision Log

Date Decision Rationale Owner
2026-08-04 Accept single AWS account (992382692655, shared with Stori) as technical debt rather than splitting into AWS Organizations before launch Keeps focus on getting production launched; account split can happen later without blocking Gabriel
2026-08-04 Architecture target (harden current EC2/Docker-Compose vs. redesign to ECS+ALB+RDS) left open Insufficient information at Section 0 time; revisit once inventory and RPO/RTO are settled Gabriel
2026-08-04 IaC tool selection deferred; Terraform/OpenTofu recommended given the actual footprint (2 EC2, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params) Small, EC2-centric footprint doesn't need CDK's language ceremony or CloudFormation's verbosity Gabriel
2026-08-04 Every "GitHub" checklist item is translated to its MuseHub equivalent (proposals for PRs, MuseHub branch protection, musehub-runner CI, MuseHub collaborator roles) Absolute ecosystem policy — git/GitHub are never the VCS, CI, or collaboration platform for this project; GitHub is a passive backup mirror only Gabriel (policy), applied throughout by Claude Code
2026-08-08 Section 6's "Namecheap vs. Route 53" question reframed as "Cloudflare vs. Route 53" Live dig NS musehub.ai confirms DNS is already hosted at Cloudflare (registrar-only at Namecheap); docs/infrastructure.md was stale on this. Recommend staying on Cloudflare DNS given it also does edge proxy/TLS/possibly R2 Claude Code (finding), decision pending Gabriel
2026-08-08 Deploy team expanded from 2 (Gabriel, Aaron) to 5 (+ interns Sol, Toby, Ben); interns get staging-only access; Section 3's IAM Identity Center rollout is scoped for all 5 from the start Sharing one AWS credential 5 ways has no accountability and no environment separation; the current key should not be distributed to interns before individual identities exist Gabriel

Accepted Risks and Deferred Work

⚠️ Highest-priority item in this table: there is currently no usable database backup, verified live on staging (no cron job, no rclone, one stale 20-byte manual dump). If the production database is lost today, there is no recovery path. This should not wait for the rest of the checklist sweep to be addressed.

Item Risk Mitigation Target Date Owner
No usable database backup exists (verified live: no cron, no rclone, one stale 20-byte manual dump from ~2 months ago) Total, unrecoverable data loss on any DB failure, corruption, or operator error Wire deploy/backup.sh in for real: install rclone, configure R2 remote, install the cron job, verify a real non-empty backup lands, then perform an actual restore test ASAP — before any other Section 8 work Gabriel
Single Postgres role used for app runtime, migrations, and admin access An app-level SQL injection or bug would have migration/admin-level DB privileges, not just runtime privileges Split into separate roles (app runtime, migrations, read-only, admin) per the checklist's explicit ask Before Section 8 sign-off Gabriel
Migration failure-handling (alembic stamp --purge head on upgrade failure) could mask a genuinely broken migration as a stale-revision-ID issue A broken migration could silently ship if the follow-up schema-parity gate ever has a false negative Document the reliance on schema_gate explicitly; consider distinguishing failure causes rather than blanket stamp-and-retry Before Section 8 sign-off Gabriel
MuseHub-native branch protection, required status checks, and CODEOWNERS-equivalent don't exist as product features (Sections 9-11 assumed they just needed configuring, like GitHub Actions would) Sections 9-11 require building MuseHub platform capability, not just deployment config — bigger scope than the checklist implies Scope this explicitly as product work before estimating Sections 9-11's timeline; prioritize required-status-check gating on merge_proposal first since everything else depends on it Before Section 9 build phase begins Gabriel
/_debug/memory endpoint exposed unconditionally in production Minor info disclosure Fixed this session — guarded behind if settings.debug: like /docs//redoc; ships on next deploy Done 2026-08-08 Claude Code
Single AWS account shared with Stori Blast radius: a MuseHub compromise/misconfig can affect Stori resources and vice versa; billing and ownership entangled Revisit AWS Organizations split post-launch; musehub-infra IAM user already scoped narrowly (verified via access-denied tests on IAM/CloudTrail/GuardDuty/Route53/etc.) TBD Gabriel
docs/infrastructure.md is stale — object storage is actually Cloudflare R2, not a local Docker volume as documented Anyone reading the doc (including Aaron) gets the wrong mental model of where pushed content lives Correct docs/infrastructure.md directly (separate from this checklist effort); resolved here via live proof this session Soon Gabriel
Secrets are hand-provisioned per instance, not via the drafted SSM pipeline (deploy/secrets.sh) — no rotation, no audit trail beyond manual memory Rotation and auditability (Section 7's actual goals) aren't met by the current hand-provisioned .env approach Gabriel to decide whether production adopts the SSM pipeline from day one rather than repeating staging's hand-provisioning Before Section 7 sign-off Gabriel
No RPO/RTO, traffic, or availability targets defined Can't size backup cadence, DR posture, or capacity correctly without them Define before Section 5 (architecture) and Section 8 (backups) are finalized Before P1 build phase Gabriel
Canonical hostname (root vs. www) and DNS host (Namecheap vs. Route 53) undecided Blocks finalizing Section 6 and the cutover plan Decide before Section 18 cutover Before cutover Gabriel
Deploys use a single shared AWS credential (musehub-infra in ~/.aws/credentials), not individual identities Violates "no shared AWS access keys"; no accountability for who ran what; single point of failure if Gabriel is unavailable Resolve via Section 3 (IAM Identity Center + per-person roles) before Section 1's independent-operation test can pass Before Section 1 sign-off Gabriel
Aaron has no AWS identity or verified Namecheap access today Aaron cannot operate independently — fails the checklist's explicit "test that Aaron can independently…" requirement Create Aaron's AWS identity via Section 3; confirm/grant Namecheap access Before Section 1 sign-off Gabriel
File History 1 commit
sha256:cd5c2fcb91a44ac9e38e9c36176c27ce079a074586d9852296da628a29fb01ff Merge 'docs/aws-identity-and-deploy-fixes' into 'dev' — pro… Human 3 days ago