# MuseHub Production-Readiness Checklist

> Master checklist for promoting MuseHub from staging to production, automating deployments, sharing operational ownership between Gabriel and Aaron, and applying security, reliability, backup, documentation, and infrastructure best practices.

> **Working method:** sections are swept in order, once each. Unresolved items are flagged inline
> and rolled up into the Decision Log / Accepted Risks tables at the bottom rather than blocking
> progress on later sections. See `docs/production-readiness/` for the per-section companion docs
> with full verification detail.

## 0. Decisions and Inventory

> Full detail: [`production-readiness/00-inventory-and-decisions.md`](production-readiness/00-inventory-and-decisions.md)

- [ ] Document the current staging architecture.
  - [x] AWS account and region — `992382692655`, `us-east-1` (verified live)
  - [x] Compute platform: ECS, EC2, App Runner, Elastic Beanstalk, Lambda, EKS, etc. — raw EC2 (`t3.small`/`t3.medium`), Docker Compose, no managed compute service
  - [x] Load balancer, API Gateway, CloudFront, or other ingress — none; Cloudflare proxy → nginx on the instance
  - [x] Database engine and topology — self-hosted PostgreSQL 16, single instance, no RDS, no replica
  - [ ] Redis, queues, object storage, search, background workers — background worker (`musehub-runner`) confirmed; object storage medium **unresolved** (Contradiction 1 — local Docker volume vs. Cloudflare R2, needs Gabriel's confirmation); no Redis/search identified
  - [x] VPC, subnets, route tables, NAT gateways, and security groups — default VPC, no custom subnets, no NAT gateway, `musehub-sg` verified live (443 from Cloudflare ranges only)
  - [ ] Secrets and configuration sources — **unresolved** (Contradiction 2 — static `.env` vs. SSM Parameter Store pipeline, needs Gabriel's confirmation)
  - [x] Logging and monitoring — confirmed none live (no CloudWatch alarms, no SNS topics exist despite a script that would create them)
  - [x] Current deployment process — documented: manual, human-triggered `deploy/push.sh`, no CI/CD automation
- [x] Inventory every manually created AWS resource. — 2 EC2 + 2 EIP, 1 SG, 1 IAM user, 1 IAM instance role, 1 ECR repo, 1 S3 bucket, 3 SSM params (all listed in companion doc)
- [ ] Inventory every external dependency.
  - [x] GitHub — backup-mirror only, never VCS/CI/collaboration (ecosystem policy)
  - [x] Namecheap — registrar + DNS host for `musehub.ai`
  - [x] Email provider — confirmed none exists
  - [x] OAuth providers — confirmed none exist (MSign/Ed25519 only)
  - [x] Payment or billing providers — confirmed none exist
  - [x] Error tracking — confirmed none exists
  - [ ] Third-party APIs — Cloudflare R2/Worker usage inferred from scripts but not independently verified against the Cloudflare dashboard
- [x] Identify all stateful and mission-critical data. — Postgres DB, object store, `musehub-releases` S3 bucket, SSM params (all listed in companion doc)
- [ ] Define expected initial traffic and reasonable growth assumptions. — **not yet defined, needs Gabriel**
- [ ] Define availability expectations. — **not yet defined, needs Gabriel**
- [ ] Define acceptable downtime during deployment. — **not yet defined, needs Gabriel**
- [ ] Define recovery objectives.
  - [ ] **RPO:** maximum acceptable data loss — **not yet defined, needs Gabriel**
  - [ ] **RTO:** maximum acceptable restoration time — **not yet defined, needs Gabriel**
- [ ] Decide the canonical public URLs.
  - [x] Root (`musehub.ai`) — exists, live
  - [x] `www.musehub.ai` — exists, live
  - [ ] API hostname — none exists; MuseHub serves API + web from the same origin, no `api.` subdomain in use today
  - [x] Staging (`staging.musehub.ai`) — exists, live
- [ ] Choose either the root domain or `www` as canonical. — **not yet decided, needs Gabriel**
- [ ] Redirect the noncanonical hostname permanently to the canonical one. — no redirect logic found in nginx config; not implemented
- [ ] Decide whether DNS remains hosted at Namecheap or moves to Route 53. — **not yet decided, needs Gabriel**
- [x] Decide whether production and non-production will use separate AWS accounts. — **Decided:** single account accepted as technical debt for now (see Decision Log)

---

## 1. Ownership and Shared Access

> No shared human accounts, shared passwords, or shared AWS access keys.

- [ ] Confirm the domain is held by the company rather than an individual wherever practical.
- [ ] Confirm AWS account ownership and billing contacts.
- [ ] Confirm GitHub organization ownership.
- [ ] Add both Gabriel and Aaron as appropriate organizational owners or administrators.
- [ ] Ensure neither person is the sole recovery path for:
  - [ ] Namecheap
  - [ ] AWS
  - [ ] GitHub
  - [ ] DNS
  - [ ] Database
  - [ ] Production secrets
  - [ ] Billing
  - [ ] Backups
- [ ] Use company-controlled email addresses for infrastructure accounts.
- [ ] Configure at least two trusted recovery paths.
- [ ] Store emergency recovery documentation in a location accessible to both founders.
- [ ] Create a documented joiner–mover–leaver access procedure.
- [ ] Create a quarterly access-review process.
- [ ] Record ownership for every operational area:
  - [ ] AWS administration
  - [ ] DNS and certificates
  - [ ] Deployments
  - [ ] Database administration
  - [ ] Incident response
  - [ ] Billing
  - [ ] Backups
- [ ] Create an emergency break-glass procedure.
- [ ] Test that Aaron can independently:
  - [ ] Sign in to AWS
  - [ ] Inspect production
  - [ ] Deploy staging
  - [ ] Approve or deploy production
  - [ ] Inspect logs and alarms
  - [ ] Restore from backup
  - [ ] Update DNS when authorized
- [ ] Test that Gabriel can perform the same tasks without relying on undocumented knowledge.

---

## 2. AWS Account Structure

### Recommended Target

- [ ] Create or verify an AWS Organization.
- [ ] Use distinct AWS accounts for:
  - [ ] Management
  - [ ] Production
  - [ ] Non-production
- [ ] Consider separate security and log-archive accounts as the operation grows.
- [ ] Place production and non-production into separate organizational units.
- [ ] Apply Service Control Policies only after testing them carefully.
- [ ] Centralize billing.
- [ ] Enable account-level budget alerts.
- [ ] Set alternate security, operations, and billing contacts on every account.
- [ ] Restrict enabled AWS regions to the regions MuseHub actually uses, where feasible.
- [ ] Document any temporary decision to keep staging and production in one AWS account as accepted technical debt.

---

## 3. Identity, Authentication, and Permissions

### Root and Recovery Access

- [ ] Protect every AWS root account with phishing-resistant MFA where possible.
- [ ] Do not create root access keys.
- [ ] Do not use root for routine administration.
- [ ] Secure the root email account with MFA.
- [ ] Store root recovery information securely.
- [ ] Document the legitimate uses of root.
- [ ] Configure multi-person approval for emergency root use if practical.

### Human Access

- [ ] Configure AWS IAM Identity Center.
- [ ] Create separate identities for Gabriel and Aaron.
- [ ] Require MFA for both.
- [ ] Prefer hardware security keys or passkeys.
- [ ] Define permission sets, for example:
  - [ ] `Administrator`
  - [ ] `ProductionOperator`
  - [ ] `Developer`
  - [ ] `ReadOnly`
  - [ ] `Billing`
- [ ] Give routine access through the least-privileged applicable role.
- [ ] Require explicit elevation for administrative work.
- [ ] Avoid permanent `AdministratorAccess` for everyday use where practical.
- [ ] Use IAM Access Analyzer to inspect and refine permissions.
- [ ] Review unused roles, users, policies, and credentials.
- [ ] Document who can approve and perform production changes.

### Workload Access

- [ ] Give every application or service its own IAM role.
- [ ] Do not place AWS access keys in source code.
- [ ] Do not place long-lived AWS access keys in GitHub secrets.
- [ ] Restrict every deployment role to:
  - [ ] The correct repository
  - [ ] The correct branch, tag, or GitHub environment
  - [ ] The correct AWS account
  - [ ] The required AWS resources only
- [ ] Separate staging and production deployment roles.
- [ ] Validate trust policies and permission boundaries.
- [ ] Prefer temporary credentials everywhere.

---

## 4. Infrastructure as Code

- [ ] Select one Infrastructure-as-Code system:
  - [ ] Terraform/OpenTofu
  - [ ] AWS CDK
  - [ ] CloudFormation
- [ ] Capture the existing staging infrastructure in code.
- [ ] Stop making undocumented console-only changes.
- [ ] Build reusable modules rather than duplicating entire configurations.
- [ ] Provide separate environment configuration for:
  - [ ] Staging
  - [ ] Production
- [ ] Keep production and staging state isolated.
- [ ] Encrypt remote IaC state.
- [ ] Enable versioning and locking for remote state.
- [ ] Restrict access to IaC state because it may contain sensitive metadata.
- [ ] Pin provider, module, and action versions.
- [ ] Add formatting and validation checks.
- [ ] Add static security scanning.
- [ ] Require a reviewed plan before production changes.
- [ ] Detect configuration drift.
- [ ] Document how to import existing resources into IaC.
- [ ] Add environment-standard tags:
  - [ ] Application
  - [ ] Environment
  - [ ] Owner
  - [ ] Managed-by
  - [ ] Cost-center
  - [ ] Data-classification
- [ ] Make a fresh environment reproducible from source control and documented prerequisites.
- [ ] Add a teardown guard so production cannot be accidentally destroyed.
- [ ] Add deletion protection to critical resources.
- [ ] Add policy checks that prevent production resources from being publicly exposed unintentionally.
- [ ] Ensure IaC changes are reviewed by someone other than the author when practical.

---

## 5. Production Architecture

- [ ] Reassess whether staging’s sizing and topology are production-appropriate.
- [ ] Do not copy staging data into production unless explicitly sanitized and required.
- [ ] Use separate production:
  - [ ] Database
  - [ ] Credentials
  - [ ] Secrets
  - [ ] Object-storage buckets
  - [ ] Queues
  - [ ] Cache
  - [ ] Logs
  - [ ] Encryption keys where appropriate
- [ ] Deploy production across at least two Availability Zones where supported and justified.
- [ ] Keep databases and internal services in private subnets.
- [ ] Expose only the public ingress layer.
- [ ] Eliminate direct public database access.
- [ ] Use security groups by service relationship rather than broad CIDR ranges.
- [ ] Prohibit unrestricted administrative ports such as public SSH.
- [ ] Use AWS Systems Manager Session Manager rather than public SSH where applicable.
- [ ] Configure health checks.
- [ ] Configure graceful application shutdown.
- [ ] Configure connection draining.
- [ ] Set sensible application and load-balancer timeouts.
- [ ] Configure autoscaling or document why fixed capacity is appropriate.
- [ ] Set resource requests, limits, or instance sizes deliberately.
- [ ] Verify production can survive a routine instance or task replacement.
- [ ] Confirm background jobs are idempotent or safely retryable.
- [ ] Confirm database migrations are compatible with rolling deployments.
- [ ] Add rate limiting and abuse controls.
- [ ] Consider AWS WAF for public endpoints.
- [ ] Configure DDoS protections appropriate to the architecture.
- [ ] Establish AWS service quota monitoring for relevant resources.
- [ ] Document every intentional single point of failure.
- [ ] Create a plan to remove unacceptable single points of failure.

---

## 6. DNS, Domain, and TLS

### Namecheap Account

- [ ] Enable MFA on Namecheap for every authorized administrator.
- [ ] Use unique individual access rather than shared credentials where the account model permits.
- [ ] Enable domain lock.
- [ ] Verify registrant and recovery contact details.
- [ ] Enable automatic renewal.
- [ ] Verify the payment method will not unexpectedly expire.
- [ ] Document who may change nameservers or DNS.
- [ ] Export or document all existing DNS records before making changes.
- [ ] Preserve mail-related MX, SPF, DKIM, and DMARC records.
- [ ] Confirm both Gabriel and Aaron have an approved recovery path.

### DNS Design

- [ ] Decide between:
  - [ ] Namecheap-hosted DNS
  - [ ] Route 53-hosted DNS with Namecheap remaining registrar
- [ ] Prefer Route 53 when AWS-native alias records and IaC-managed DNS are valuable.
- [ ] Create a Route 53 hosted zone if selected.
- [ ] Reproduce every required existing record before switching nameservers.
- [ ] Lower DNS TTL ahead of cutover.
- [ ] Configure the root/apex record.
- [ ] Configure `www`.
- [ ] Configure `api` if needed.
- [ ] Keep staging on a clearly separate hostname.
- [ ] Do not expose internal AWS service hostnames unnecessarily.
- [ ] Verify IPv4 and IPv6 behavior.
- [ ] Configure a canonical-host redirect.
- [ ] Test DNS from multiple resolvers after cutover.
- [ ] Raise TTL again after the deployment is stable.
- [ ] Manage DNS records through IaC where practical.

### TLS

- [ ] Request an AWS Certificate Manager certificate.
- [ ] Include every required hostname.
- [ ] Prefer DNS validation.
- [ ] Retain ACM validation records for automatic renewal.
- [ ] Redirect all HTTP traffic to HTTPS.
- [ ] Use modern TLS policies.
- [ ] Configure HSTS only after HTTPS and subdomain behavior are verified.
- [ ] Confirm certificate-expiration monitoring.
- [ ] Test the full certificate chain.
- [ ] Confirm cookies use:
  - [ ] `Secure`
  - [ ] `HttpOnly`
  - [ ] Appropriate `SameSite` settings

---

## 7. Secrets and Application Configuration

- [ ] Inventory all secrets.
- [ ] Remove secrets from source control and repository history.
- [ ] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate.
- [ ] Encrypt secrets with KMS.
- [ ] Use separate staging and production secrets.
- [ ] Give workloads access only to the secrets they require.
- [ ] Do not expose production secrets to pull-request workflows.
- [ ] Do not print secrets in logs.
- [ ] Configure secret rotation where supported.
- [ ] Rotate credentials during the production launch.
- [ ] Document emergency rotation procedures.
- [ ] Separate secrets from nonsecret configuration.
- [ ] Validate required configuration at application startup.
- [ ] Fail safely when critical configuration is missing.
- [ ] Remove default credentials and sample keys.
- [ ] Audit OAuth callback URLs and API allowlists for production.
- [ ] Audit every external webhook secret.
- [ ] Document ownership and rotation frequency for every production secret.

---

## 8. Database Security, Migrations, and Backups

### Database Security

- [ ] Enable encryption at rest.
- [ ] Enforce encryption in transit.
- [ ] Use a production-specific KMS key where appropriate.
- [ ] Store credentials in Secrets Manager.
- [ ] Avoid administrator credentials for normal application traffic.
- [ ] Create separate roles for:
  - [ ] Application runtime
  - [ ] Migrations
  - [ ] Read-only operations
  - [ ] Administration
- [ ] Keep the database off the public internet.
- [ ] Restrict security-group access to the application layer.
- [ ] Enable deletion protection.
- [ ] Enable database-level audit logging where appropriate.
- [ ] Monitor storage, CPU, memory, connections, locks, and slow queries.

### Backups and Recovery

- [ ] Enable automated backups.
- [ ] Set an explicit retention period.
- [ ] Enable point-in-time recovery where supported.
- [ ] Configure scheduled longer-term snapshots.
- [ ] Configure cross-account backup copies for critical production data.
- [ ] Consider cross-region copies based on the agreed disaster model.
- [ ] Protect backup vaults from routine production credentials.
- [ ] Encrypt backup recovery points.
- [ ] Enable backup-vault lock or equivalent immutability where justified.
- [ ] Monitor backup-job failures.
- [ ] Define retention and deletion policies.
- [ ] Document the restoration procedure.
- [ ] Perform an actual restore test before launch.
- [ ] Schedule recurring restore tests.
- [ ] Measure restoration time against the RTO.
- [ ] Confirm restored data satisfies the RPO.
- [ ] Record the latest successful restore test.
- [ ] Verify backups are restorable by both Gabriel and Aaron.
- [ ] Back up any other mission-critical state, including object storage and configuration data.

### Schema Migrations

- [ ] Run migrations automatically but as an explicit deployment step.
- [ ] Ensure only one migration runner executes at a time.
- [ ] Back up before high-risk migrations.
- [ ] Prefer expand-and-contract migrations.
- [ ] Avoid combining destructive schema changes with dependent application changes in one irreversible step.
- [ ] Test migrations against a production-like dataset.
- [ ] Define rollback or forward-repair procedures.
- [ ] Record migration state and duration.
- [ ] Alert on migration failure.
- [ ] Ensure older and newer application versions can coexist during rolling or blue/green deployments where required.

---

## 9. CI Pipeline

- [ ] Require pull requests for protected branches.
- [ ] Protect `main`.
- [ ] Require CI checks before merging.
- [ ] Require review from another maintainer for sensitive changes.
- [ ] Define `CODEOWNERS` for:
  - [ ] Infrastructure
  - [ ] Deployment workflows
  - [ ] Authentication and security
  - [ ] Database migrations
- [ ] Run:
  - [ ] Formatting
  - [ ] Linting
  - [ ] Type checking
  - [ ] Unit tests
  - [ ] Integration tests
  - [ ] Migration checks
  - [ ] Dependency vulnerability scanning
  - [ ] Secret scanning
  - [ ] Container or artifact scanning
  - [ ] IaC validation and security scanning
- [ ] Generate immutable, versioned build artifacts.
- [ ] Record the Git commit SHA in every artifact.
- [ ] Produce the artifact once and promote the same artifact through environments.
- [ ] Generate an SBOM where practical.
- [ ] Pin third-party GitHub Actions to immutable commit SHAs for sensitive workflows.
- [ ] Minimize GitHub Actions token permissions.
- [ ] Avoid executing untrusted pull-request code with production-capable credentials.
- [ ] Configure dependency update automation.
- [ ] Define an urgent-patch workflow.
- [ ] Retain CI and deployment logs for an explicit period.
- [ ] Prevent force-pushes and branch deletion on protected branches.

---

## 10. Automated Staging Deployment

- [ ] Create a GitHub `staging` environment.
- [ ] Deploy automatically after successful merges to `main`, or adopt a deliberate release branch if preferred.
- [ ] Authenticate to AWS through GitHub OIDC.
- [ ] Do not store long-lived AWS keys in GitHub.
- [ ] Restrict the staging role to the staging AWS account and resources.
- [ ] Run infrastructure planning.
- [ ] Apply approved or policy-compliant infrastructure changes.
- [ ] Deploy the immutable application artifact.
- [ ] Run database migrations.
- [ ] Run smoke tests.
- [ ] Run health checks.
- [ ] Publish the deployment URL and commit SHA.
- [ ] Notify both maintainers of success or failure.
- [ ] Preserve deployment logs.
- [ ] Automatically roll back application deployment when health checks fail, where safe.
- [ ] Keep database rollback separate and explicit.
- [ ] Prevent concurrent staging deployments from colliding.
- [ ] Make the deployment status visible in GitHub.
- [ ] Record artifact digest, migration version, deployer, and timestamp.

---

## 11. Automated Production Deployment

- [ ] Create a GitHub `production` environment.
- [ ] Use a distinct production AWS role.
- [ ] Restrict that role to the production account and resources.
- [ ] Trigger production deployment through an explicit release action:
  - [ ] Version tag
  - [ ] GitHub release
  - [ ] Manual promotion of a verified artifact
- [ ] Do not rebuild the application for production.
- [ ] Promote the exact artifact tested in staging.
- [ ] Require successful staging deployment and tests first.
- [ ] Require production approval.
- [ ] Prevent self-approval when practical.
- [ ] Permit both Gabriel and Aaron to approve.
- [ ] Require approval from someone other than the person initiating the deployment when practical.
- [ ] Display the complete deployment plan before approval.
- [ ] Back up stateful systems before risky changes.
- [ ] Run migrations as a controlled step.
- [ ] Deploy using:
  - [ ] Rolling deployment
  - [ ] Blue/green deployment
  - [ ] Canary deployment
- [ ] Run post-deployment smoke tests.
- [ ] Monitor error rate, latency, and health during rollout.
- [ ] Stop or roll back when thresholds are exceeded.
- [ ] Record who approved and initiated each deployment.
- [ ] Record commit SHA, artifact identifier, and migration version.
- [ ] Generate release notes.
- [ ] Notify both operators after completion.
- [ ] Prevent concurrent production deployments.
- [ ] Add a documented manual deployment path for emergencies.
- [ ] Add a documented rollback command or workflow.
- [ ] Test rollback before launch.

---

## 12. Observability

- [ ] Centralize application logs.
- [ ] Use structured logs.
- [ ] Include:
  - [ ] Timestamp
  - [ ] Environment
  - [ ] Service
  - [ ] Request or correlation ID
  - [ ] Release version
  - [ ] Severity
- [ ] Prevent credentials, tokens, and sensitive personal data from entering logs.
- [ ] Set explicit log-retention periods.
- [ ] Encrypt logs.
- [ ] Collect infrastructure and application metrics.
- [ ] Build dashboards for:
  - [ ] Request rate
  - [ ] Error rate
  - [ ] Latency
  - [ ] Saturation
  - [ ] Task or instance health
  - [ ] Database capacity
  - [ ] Database connections
  - [ ] Queue depth
  - [ ] Background-job failures
- [ ] Add external uptime checks.
- [ ] Configure alerts for:
  - [ ] Site unavailable
  - [ ] Elevated 5xx responses
  - [ ] Elevated latency
  - [ ] Unhealthy deployment
  - [ ] Database storage pressure
  - [ ] CPU or memory exhaustion
  - [ ] Backup failure
  - [ ] Certificate problems
  - [ ] Queue backlog
  - [ ] Security findings
  - [ ] Unexpected cost growth
- [ ] Route alerts to both Gabriel and Aaron.
- [ ] Define warning versus page-worthy alert severity.
- [ ] Test alert delivery.
- [ ] Ensure every urgent alert links to a runbook.
- [ ] Configure error tracking with release correlation.
- [ ] Define service-level indicators and initial service objectives.
- [ ] Add deployment markers to dashboards.
- [ ] Verify logs and metrics remain available during an application outage.

---

## 13. Security Monitoring and Auditability

- [ ] Enable CloudTrail.
- [ ] Use a multi-region trail for management activity.
- [ ] Include read and write management events.
- [ ] Encrypt CloudTrail logs.
- [ ] Enable log-file validation.
- [ ] Store logs in a nonpublic, access-restricted bucket.
- [ ] Send relevant CloudTrail events to CloudWatch Logs.
- [ ] Enable AWS Config where cost and scope permit.
- [ ] Enable GuardDuty.
- [ ] Enable Security Hub and AWS Foundational Security Best Practices controls.
- [ ] Enable IAM Access Analyzer.
- [ ] Review public and cross-account resource access.
- [ ] Enable S3 Block Public Access at the account level unless explicitly incompatible.
- [ ] Enable ECR or image vulnerability scanning if containers are used.
- [ ] Monitor changes to IAM, KMS, CloudTrail, DNS, and security groups.
- [ ] Alert on root-account activity.
- [ ] Alert on failed console logins and suspicious authentication.
- [ ] Define a vulnerability-remediation SLA.
- [ ] Run a production security review before launch.
- [ ] Record and assign every security finding.
- [ ] Review high-severity findings before every production launch.

---

## 14. Application Security

- [ ] Document the authentication and authorization model.
- [ ] Test authorization independently of UI visibility.
- [ ] Enforce server-side access control.
- [ ] Use secure password hashing if MuseHub stores passwords.
- [ ] Require email verification where relevant.
- [ ] Implement secure account recovery.
- [ ] Rate-limit login and recovery endpoints.
- [ ] Protect against CSRF where applicable.
- [ ] Validate and sanitize untrusted input.
- [ ] Use parameterized database queries.
- [ ] Restrict CORS to deliberate origins.
- [ ] Define a Content Security Policy.
- [ ] Add standard security headers.
- [ ] Validate uploaded file type, size, and storage behavior.
- [ ] Protect against path traversal and unsafe archive extraction.
- [ ] Scan uploaded content if the risk model warrants it.
- [ ] Set request-body and upload limits.
- [ ] Review SSRF exposure from URL-fetching features.
- [ ] Ensure production debug mode is disabled.
- [ ] Ensure stack traces are not returned to users.
- [ ] Review WebSocket or SSE authentication if used.
- [ ] Document data classification and retention.
- [ ] Add an account and data deletion path where applicable.
- [ ] Run an OWASP-oriented review before launch.
- [ ] Review session expiration and token revocation behavior.
- [ ] Verify privileged actions require fresh authorization where appropriate.

---

## 15. Cost Controls

- [ ] Apply consistent cost-allocation tags.
- [ ] Create monthly AWS budgets.
- [ ] Alert Gabriel and Aaron at multiple thresholds.
- [ ] Enable Cost Anomaly Detection.
- [ ] Review NAT Gateway, data-transfer, and logging costs.
- [ ] Set log-retention limits.
- [ ] Set backup-retention limits consistent with recovery requirements.
- [ ] Right-size production after load testing.
- [ ] Review idle resources.
- [ ] Document expected baseline monthly cost.
- [ ] Document the cost implications of autoscaling.
- [ ] Configure service quotas and limits deliberately.
- [ ] Review production cost weekly during the first month.
- [ ] Add owner and purpose tags to every billable resource.
- [ ] Alert on untagged or unexpectedly expensive resources where practical.

---

## 16. Documentation and Runbooks

- [ ] Create a system architecture diagram.
- [ ] Create a data-flow diagram.
- [ ] Document every AWS account and region.
- [ ] Document DNS and certificate ownership.
- [ ] Document environment differences.
- [ ] Document the IaC repository structure.
- [ ] Document local development setup.
- [ ] Document staging deployment.
- [ ] Document production deployment.
- [ ] Document rollback.
- [ ] Document database migration procedures.
- [ ] Document database restoration.
- [ ] Document secret rotation.
- [ ] Document certificate and DNS troubleshooting.
- [ ] Document user-access provisioning and removal.
- [ ] Document break-glass access.
- [ ] Document incident response.
- [ ] Document backup retention and restoration tests.
- [ ] Document common alarms and remediation.
- [ ] Create a service inventory with owners.
- [ ] Create a dependency inventory.
- [ ] Create an operational decision log.
- [ ] Record accepted risks and deferred work.
- [ ] Keep documentation versioned beside the relevant code where practical.
- [ ] Confirm Aaron can follow the documentation without Gabriel’s help.
- [ ] Confirm Gabriel can follow it without relying on personal memory.
- [ ] Add a production launch checklist and rollback decision tree.
- [ ] Add contact information and escalation paths.
- [ ] Review documentation after every incident or major deployment.

---

## 17. Testing Before Launch

- [ ] Run the complete automated test suite.
- [ ] Run production-like integration tests.
- [ ] Test authentication and authorization.
- [ ] Test a clean infrastructure deployment in a disposable environment.
- [ ] Test migration from the current schema.
- [ ] Test application rollback.
- [ ] Test failed migration handling.
- [ ] Test a database restore.
- [ ] Test backup integrity.
- [ ] Test task or instance failure.
- [ ] Test deployment during active traffic.
- [ ] Test autoscaling if enabled.
- [ ] Run basic load and soak tests.
- [ ] Verify timeout behavior.
- [ ] Verify retry behavior.
- [ ] Verify idempotency for jobs and webhooks.
- [ ] Test DNS and TLS from external networks.
- [ ] Test both root and `www`.
- [ ] Verify canonical redirects.
- [ ] Test monitoring and alert delivery.
- [ ] Run a tabletop incident exercise with Gabriel and Aaron.
- [ ] Test the break-glass process.
- [ ] Verify neither maintainer is a single point of operational failure.
- [ ] Test production access from a clean workstation.
- [ ] Test what happens when a secret, database, queue, or external API is unavailable.

---

## 18. Cutover Plan

- [ ] Freeze unrelated infrastructure changes during cutover.
- [ ] Confirm the production environment is healthy.
- [ ] Confirm production secrets are installed.
- [ ] Confirm migrations have been rehearsed.
- [ ] Confirm current backups succeeded.
- [ ] Confirm a restoration has been tested.
- [ ] Confirm dashboards and alerts are active.
- [ ] Confirm Aaron and Gabriel are both available for the cutover.
- [ ] Lower DNS TTL in advance.
- [ ] Deploy the approved production artifact.
- [ ] Run migrations.
- [ ] Run internal smoke tests.
- [ ] Update DNS.
- [ ] Verify:
  - [ ] DNS resolution
  - [ ] TLS
  - [ ] Root domain
  - [ ] `www`
  - [ ] API
  - [ ] Authentication
  - [ ] Critical user paths
  - [ ] Background workers
  - [ ] Outbound email
  - [ ] Object uploads and downloads
- [ ] Monitor logs, error rate, and latency closely.
- [ ] Keep the previous environment available for rollback during the agreed window.
- [ ] Execute rollback if predetermined thresholds are crossed.
- [ ] Raise DNS TTL after stability is confirmed.
- [ ] Record the launch artifact and configuration versions.
- [ ] Announce launch status to stakeholders.
- [ ] Record all manual actions taken during cutover.
- [ ] Confirm production backups run successfully after cutover.

---

## 19. Post-Launch

- [ ] Review production metrics after the first hour.
- [ ] Review again after 24 hours.
- [ ] Review again after one week.
- [ ] Inspect AWS costs after 24 hours and one week.
- [ ] Review security findings.
- [ ] Confirm backups continue to succeed.
- [ ] Perform another restore test after production has real data.
- [ ] Remove temporary cutover access and resources.
- [ ] Remove obsolete staging-derived credentials.
- [ ] Resolve or formally record every launch exception.
- [ ] Conduct a blameless launch retrospective.
- [ ] Update documentation with lessons learned.
- [ ] Schedule:
  - [ ] Monthly dependency updates
  - [ ] Monthly access review during the early production period
  - [ ] Quarterly disaster-recovery exercise
  - [ ] Quarterly permissions review
  - [ ] Quarterly cost review
  - [ ] Annual domain and recovery-contact review

---

# Recommended Implementation Order

## P0 — Before Creating Production

- [ ] Inventory staging.
- [ ] Establish shared organizational ownership and MFA.
- [ ] Decide AWS account boundaries.
- [ ] Select and implement Infrastructure as Code.
- [ ] Define RPO, RTO, and domain topology.
- [ ] Decide the canonical hostname and DNS provider.

## P1 — Build Production

- [ ] Provision the production account and environment through IaC.
- [ ] Configure identity, networking, encryption, and secrets.
- [ ] Configure the database, automated backups, and restore testing.
- [ ] Configure GitHub OIDC and separate deployment roles.
- [ ] Build staging and production pipelines.
- [ ] Add logs, dashboards, alarms, and audit trails.
- [ ] Complete operational documentation.

## P2 — Launch

- [ ] Configure certificates and DNS.
- [ ] Rehearse migrations, deployment, and rollback.
- [ ] Run security and load tests.
- [ ] Complete the cutover.
- [ ] Validate production and document the outcome.

---

# Suggested Ownership Split

| Workstream | Primary | Reviewer / Backup | Status |
|---|---|---|---|
| Staging architecture inventory | Gabriel | Claude Code | In progress — see Section 0 |
| AWS Organizations and accounts |  |  | Not started |
| IAM Identity Center and permissions |  |  | Not started |
| Infrastructure as Code |  |  | Not started |
| Networking and production architecture |  |  | Not started |
| Namecheap, DNS, and TLS |  |  | Not started |
| Database and backups |  |  | Not started |
| GitHub CI/CD |  |  | Not started |
| Logging, metrics, and alerts |  |  | Not started |
| Security review |  |  | Not started |
| Documentation and runbooks |  |  | Not started |
| Cutover coordination |  |  | Not started |

---

# Decision Log

| Date | Decision | Rationale | Owner |
|---|---|---|---|
| 2026-08-04 | Accept single AWS account (`992382692655`, shared with Stori) as technical debt rather than splitting into AWS Organizations before launch | Keeps focus on getting production launched; account split can happen later without blocking | Gabriel |
| 2026-08-04 | Architecture target (harden current EC2/Docker-Compose vs. redesign to ECS+ALB+RDS) left open | Insufficient information at Section 0 time; revisit once inventory and RPO/RTO are settled | Gabriel |
| 2026-08-04 | IaC tool selection deferred; Terraform/OpenTofu recommended given the actual footprint (2 EC2, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params) | Small, EC2-centric footprint doesn't need CDK's language ceremony or CloudFormation's verbosity | Gabriel |
| 2026-08-04 | Every "GitHub" checklist item is translated to its MuseHub equivalent (proposals for PRs, MuseHub branch protection, `musehub-runner` CI, MuseHub collaborator roles) | Absolute ecosystem policy — git/GitHub are never the VCS, CI, or collaboration platform for this project; GitHub is a passive backup mirror only | Gabriel (policy), applied throughout by Claude Code |

---

# Accepted Risks and Deferred Work

| Item | Risk | Mitigation | Target Date | Owner |
|---|---|---|---|---|
| Single AWS account shared with Stori | Blast radius: a MuseHub compromise/misconfig can affect Stori resources and vice versa; billing and ownership entangled | Revisit AWS Organizations split post-launch; `musehub-infra` IAM user already scoped narrowly (verified via access-denied tests on IAM/CloudTrail/GuardDuty/Route53/etc.) | TBD | Gabriel |
| Object storage ground truth unconfirmed (local Docker volume vs. Cloudflare R2) | Sections 7 and 8 could be built on the wrong baseline | Gabriel to confirm before Section 7/8 work is finalized | Before Section 7/8 sign-off | Gabriel |
| Secrets/backup pipeline ground truth unconfirmed (static `.env` vs. SSM+R2 scripts already in `deploy/`) | Same as above — hardening work could target a pipeline that isn't actually live | Gabriel to confirm before Section 7/8 work is finalized | Before Section 7/8 sign-off | Gabriel |
| No RPO/RTO, traffic, or availability targets defined | Can't size backup cadence, DR posture, or capacity correctly without them | Define before Section 5 (architecture) and Section 8 (backups) are finalized | Before P1 build phase | Gabriel |
| Canonical hostname (root vs. `www`) and DNS host (Namecheap vs. Route 53) undecided | Blocks finalizing Section 6 and the cutover plan | Decide before Section 18 cutover | Before cutover | Gabriel |