MuseHub Production Infrastructure Launch TODO

Goal

Reproduce the proven staging architecture in the dedicated MuseHub Production AWS account, deploy and validate the application, configure production DNS and TLS, and establish a safe, repeatable production deployment and rollback workflow.

This is an infrastructure reproduction and promotion project—not a blind copy. Do not copy credentials, secrets, mutable data, IAM identities, or obsolete resources from Nonproduction without an explicit decision.

Account map

Non-negotiable safety rules

• [ ] Perform production work through IAM Identity Center using [email protected] and the AdministratorAccess permission set—not root credentials.
• [ ] Use an explicit AWS CLI profile for every command; never rely on an implicit default profile.
• [ ] Before any mutation, verify the target account with aws sts get-caller-identity.
• [ ] Production mutations must target account 672469410277.
• [ ] Nonproduction discovery must target account 992382692655 and remain read-only until a change is separately approved.
• [ ] Add account-ID guards to deployment and provisioning scripts so they fail closed in the wrong account.
• [ ] Do not reuse the old long-lived musehub-infra credentials for Production.
• [ ] Do not expose secrets in terminal output, screenshots, commits, agent prompts, or logs.
• [ ] Do not modify public DNS until Production passes pre-cutover validation.
• [ ] Define and test rollback before routing production traffic.

Phase 0 — Access and account preflight

• [x] Confirm [email protected] has the required IAM Identity Center access to:

  • [x] MuseHub Nonproduction for read-only discovery initially. — assigned AdministratorAccess 2026-08-23
  • [x] MuseHub Production with AdministratorAccess for initial provisioning. — already assigned

• [x] Configure named AWS CLI SSO profiles, preferably:

  • [x] musehub-nonproduction
  • [x] musehub-production

• [x] Use the working dual-stack AWS access portal URL for SSO configuration. — https://ssoins-7223bcb94598e297.portal.us-east-1.app.aws; the Regional IPv4 URL (*.us-east-1.portal.amazonaws.com) fails SAML sign-in the same way the legacy awsapps.com/start URL does — dual-stack is the only one that works for this SAML app config. Configured directly in ~/.aws/config as [sso-session musehub] rather than via the aws configure sso wizard, whose start-URL format validator rejects this URL even though it's valid (client-side bug, not a URL problem).

• [x] Set the SSO region and default working region to us-east-1 unless discovery proves a resource belongs elsewhere.

• [x] Authenticate and verify both profiles:

  aws sso login --sso-session musehub
  aws sts get-caller-identity --profile musehub-nonproduction
  aws sts get-caller-identity --profile musehub-production
  

• [x] Confirm the first command reports 992382692655 and the second reports 672469410277. — verified 2026-08-23

• [ ] Confirm production billing visibility and create an initial AWS Budget with email alerts. — needs Gabriel, not yet done

• [x] Record the chosen AWS region, SSO profile names, and account IDs in the repository documentation without recording credentials. — this section

Phase 1 — Read-only staging and repository discovery

Repository inspection

• [x] Locate all infrastructure and deployment assets using repository search:
  • [x] Deployment scripts and commands. — deploy/push.sh, deploy/deploy.sh, deploy/set-active-slot.sh, deploy/aws-provision*.sh, deploy/setup-ec2*.sh, deploy/bootstrap-instance.sh, deploy/backup.sh, deploy/cloudwatch-alerts.sh
  • [x] Dockerfiles and Compose files. — Dockerfile, docker-compose.yml, docker-compose.override.yml
  • [x] Infrastructure-as-code templates. — none exist; confirmed by docs/musehub-production-readiness-checklist.md Section 4 (IaC tool not yet chosen, Terraform/OpenTofu recommended but not built)
  • [x] Environment templates. — .env.example; live .env is hand-provisioned per instance, not templated
  • [x] Reverse-proxy configuration. — deploy/nginx-cf.conf (Cloudflare Origin Certificate model, not Let's Encrypt/ACM)
  • [x] Database migration and backup commands. — Alembic via deploy/deploy.sh; deploy/backup.sh exists but is not wired in (no cron, no rclone configured — see checklist Accepted Risks)
  • [x] Existing staging/production documentation. — docs/infrastructure.md, docs/deploy.md, docs/musehub-production-readiness-checklist.md and its production-readiness/ companion docs
• [x] Document the exact current staging deployment command and its inputs. — bash deploy/push.sh staging; builds via Docker, pushes to ECR with crane, triggers deploy.sh over SSM
• [x] Determine how the existing blue/green deployment works, including ports, health checks, active-slot selection, and rollback. — blue=1337/green=1338, /healthz gate, musehub-set-slot script writes the nginx upstream file, rollback via IMAGE_TAG=<prev> bash deploy/push.sh <env>
• [x] Identify every place where staging-specific account IDs, hostnames, IP addresses, paths, ports, repository names, or secrets are hard-coded. — ECR registry 992382692655... is hard-coded in deploy/push.sh, deploy/deploy.sh, deploy/bootstrap-instance.sh; instance IDs hard-coded in deploy/push.sh, deploy/bootstrap-instance.sh, .museagent.md; see discovery report for full list

Key discovery note (2026-08-23): docs/musehub-production-readiness-checklist.md (dated 2026-08-04/08-08) already contains a much deeper, earlier production-readiness sweep than this TODO. It documents that the "prod" EC2 instance (i-0855d6efe7fa1a49d) referenced throughout deploy/ scripts lives in the old shared Nonproduction/Stori account (992382692655), is currently broken (no IAM instance profile, push.sh prod fails with InvalidInstanceId), and that DNS/TLS actually run through Cloudflare (DNS hosting + Origin Certificate), not Namecheap/Let's Encrypt/ACM as this TODO's Phase 5 assumes. See the full discovery report delivered separately for details before Phase 2 proceeds.

AWS inventory in MuseHub Nonproduction

• [x] Inventory relevant resources in every used region, starting with us-east-1:
  • [x] VPCs, subnets, route tables, internet gateways, NAT gateways, and network ACLs. — single default VPC (vpc-0552bdd377de8a9a8, 172.31.0.0/16); no custom networking
  • [x] EC2 instances, AMIs, launch templates, volumes, snapshots, and Elastic IPs. — musehub-staging (i-07547cd20bee2dea5, t3.medium, running, EIP 23.22.27.39) and musehub-prod (i-0855d6efe7fa1a49d, t3.small, running, EIP 98.89.99.211); no launch templates/ASGs
  • [x] Security groups and inbound/outbound rules. — musehub-sg (sg-05815872537fcfe76): 443 open to Cloudflare IP ranges only, as documented. Finding, resolved 2026-08-23: an additional, undocumented rule opened TCP 1337 directly to a single IP (98.51.144.179/32), bypassing nginx/Cloudflare entirely. Confirmed dead (both docker-compose.yml and deploy/deploy.sh bind the app container to 127.0.0.1 only, so nothing was listening on the public interface for it) and revoked (sgr-03be7da55eece078f). Only port 443 remains open.
  • [x] IAM roles, instance profiles, policies, and deployment identities. — Users: musehub-infra, musehub-app, stori-assets-app (all expected), plus s3-migration-temp — created 2026-02-10 with AmazonS3FullAccess, used exactly once the day it was created (R2 migration, per deploy/migrate_r2_keys.py) and never since. Resolved 2026-08-23: access key deactivated (reversible — reactivate with aws iam update-access-key --status Active if ever needed). Roles: musehub-ec2-ssm, rds-monitoring-role, tus-image-processing-1-role-7dl2zvb1 (Stori-owned, not ours), plus the new SSO role.
  • [x] ECR repositories, image tags/digests, scan settings, and lifecycle policies. — one repo musehub/musehub, scan-on-push enabled, AES256 encryption, mutable tags, ~420 images accumulated with no pruning. Resolved 2026-08-23: lifecycle policy applied — expire untagged images after 14 days, keep only the most recent 60 tagged images.
  • [x] S3 buckets, policies, encryption, lifecycle rules, and CORS settings. — musehub-releases (ours) and stori-assets-992382692655-useast2 (Stori's, confirms shared-account risk from the checklist); no bucket found for DB backups.
  • [x] Databases, caches, queues, and other stateful services. — Finding: a managed RDS Postgres instance exists (musehub-staging-db, db.t3.micro, 20GB, single-AZ, not publicly accessible, 7-day automated backups) that isn't mentioned in any doc. The app's actual DATABASE_URL points at the containerized postgres:5432 service per docs/infrastructure.md, so this RDS instance is very likely orphaned/unused — needs Gabriel to confirm before considering decommission (it's actively costing money and, confusingly, is backed up, unlike the real database). No Lambda, DynamoDB, SQS, or SNS found.
  • [x] Systems Manager parameters and Secrets Manager secret names only—not their values. — Secrets Manager: empty. SSM: /musehub/staging/PACK_WORKER_URL, /musehub/staging/WEBHOOK_SECRET_KEY, /musehub/staging/WORKER_INTERNAL_KEY — a partial, abandoned attempt at the SSM secrets pipeline (missing DB_PASSWORD/RUNNER_TOKEN), consistent with docs/infrastructure.md noting deploy/secrets.sh "is not the live path."
  • [x] CloudWatch log groups, alarms, dashboards, and retention settings. — one log group /musehub/staging, retention set to Never Expire, already 1.9GB stored; zero alarms configured anywhere (cloudwatch-alerts.sh has evidently never been run for real — matches checklist Section 12 status).
  • [x] Load balancers, target groups, listeners, and health checks, if present. — none; direct EC2 + Elastic IP + nginx, as documented.
  • [x] DNS records hosted in AWS, if any. — none; DNS is Cloudflare-hosted (Route 53 not used).
  • [x] Backup plans, retention policies, and restore procedures. — none for the real (containerized) database; confirms the checklist's highest-priority open risk.
• [x] Determine whether the existing musehub-prod EC2 resource inside Nonproduction is active, obsolete, or safe to retire later. Do not alter it during discovery. — running but non-functional: it's up and billing, but per docs/infrastructure.md has no IAM instance profile attached, so SSM can't reach it and push.sh prod fails outright. Not touched during this discovery. Disposition (fix in place vs. retire in favor of the new Production account) is a Phase 2/7 decision for Gabriel.

AWS starting state in MuseHub Production

• [x] Confirmed clean: default VPC only (vpc-038dc9e464d82a05d), zero EC2 instances, zero S3 buckets, zero ECR repositories, zero IAM users, zero Elastic IPs, zero RDS instances. Only roles present are the SSO AdministratorAccess role and the standard AWS Organizations OrganizationAccountAccessRole. Ready for a from-scratch build.

Staging host inspection

• [x] Record the staging host's:
  • [x] OS and architecture. — Ubuntu 22.04.5 LTS, x86_64
  • [x] Docker and Compose versions. — Docker 29.3.1, Compose v5.1.1
  • [x] Reverse proxy and configuration paths. — nginx, /etc/nginx/sites-available/musehub-staging (only site enabled); Cloudflare Origin Cert at /etc/ssl/cloudflare/origin.{pem,key} with correct permissions (644/640)
  • [x] Listening ports and firewall rules. — publicly: 443 (nginx) and 22 (sshd) on 0.0.0.0/[::]. Note: the security group has no port-22 ingress rule, so SSH isn't reachable from the internet today — but sshd is running and bound to all interfaces as a live daemon, meaning the SG is the only thing preventing exposure. Worth disabling sshd outright on the new Production host (SSM-only, per the existing "no SSH" design intent) rather than relying solely on the SG. App ports 1337/1338 confirmed bound to 127.0.0.1 only (docker-proxy), matching docker-compose.yml/deploy.sh — this is what made the now-removed SG rule for port 1337 a dead rule.
  • [x] systemd units, scheduled jobs, and deployment hooks. — no user crontab (no crontab for ubuntu); /etc/cron.d/ has only default OS entries (certbot, e2scrub_all) — deploy/backup.sh is confirmed never installed. A leftover certbot.timer fires every ~14h from the default Ubuntu certbot package install but has nothing to renew (Cloudflare handles TLS) — harmless noise, not a real risk.
  • [x] Environment-file and secret paths without displaying values. — /opt/musehub/.env, root-owned, 644 permissions (world-readable) — should be tightened to 600/640 on the new Production host.
  • [x] Certbot/Let's Encrypt configuration and renewal timers, if used. — not used for TLS (Cloudflare Origin Cert instead); default certbot package/timer present but inert.
  • [x] Log locations and rotation policy. — no dedicated log files; app logs go to stdout, captured by the awslogs Docker log driver into CloudWatch (/musehub/staging, never-expire retention — see Phase 1 AWS inventory finding above).
• [x] Map all external dependencies: databases, object storage, email, OAuth, webhooks, APIs, payment services, analytics, and DNS. — containerized Postgres (not the orphaned RDS instance), Cloudflare R2 for object storage, Cloudflare for DNS/edge/TLS; no email/payment/analytics dependencies identified in this pass.
• [x] Produce a staging inventory and dependency map before provisioning Production. — see this section plus the Phase 1 AWS inventory above.

Additional findings from host inspection (2026-08-24), not yet acted on:

• /opt/musehub on the staging host contains the entire repository — node_modules/, tests/, tools/, .mypy_cache/, .cursor/, and large standalone design docs (MUSEHUB_V2.md at ~100KB, several *_REDESIGN.md files) — not just the running app. This is a leftover from the pre-image-based deploy method (the old rsync-based setup scripts), and contradicts docs/deploy.md's current claim that deploys are "image-based via ECR. No SSH, no rsync, no code on the instance after provisioning." It's dead weight (likely a meaningful share of the 14GB/20GB, 70% disk usage observed) but not a safety issue — recommend a separate, deliberate cleanup task rather than doing it mid-discovery.
• The one existing manual backup is empty: /opt/backups/musehub/musehub_staging_manual_20260613_003724.sql.gz is 20 bytes — independently reproduces the checklist's flagged finding that even the sole manual dump ever taken is unusable. Reinforces that this is the single highest-priority risk to close before or immediately after production launch.
• Disk usage on staging is already at 70% (14G/20G) with no automated cleanup beyond the on-host image prune (last 3) — worth sizing the Production EBS volume larger and/or cleaning up the stray repo copy before it becomes a real constraint.

Phase 2 — Decide the production design

• [ ] Decide whether the current repository already contains usable infrastructure as code.
• [ ] If it does, adapt it with isolated production variables and state.
• [ ] If it does not, capture the required production infrastructure as code rather than relying on undocumented console-only creation.
• [ ] Choose the production ingress/TLS topology:
  • [ ] Direct EC2 + Elastic IP + reverse proxy + Let's Encrypt; or
  • [ ] Application Load Balancer + ACM certificate.
• [ ] Confirm the production hostname(s), such as the primary domain and/or www/app subdomain.
• [ ] Decide how application images are promoted:
  • [ ] Preferred: build once and promote/deploy an immutable ECR image digest.
  • [ ] Decide whether Production uses a dedicated ECR repository or controlled cross-account access.
• [ ] Decide how state and data will be handled:
  • [ ] Fresh production database or migration from an existing source.
  • [ ] Asset/object migration requirements.
  • [ ] Backup, restore, and rollback requirements.
• [ ] Establish resource naming and tagging conventions, including at least Environment=production, Application=musehub, and ownership/cost tags.
• [ ] Explicitly identify which staging settings must differ in Production: domains, credentials, OAuth callbacks, email/webhooks, storage, database, logging, and rate limits.

Phase 3 — Provision the production baseline

• [ ] Create or confirm an AWS Budget and cost alerts for MuseHub Production.
• [ ] Confirm CloudTrail/account audit coverage and appropriate log retention.
• [ ] Provision the production network or intentionally use the account default VPC only if documented and accepted.
• [ ] Provision production security groups:
  • [ ] Public access only on required ingress ports, normally 80 and 443.
  • [ ] Restrict SSH to known sources or prefer AWS Systems Manager Session Manager.
  • [ ] Restrict database and internal service access to required security groups.
• [ ] Create the compute layer using the chosen architecture.
• [ ] Encrypt storage and configure snapshots/backups.
• [ ] Allocate an Elastic IP or provision a load balancer, according to the chosen ingress design.
• [ ] Create least-privilege production IAM roles and instance profiles.
• [ ] Create production ECR repositories/access and lifecycle policies.
• [ ] Create production parameters and secrets separately; never copy staging secrets wholesale.
• [ ] Configure CloudWatch logs, retention, alarms, and basic health monitoring.
• [ ] Verify the instance can pull the approved image and access only its required production services.

Phase 4 — Deploy and validate before public DNS

• [ ] Select an immutable application image digest known to work in staging.
• [ ] Deploy it to Production using the same blue/green pattern where practical.
• [ ] Run database migrations only after:
  • [ ] Taking a backup/snapshot.
  • [ ] Reviewing forward and rollback behavior.
  • [ ] Confirming the target is the production database.
• [ ] Test Production without changing public DNS using an Elastic IP, load-balancer hostname, temporary hostname, or local hosts-file override.
• [ ] Validate:
  • [ ] Application health endpoint.
  • [ ] Home page and critical routes.
  • [ ] Authentication and authorization.
  • [ ] Database reads/writes.
  • [ ] Uploads and media delivery.
  • [ ] Email and webhook integrations.
  • [ ] Background jobs/queues.
  • [ ] Logs and alarms.
  • [ ] Restart and reboot recovery.
• [ ] Run a rollback rehearsal to the previous production slot/image.

Phase 5 — Namecheap DNS and TLS

• [ ] Confirm the final production hostname(s) and current DNS records in Namecheap.
• [ ] Lower DNS TTL in advance of cutover if appropriate.
• [ ] Create the appropriate record:
  • [ ] A record to an Elastic IP for direct EC2 ingress; or
  • [ ] CNAME/alias-compatible record to a load balancer hostname.
• [ ] Verify propagation with dig/nslookup from more than one resolver.
• [ ] Complete TLS according to the chosen topology:
  • [ ] Direct EC2: issue a Let's Encrypt certificate with Certbot only after DNS resolves correctly and ports 80/443 are reachable.
  • [ ] Load balancer: request and validate an ACM certificate, then attach it to the HTTPS listener.
• [ ] Configure HTTP-to-HTTPS redirection.
• [ ] Validate the full certificate chain, hostname coverage, and expiration.
• [ ] For Let's Encrypt, confirm the renewal timer and run certbot renew --dry-run.
• [ ] Re-run the complete production smoke test over the final HTTPS hostname.

Phase 6 — Make production deployment repeatable

• [ ] Update the deployment tooling to require an explicit environment, such as staging or production.
• [ ] Add expected-account guards:
  • [ ] Staging/nonproduction requires 992382692655.
  • [ ] Production requires 672469410277.
• [ ] Require an explicit production release version/tag or immutable image digest.
• [ ] Make production deployment use the production SSO profile and production-specific configuration.
• [ ] Preserve the working blue/green deployment behavior and health gate.
• [ ] Add a documented one-command or short-command rollback.
• [ ] Prevent accidental use of staging secrets, hostnames, databases, or buckets in Production.
• [ ] Ensure deploy logs clearly state the target account, region, environment, host, image digest, and resulting active slot without printing secrets.
• [ ] Add CI/CD later only after the manual scripted flow is deterministic and verified.

Phase 7 — Cutover and post-launch

• [ ] Capture the pre-cutover production backup/snapshot and rollback point.
• [ ] Confirm owners, decision-maker, and rollback trigger.
• [ ] Route production DNS only after all acceptance checks pass.
• [ ] Monitor application health, error logs, CPU/memory/disk, database health, and external integrations during the launch window.
• [ ] Verify AWS billing/cost telemetry after resources have run long enough to report.
• [ ] Raise DNS TTL after the environment is stable, if it was lowered.
• [ ] Document the deployed architecture, resource identifiers, operational commands, backup/restore procedure, and incident contacts.
• [ ] Decide whether and when to retire the obsolete musehub-prod resource in Nonproduction. Treat retirement as a separate, approved task.

Acceptance criteria

• [ ] Production resources reside only in AWS account 672469410277.
• [ ] Staging remains functional and unchanged throughout the launch.
• [ ] Engineers authenticate through Google Workspace and IAM Identity Center; no root access keys are used.
• [ ] Production is accessible on its intended HTTPS hostname with a valid, renewable certificate.
• [ ] Production secrets and state are isolated from staging.
• [ ] A known-good image can be deployed to Production through a documented, guarded command.
• [ ] Rollback is documented and successfully rehearsed.
• [ ] Monitoring, logs, backups, budgets, and basic alerts are active.
• [ ] Repository documentation accurately describes the final architecture and operating procedure.

Decisions that must be resolved during discovery

• [ ] Exact production hostname(s).
• [ ] Direct EC2/Let's Encrypt versus load balancer/ACM.
• [ ] Existing infrastructure-as-code strategy and state location.
• [ ] Production database and data-migration plan.
• [ ] ECR promotion/cross-account strategy.
• [ ] Production instance size and scaling requirements.
• [ ] Required OAuth, email, webhook, storage, and third-party production configuration.
• [ ] Fate of the old musehub-prod resource in Nonproduction.

Agent execution checkpoints

An implementation agent should stop and report at these gates:

1. After access verification: report both caller identities and profiles.
2. After read-only discovery: deliver the inventory, dependency map, proposed design, cost-impacting resources, and unresolved questions.
3. Before provisioning: obtain approval for the exact production plan.
4. Before any state/data migration: provide backup and rollback procedures.
5. Before DNS changes: provide validation results and the DNS/TLS cutover plan.
6. After launch: provide smoke-test evidence, deployed image digest, resource summary, and rollback command.