# MuseHub Production Infrastructure Launch TODO

## Goal

Reproduce the proven staging architecture in the dedicated **MuseHub Production** AWS account, deploy and validate the application, configure production DNS and TLS, and establish a safe, repeatable production deployment and rollback workflow.

This is an infrastructure reproduction and promotion project—not a blind copy. Do not copy credentials, secrets, mutable data, IAM identities, or obsolete resources from Nonproduction without an explicit decision.

## Account map

| Purpose | AWS account | Account ID | Root mailbox |
|---|---|---:|---|
| Governance, billing, IAM Identity Center | MuseHub Management | Record in password manager/docs | `aws-management@musehub.ai` |
| Staging and existing workloads | MuseHub Nonproduction | `992382692655` | `aws-nonproduction@musehub.ai` |
| New production environment | MuseHub Production | `672469410277` | `aws-production@musehub.ai` |

## Non-negotiable safety rules

- [ ] Perform production work through IAM Identity Center using `gabriel@musehub.ai` and the `AdministratorAccess` permission set—not root credentials.
- [ ] Use an explicit AWS CLI profile for every command; never rely on an implicit `default` profile.
- [ ] Before any mutation, verify the target account with `aws sts get-caller-identity`.
- [ ] Production mutations must target account `672469410277`.
- [ ] Nonproduction discovery must target account `992382692655` and remain read-only until a change is separately approved.
- [ ] Add account-ID guards to deployment and provisioning scripts so they fail closed in the wrong account.
- [ ] Do not reuse the old long-lived `musehub-infra` credentials for Production.
- [ ] Do not expose secrets in terminal output, screenshots, commits, agent prompts, or logs.
- [ ] Do not modify public DNS until Production passes pre-cutover validation.
- [ ] Define and test rollback before routing production traffic.

## Phase 0 — Access and account preflight

- [x] Confirm `gabriel@musehub.ai` has the required IAM Identity Center access to:
  - [x] MuseHub Nonproduction for read-only discovery initially. — assigned `AdministratorAccess` 2026-08-23
  - [x] MuseHub Production with `AdministratorAccess` for initial provisioning. — already assigned
- [x] Configure named AWS CLI SSO profiles, preferably:
  - [x] `musehub-nonproduction`
  - [x] `musehub-production`
- [x] Use the working **dual-stack** AWS access portal URL for SSO configuration. — `https://ssoins-7223bcb94598e297.portal.us-east-1.app.aws`; the Regional IPv4 URL (`*.us-east-1.portal.amazonaws.com`) fails SAML sign-in the same way the legacy `awsapps.com/start` URL does — dual-stack is the only one that works for this SAML app config. Configured directly in `~/.aws/config` as `[sso-session musehub]` rather than via the `aws configure sso` wizard, whose start-URL format validator rejects this URL even though it's valid (client-side bug, not a URL problem).
- [x] Set the SSO region and default working region to `us-east-1` unless discovery proves a resource belongs elsewhere.
- [x] Authenticate and verify both profiles:

  ```bash
  aws sso login --sso-session musehub
  aws sts get-caller-identity --profile musehub-nonproduction
  aws sts get-caller-identity --profile musehub-production
  ```

- [x] Confirm the first command reports `992382692655` and the second reports `672469410277`. — verified 2026-08-23
- [ ] Confirm production billing visibility and create an initial AWS Budget with email alerts. — **needs Gabriel**, not yet done
- [x] Record the chosen AWS region, SSO profile names, and account IDs in the repository documentation without recording credentials. — this section

## Phase 1 — Read-only staging and repository discovery

### Repository inspection

- [x] Locate all infrastructure and deployment assets using repository search:
  - [x] Deployment scripts and commands. — `deploy/push.sh`, `deploy/deploy.sh`, `deploy/set-active-slot.sh`, `deploy/aws-provision*.sh`, `deploy/setup-ec2*.sh`, `deploy/bootstrap-instance.sh`, `deploy/backup.sh`, `deploy/cloudwatch-alerts.sh`
  - [x] Dockerfiles and Compose files. — `Dockerfile`, `docker-compose.yml`, `docker-compose.override.yml`
  - [x] Infrastructure-as-code templates. — none exist; confirmed by `docs/musehub-production-readiness-checklist.md` Section 4 (IaC tool not yet chosen, Terraform/OpenTofu recommended but not built)
  - [x] Environment templates. — `.env.example`; live `.env` is hand-provisioned per instance, not templated
  - [x] Reverse-proxy configuration. — `deploy/nginx-cf.conf` (Cloudflare Origin Certificate model, not Let's Encrypt/ACM)
  - [x] Database migration and backup commands. — Alembic via `deploy/deploy.sh`; `deploy/backup.sh` exists but is **not wired in** (no cron, no rclone configured — see checklist Accepted Risks)
  - [x] Existing staging/production documentation. — `docs/infrastructure.md`, `docs/deploy.md`, `docs/musehub-production-readiness-checklist.md` and its `production-readiness/` companion docs
- [x] Document the exact current staging deployment command and its inputs. — `bash deploy/push.sh staging`; builds via Docker, pushes to ECR with `crane`, triggers `deploy.sh` over SSM
- [x] Determine how the existing blue/green deployment works, including ports, health checks, active-slot selection, and rollback. — blue=1337/green=1338, `/healthz` gate, `musehub-set-slot` script writes the nginx upstream file, rollback via `IMAGE_TAG=<prev> bash deploy/push.sh <env>`
- [x] Identify every place where staging-specific account IDs, hostnames, IP addresses, paths, ports, repository names, or secrets are hard-coded. — ECR registry `992382692655...` is hard-coded in `deploy/push.sh`, `deploy/deploy.sh`, `deploy/bootstrap-instance.sh`; instance IDs hard-coded in `deploy/push.sh`, `deploy/bootstrap-instance.sh`, `.museagent.md`; see discovery report for full list

**Key discovery note (2026-08-23):** `docs/musehub-production-readiness-checklist.md` (dated 2026-08-04/08-08) already contains a much deeper, earlier production-readiness sweep than this TODO. It documents that the "prod" EC2 instance (`i-0855d6efe7fa1a49d`) referenced throughout `deploy/` scripts lives in the **old shared Nonproduction/Stori account** (`992382692655`), is **currently broken** (no IAM instance profile, `push.sh prod` fails with `InvalidInstanceId`), and that DNS/TLS actually run through **Cloudflare** (DNS hosting + Origin Certificate), not Namecheap/Let's Encrypt/ACM as this TODO's Phase 5 assumes. See the full discovery report delivered separately for details before Phase 2 proceeds.

### AWS inventory in MuseHub Nonproduction

- [x] Inventory relevant resources in every used region, starting with `us-east-1`:
  - [x] VPCs, subnets, route tables, internet gateways, NAT gateways, and network ACLs. — single default VPC (`vpc-0552bdd377de8a9a8`, `172.31.0.0/16`); no custom networking
  - [x] EC2 instances, AMIs, launch templates, volumes, snapshots, and Elastic IPs. — `musehub-staging` (`i-07547cd20bee2dea5`, t3.medium, running, EIP `23.22.27.39`) and `musehub-prod` (`i-0855d6efe7fa1a49d`, t3.small, running, EIP `98.89.99.211`); no launch templates/ASGs
  - [x] Security groups and inbound/outbound rules. — `musehub-sg` (`sg-05815872537fcfe76`): 443 open to Cloudflare IP ranges only, as documented. **Finding, resolved 2026-08-23**: an additional, undocumented rule opened TCP 1337 directly to a single IP (`98.51.144.179/32`), bypassing nginx/Cloudflare entirely. Confirmed dead (both `docker-compose.yml` and `deploy/deploy.sh` bind the app container to `127.0.0.1` only, so nothing was listening on the public interface for it) and revoked (`sgr-03be7da55eece078f`). Only port 443 remains open.
  - [x] IAM roles, instance profiles, policies, and deployment identities. — Users: `musehub-infra`, `musehub-app`, `stori-assets-app` (all expected), plus **`s3-migration-temp`** — created 2026-02-10 with `AmazonS3FullAccess`, used exactly once the day it was created (R2 migration, per `deploy/migrate_r2_keys.py`) and never since. **Resolved 2026-08-23**: access key deactivated (reversible — reactivate with `aws iam update-access-key --status Active` if ever needed). Roles: `musehub-ec2-ssm`, `rds-monitoring-role`, `tus-image-processing-1-role-7dl2zvb1` (Stori-owned, not ours), plus the new SSO role.
  - [x] ECR repositories, image tags/digests, scan settings, and lifecycle policies. — one repo `musehub/musehub`, scan-on-push enabled, AES256 encryption, mutable tags, ~420 images accumulated with no pruning. **Resolved 2026-08-23**: lifecycle policy applied — expire untagged images after 14 days, keep only the most recent 60 tagged images.
  - [x] S3 buckets, policies, encryption, lifecycle rules, and CORS settings. — `musehub-releases` (ours) and `stori-assets-992382692655-useast2` (Stori's, confirms shared-account risk from the checklist); no bucket found for DB backups.
  - [x] Databases, caches, queues, and other stateful services. — **Finding: a managed RDS Postgres instance exists (`musehub-staging-db`, db.t3.micro, 20GB, single-AZ, not publicly accessible, 7-day automated backups)** that isn't mentioned in any doc. The app's actual `DATABASE_URL` points at the containerized `postgres:5432` service per `docs/infrastructure.md`, so this RDS instance is very likely orphaned/unused — needs Gabriel to confirm before considering decommission (it's actively costing money and, confusingly, *is* backed up, unlike the real database). No Lambda, DynamoDB, SQS, or SNS found.
  - [x] Systems Manager parameters and Secrets Manager secret names only—not their values. — Secrets Manager: empty. SSM: `/musehub/staging/PACK_WORKER_URL`, `/musehub/staging/WEBHOOK_SECRET_KEY`, `/musehub/staging/WORKER_INTERNAL_KEY` — a partial, abandoned attempt at the SSM secrets pipeline (missing `DB_PASSWORD`/`RUNNER_TOKEN`), consistent with `docs/infrastructure.md` noting `deploy/secrets.sh` "is not the live path."
  - [x] CloudWatch log groups, alarms, dashboards, and retention settings. — one log group `/musehub/staging`, **retention set to Never Expire**, already 1.9GB stored; zero alarms configured anywhere (`cloudwatch-alerts.sh` has evidently never been run for real — matches checklist Section 12 status).
  - [x] Load balancers, target groups, listeners, and health checks, if present. — none; direct EC2 + Elastic IP + nginx, as documented.
  - [x] DNS records hosted in AWS, if any. — none; DNS is Cloudflare-hosted (Route 53 not used).
  - [x] Backup plans, retention policies, and restore procedures. — none for the real (containerized) database; confirms the checklist's highest-priority open risk.
- [x] Determine whether the existing `musehub-prod` EC2 resource inside Nonproduction is active, obsolete, or safe to retire later. Do not alter it during discovery. — **running but non-functional**: it's up and billing, but per `docs/infrastructure.md` has no IAM instance profile attached, so SSM can't reach it and `push.sh prod` fails outright. Not touched during this discovery. Disposition (fix in place vs. retire in favor of the new Production account) is a Phase 2/7 decision for Gabriel.

### AWS starting state in MuseHub Production

- [x] Confirmed clean: default VPC only (`vpc-038dc9e464d82a05d`), zero EC2 instances, zero S3 buckets, zero ECR repositories, zero IAM users, zero Elastic IPs, zero RDS instances. Only roles present are the SSO `AdministratorAccess` role and the standard AWS Organizations `OrganizationAccountAccessRole`. Ready for a from-scratch build.

### Staging host inspection

- [x] Record the staging host's:
  - [x] OS and architecture. — Ubuntu 22.04.5 LTS, x86_64
  - [x] Docker and Compose versions. — Docker 29.3.1, Compose v5.1.1
  - [x] Reverse proxy and configuration paths. — nginx, `/etc/nginx/sites-available/musehub-staging` (only site enabled); Cloudflare Origin Cert at `/etc/ssl/cloudflare/origin.{pem,key}` with correct permissions (644/640)
  - [x] Listening ports and firewall rules. — publicly: 443 (nginx) and **22 (sshd)** on `0.0.0.0`/`[::]`. **Note**: the security group has no port-22 ingress rule, so SSH isn't reachable from the internet today — but sshd is running and bound to all interfaces as a live daemon, meaning the SG is the *only* thing preventing exposure. Worth disabling sshd outright on the new Production host (SSM-only, per the existing "no SSH" design intent) rather than relying solely on the SG. App ports 1337/1338 confirmed bound to `127.0.0.1` only (docker-proxy), matching `docker-compose.yml`/`deploy.sh` — this is what made the now-removed SG rule for port 1337 a dead rule.
  - [x] systemd units, scheduled jobs, and deployment hooks. — no user crontab (`no crontab for ubuntu`); `/etc/cron.d/` has only default OS entries (`certbot`, `e2scrub_all`) — **`deploy/backup.sh` is confirmed never installed**. A leftover `certbot.timer` fires every ~14h from the default Ubuntu certbot package install but has nothing to renew (Cloudflare handles TLS) — harmless noise, not a real risk.
  - [x] Environment-file and secret paths without displaying values. — `/opt/musehub/.env`, root-owned, **644 permissions (world-readable)** — should be tightened to 600/640 on the new Production host.
  - [x] Certbot/Let's Encrypt configuration and renewal timers, if used. — not used for TLS (Cloudflare Origin Cert instead); default certbot package/timer present but inert.
  - [x] Log locations and rotation policy. — no dedicated log files; app logs go to stdout, captured by the `awslogs` Docker log driver into CloudWatch (`/musehub/staging`, never-expire retention — see Phase 1 AWS inventory finding above).
- [x] Map all external dependencies: databases, object storage, email, OAuth, webhooks, APIs, payment services, analytics, and DNS. — containerized Postgres (not the orphaned RDS instance), Cloudflare R2 for object storage, Cloudflare for DNS/edge/TLS; no email/payment/analytics dependencies identified in this pass.
- [x] Produce a staging inventory and dependency map before provisioning Production. — see this section plus the Phase 1 AWS inventory above.

**Additional findings from host inspection (2026-08-24), not yet acted on:**
- **`/opt/musehub` on the staging host contains the entire repository** — `node_modules/`, `tests/`, `tools/`, `.mypy_cache/`, `.cursor/`, and large standalone design docs (`MUSEHUB_V2.md` at ~100KB, several `*_REDESIGN.md` files) — not just the running app. This is a leftover from the pre-image-based deploy method (the old `rsync`-based setup scripts), and contradicts `docs/deploy.md`'s current claim that deploys are "image-based via ECR. No SSH, no rsync, no code on the instance after provisioning." It's dead weight (likely a meaningful share of the 14GB/20GB, 70% disk usage observed) but not a safety issue — recommend a separate, deliberate cleanup task rather than doing it mid-discovery.
- **The one existing manual backup is empty**: `/opt/backups/musehub/musehub_staging_manual_20260613_003724.sql.gz` is **20 bytes** — independently reproduces the checklist's flagged finding that even the sole manual dump ever taken is unusable. Reinforces that this is the single highest-priority risk to close before or immediately after production launch.
- Disk usage on staging is already at 70% (14G/20G) with no automated cleanup beyond the on-host image prune (last 3) — worth sizing the Production EBS volume larger and/or cleaning up the stray repo copy before it becomes a real constraint.

## Phase 2 — Decide the production design

- [ ] Decide whether the current repository already contains usable infrastructure as code.
- [ ] If it does, adapt it with isolated production variables and state.
- [ ] If it does not, capture the required production infrastructure as code rather than relying on undocumented console-only creation.
- [ ] Choose the production ingress/TLS topology:
  - [ ] Direct EC2 + Elastic IP + reverse proxy + Let's Encrypt; or
  - [ ] Application Load Balancer + ACM certificate.
- [ ] Confirm the production hostname(s), such as the primary domain and/or `www`/`app` subdomain.
- [ ] Decide how application images are promoted:
  - [ ] Preferred: build once and promote/deploy an immutable ECR image digest.
  - [ ] Decide whether Production uses a dedicated ECR repository or controlled cross-account access.
- [ ] Decide how state and data will be handled:
  - [ ] Fresh production database or migration from an existing source.
  - [ ] Asset/object migration requirements.
  - [ ] Backup, restore, and rollback requirements.
- [ ] Establish resource naming and tagging conventions, including at least `Environment=production`, `Application=musehub`, and ownership/cost tags.
- [ ] Explicitly identify which staging settings must differ in Production: domains, credentials, OAuth callbacks, email/webhooks, storage, database, logging, and rate limits.

## Phase 3 — Provision the production baseline

- [ ] Create or confirm an AWS Budget and cost alerts for MuseHub Production.
- [ ] Confirm CloudTrail/account audit coverage and appropriate log retention.
- [ ] Provision the production network or intentionally use the account default VPC only if documented and accepted.
- [ ] Provision production security groups:
  - [ ] Public access only on required ingress ports, normally `80` and `443`.
  - [ ] Restrict SSH to known sources or prefer AWS Systems Manager Session Manager.
  - [ ] Restrict database and internal service access to required security groups.
- [ ] Create the compute layer using the chosen architecture.
- [ ] Encrypt storage and configure snapshots/backups.
- [ ] Allocate an Elastic IP or provision a load balancer, according to the chosen ingress design.
- [ ] Create least-privilege production IAM roles and instance profiles.
- [ ] Create production ECR repositories/access and lifecycle policies.
- [ ] Create production parameters and secrets separately; never copy staging secrets wholesale.
- [ ] Configure CloudWatch logs, retention, alarms, and basic health monitoring.
- [ ] Verify the instance can pull the approved image and access only its required production services.

## Phase 4 — Deploy and validate before public DNS

- [ ] Select an immutable application image digest known to work in staging.
- [ ] Deploy it to Production using the same blue/green pattern where practical.
- [ ] Run database migrations only after:
  - [ ] Taking a backup/snapshot.
  - [ ] Reviewing forward and rollback behavior.
  - [ ] Confirming the target is the production database.
- [ ] Test Production without changing public DNS using an Elastic IP, load-balancer hostname, temporary hostname, or local hosts-file override.
- [ ] Validate:
  - [ ] Application health endpoint.
  - [ ] Home page and critical routes.
  - [ ] Authentication and authorization.
  - [ ] Database reads/writes.
  - [ ] Uploads and media delivery.
  - [ ] Email and webhook integrations.
  - [ ] Background jobs/queues.
  - [ ] Logs and alarms.
  - [ ] Restart and reboot recovery.
- [ ] Run a rollback rehearsal to the previous production slot/image.

## Phase 5 — Namecheap DNS and TLS

- [ ] Confirm the final production hostname(s) and current DNS records in Namecheap.
- [ ] Lower DNS TTL in advance of cutover if appropriate.
- [ ] Create the appropriate record:
  - [ ] `A` record to an Elastic IP for direct EC2 ingress; or
  - [ ] `CNAME`/alias-compatible record to a load balancer hostname.
- [ ] Verify propagation with `dig`/`nslookup` from more than one resolver.
- [ ] Complete TLS according to the chosen topology:
  - [ ] Direct EC2: issue a Let's Encrypt certificate with Certbot only after DNS resolves correctly and ports `80`/`443` are reachable.
  - [ ] Load balancer: request and validate an ACM certificate, then attach it to the HTTPS listener.
- [ ] Configure HTTP-to-HTTPS redirection.
- [ ] Validate the full certificate chain, hostname coverage, and expiration.
- [ ] For Let's Encrypt, confirm the renewal timer and run `certbot renew --dry-run`.
- [ ] Re-run the complete production smoke test over the final HTTPS hostname.

## Phase 6 — Make production deployment repeatable

- [ ] Update the deployment tooling to require an explicit environment, such as `staging` or `production`.
- [ ] Add expected-account guards:
  - [ ] Staging/nonproduction requires `992382692655`.
  - [ ] Production requires `672469410277`.
- [ ] Require an explicit production release version/tag or immutable image digest.
- [ ] Make production deployment use the production SSO profile and production-specific configuration.
- [ ] Preserve the working blue/green deployment behavior and health gate.
- [ ] Add a documented one-command or short-command rollback.
- [ ] Prevent accidental use of staging secrets, hostnames, databases, or buckets in Production.
- [ ] Ensure deploy logs clearly state the target account, region, environment, host, image digest, and resulting active slot without printing secrets.
- [ ] Add CI/CD later only after the manual scripted flow is deterministic and verified.

## Phase 7 — Cutover and post-launch

- [ ] Capture the pre-cutover production backup/snapshot and rollback point.
- [ ] Confirm owners, decision-maker, and rollback trigger.
- [ ] Route production DNS only after all acceptance checks pass.
- [ ] Monitor application health, error logs, CPU/memory/disk, database health, and external integrations during the launch window.
- [ ] Verify AWS billing/cost telemetry after resources have run long enough to report.
- [ ] Raise DNS TTL after the environment is stable, if it was lowered.
- [ ] Document the deployed architecture, resource identifiers, operational commands, backup/restore procedure, and incident contacts.
- [ ] Decide whether and when to retire the obsolete `musehub-prod` resource in Nonproduction. Treat retirement as a separate, approved task.

## Acceptance criteria

- [ ] Production resources reside only in AWS account `672469410277`.
- [ ] Staging remains functional and unchanged throughout the launch.
- [ ] Engineers authenticate through Google Workspace and IAM Identity Center; no root access keys are used.
- [ ] Production is accessible on its intended HTTPS hostname with a valid, renewable certificate.
- [ ] Production secrets and state are isolated from staging.
- [ ] A known-good image can be deployed to Production through a documented, guarded command.
- [ ] Rollback is documented and successfully rehearsed.
- [ ] Monitoring, logs, backups, budgets, and basic alerts are active.
- [ ] Repository documentation accurately describes the final architecture and operating procedure.

## Decisions that must be resolved during discovery

- [ ] Exact production hostname(s).
- [ ] Direct EC2/Let's Encrypt versus load balancer/ACM.
- [ ] Existing infrastructure-as-code strategy and state location.
- [ ] Production database and data-migration plan.
- [ ] ECR promotion/cross-account strategy.
- [ ] Production instance size and scaling requirements.
- [ ] Required OAuth, email, webhook, storage, and third-party production configuration.
- [ ] Fate of the old `musehub-prod` resource in Nonproduction.

## Agent execution checkpoints

An implementation agent should stop and report at these gates:

1. **After access verification:** report both caller identities and profiles.
2. **After read-only discovery:** deliver the inventory, dependency map, proposed design, cost-impacting resources, and unresolved questions.
3. **Before provisioning:** obtain approval for the exact production plan.
4. **Before any state/data migration:** provide backup and rollback procedures.
5. **Before DNS changes:** provide validation results and the DNS/TLS cutover plan.
6. **After launch:** provide smoke-test evidence, deployed image digest, resource summary, and rollback command.