# MuseHub Production Launch — Discovery Report

**Status:** Discovery complete (Phases 1–3) — awaiting approval before any provisioning
**Date:** 2026-08-24
**Scope:** Reproduce the working staging architecture in the dedicated MuseHub Production AWS account (`672469410277`), configure production DNS/TLS, and make production deploys as safe and repeatable as staging's.
**Companion documents:** [`musehub-production-infrastructure-launch-todo.md`](../production-readiness/musehub-production-infrastructure-launch-todo.md) (working checklist, kept up to date throughout this discovery), [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md) (broader, earlier 19-section readiness sweep), [`musehub-cloud-identity-and-aws-operating-model.md`](musehub-cloud-identity-and-aws-operating-model.md) (workforce identity model).

No AWS resource, DNS record, certificate, or credential has been created, changed, or deleted in the Production account. Three narrowly-scoped, pre-approved hygiene fixes were applied in Nonproduction (detailed in §6). Everything else in this report is read-only findings and recommendations pending your approval.

---

## 1. Current-state architecture

```
Google Workspace (musehub.ai)
  │  SAML + SCIM
  ▼
AWS IAM Identity Center (MuseHub-Workforce, us-east-1)
  │
  ▼
AWS Organization (o-qmi72zxjxd)
  ├── MuseHub Management       (797661577790) — governance, Identity Center, billing
  ├── MuseHub Nonproduction    (992382692655) — staging + legacy "prod", SHARED with Stori (unrelated project)
  └── MuseHub Production       (672469410277) — new, currently empty
```

## 1a. Prior research already done — reuse, don't redo

Before this discovery pass, an earlier, much deeper 19-section readiness sweep (2026-08-04 to 08-08) already covered most of this ground in detail, one companion doc per section under [`docs/production-readiness/`](../production-readiness/). That sweep's own working method was explicitly "document gaps now, build later, once every section is surveyed" — the same discovery-before-provisioning approach this launch is following. Below is what it already settled, so none of it needs re-investigating:

| Companion doc | Already established |
|---|---|
| [`00-inventory-and-decisions.md`](../production-readiness/00-inventory-and-decisions.md) | Architecture is "materially simpler than assumed" — 2 raw EC2 + Docker Compose, no ECS/ALB/RDS (see §6 below for a correction to this). Object storage is confirmed Cloudflare R2 (not a local volume — `docs/infrastructure.md` was stale). Secrets are hand-provisioned `.env` with real R2 keys the docs never mentioned. Single AWS account (shared with Stori) accepted as tech debt — **since superseded**, we've since built the 3-account Organization split. |
| [`01-ownership-and-access.md`](../production-readiness/01-ownership-and-access.md) | Aaron already has MuseHub `write` collaborator access (code, not infra). No IAM Identity Center visibility existed at the time — **since resolved**, this session built it. "Aaron operates independently" test explicitly cannot pass until Identity Center + individual credentials exist — this session's SSO work is the direct fix. |
| [`02-aws-account-structure.md`](../production-readiness/02-aws-account-structure.md) | Single-account decision recorded with an explicit revisit trigger (if Stori/MuseHub security requirements diverge, or billing attribution gets hard). **Superseded** by the Organization split. |
| [`03-identity-and-permissions.md`](../production-readiness/03-identity-and-permissions.md) | Recommended target: `Administrator` (Gabriel) + `Operator` (Gabriel+Aaron) permission sets — later superseded by the 3-tier model in `runbook-deploy-onboarding.md` (`Administrator`/`ProductionOperator`/`StagingOperator`, scoped for all 5 team members) once the team grew. That 3-tier model is what this session's `AWS Workforce` SCIM/Identity Center work implements. |
| [`04-iac.md`](../production-readiness/04-iac.md) | Terraform/OpenTofu already recommended (not just "an option") specifically because infra now spans **both AWS and Cloudflare** (DNS/TLS/R2) and Terraform has a mature provider for each. Tool choice is settled; only timing (now vs. later) is open. |
| [`05-production-architecture.md`](../production-readiness/05-production-architecture.md) | **The app itself is already well-hardened** — see §9 below for the specifics folded into the recommendation. Documents 5 accepted SPOFs explicitly (single instance, single DB, single AWS account, single Cloudflare account, single human deployer). |
| [`06-dns-domain-tls.md`](../production-readiness/06-dns-domain-tls.md) | Live-verified via `dig`: Cloudflare is authoritative for the whole zone (NS records themselves point at Cloudflare), not just proxying. Recommends staying on Cloudflare DNS over migrating to Route 53. Canonical hostname (root vs. `www`) confirmed still undecided. |
| [`07-secrets-and-config.md`](../production-readiness/07-secrets-and-config.md) | Full secrets inventory already compiled (table of all 6 secrets and where each lives). `secrets.sh`'s SSM path is real code, just short 4 of 7 required parameters — the concrete work item is populating those, not building anything new. |
| [`08-database-backups-migrations.md`](../production-readiness/08-database-backups-migrations.md) | Independently reached the same "no usable backup" finding this pass re-confirmed live. **New finding this pass, missed by this doc**: the orphaned `musehub-staging-db` RDS instance (§6/§8 below) — this companion doc asserted "no RDS" in its architecture summary, which this session's live AWS query shows was already inaccurate even then (the RDS instance was created 2026-04-05, before this doc's 2026-08-04 date). Also flags a real migration-failure-handling risk (`alembic stamp --purge head` on failure could mask a genuinely broken migration) and recommends splitting the single Postgres role (app/migrations/admin currently share one role). |
| [`09-ci-pipeline.md`](../production-readiness/09-ci-pipeline.md) | Important scope correction: MuseHub-native branch protection, required status checks, and CODEOWNERS-equivalent **don't exist as product features at all** — this is bigger than a deployment-config gap, it's unbuilt product work. Directly relevant to this launch's Phase 6 goal of "repeatable deploys." The background job queue itself (`musehub_jobs.py`) is confirmed well-built and idempotent (`SELECT ... FOR UPDATE SKIP LOCKED`). |
| [`10-11-automated-deployment.md`](../production-readiness/10-11-automated-deployment.md) | Already read in this session's Phase 1 — item-by-item automation gap analysis; blue/green + health-gate + rollback command already exist manually, nothing else is automated. |
| [`12-observability.md`](../production-readiness/12-observability.md) | **Logging is much further along than it first appears**: `/musehub/staging` already has structured JSON logs with request IDs, and a `PiiFilter` already scrubs tokens/passwords/secrets before they're logged. What's missing is purely alarms/dashboards on top of data that already exists — not new logging infrastructure. |
| [`13-security-monitoring.md`](../production-readiness/13-security-monitoring.md) | At the time, ECR image scanning was **not enabled** and `musehub-releases`'s Block Public Access was fully disabled (assessed as intentional, given it serves the public installer). **Update this pass**: ECR scan-on-push is now confirmed enabled (§3 of this report) — resolved since that doc was written. CloudTrail/GuardDuty/Security Hub/Access Analyzer state still unconfirmed (needs broader IAM read access or Gabriel checking directly). |
| [`14-application-security.md`](../production-readiness/14-application-security.md) | Real code-level security review already done: no SQL injection surface, path traversal rejected, archive-bomb protection, MSign requires fresh signed auth for privileged actions, CORS fails closed, thorough upload quotas. One real finding (`/_debug/memory` exposed without the debug guard) was **found and fixed in that same session**. |
| [`15-cost-controls.md`](../production-readiness/15-cost-controls.md) | Baseline cost already documented (~$35/mo today). Tagging confirmed inconsistent/absent across every resource — worth getting right from day one on the new Production resources rather than retrofitting. |
| [`16-19-docs-testing-cutover-postlaunch.md`](../production-readiness/16-19-docs-testing-cutover-postlaunch.md) | Test suite confirmed to collect cleanly at **7,028 tests**, no collection errors (pass/fail not run). Cutover/post-launch sections correctly deferred as blocked on everything above. |

Today, both "staging" and "prod" run as two EC2 instances **inside the same Nonproduction account**, alongside unrelated Stori resources:

```
Nonproduction (992382692655), us-east-1, default VPC (172.31.0.0/16)
  ├── EC2 musehub-staging (i-07547cd20bee2dea5, t3.medium) ✅ working
  │     nginx :443 (Cloudflare Origin Cert) → 127.0.0.1:1337/1338 (blue/green)
  │     containers: musehub-{blue,green}, musehub-worker, postgres, musehub-runner
  │     volumes: musehub_data, musehub_postgres_data, musehub_runner_workspace
  ├── EC2 musehub-prod (i-0855d6efe7fa1a49d, t3.small) ⚠️ running but broken
  │     no IAM instance profile → SSM unreachable → `push.sh prod` fails outright
  ├── RDS musehub-staging-db (db.t3.micro) — orphaned, not referenced by the app
  ├── ECR musehub/musehub (single repo, both environments' images)
  └── S3 musehub-releases (muse CLI tarballs) + stori-assets-... (not ours)

Cloudflare (musehub.ai)
  ├── DNS (nameservers point here — Namecheap is registrar only)
  ├── Edge TLS termination + Origin Certificates (15-yr, not Let's Encrypt/ACM)
  └── R2 object storage (musehub-staging / musehub-prod buckets — separate from AWS)
```

**Production account (672469410277) currently contains nothing** — default VPC only, confirmed in §4.

---

## 2. Relevant repository files and what each controls

| File | Controls |
|---|---|
| [`deploy/push.sh`](../../deploy/push.sh) | Build → `crane` push to ECR → SSM-triggers `deploy.sh`. Hard-codes the ECR registry (`992382692655...`) and both instance IDs. Currently only knows `staging`/`prod` as targets, both pointing into Nonproduction. |
| [`deploy/deploy.sh`](../../deploy/deploy.sh) | Blue/green swap logic on the instance: pulls image, runs Alembic + schema-gate, starts new slot, health-gates on `/healthz`, flips nginx via `musehub-set-slot`, stops old slot, restarts worker, prunes local images (keeps 3). |
| [`deploy/set-active-slot.sh`](../../deploy/set-active-slot.sh), [`deploy/nginx-cf.conf`](../../deploy/nginx-cf.conf) | nginx upstream flip; per-route timeouts; Cloudflare IP allowlist + real-IP restoration; rate limiting. TLS is **Cloudflare Origin Certificate**, not Let's Encrypt or ACM. |
| [`deploy/aws-provision.sh`](../../deploy/aws-provision.sh), [`deploy/aws-provision-staging.sh`](../../deploy/aws-provision-staging.sh) | Original console/CLI-driven EC2 + EIP provisioning. No IaC exists anywhere in the repo. |
| [`deploy/setup-ec2.sh`](../../deploy/setup-ec2.sh), [`deploy/setup-ec2-staging.sh`](../../deploy/setup-ec2-staging.sh) | One-time host bootstrap: Docker, nginx, Cloudflare cert install, hand-generates `.env` with `openssl rand`/Fernet secrets on first run. |
| [`deploy/bootstrap-instance.sh`](../../deploy/bootstrap-instance.sh) | Installs AWS CLI v2 on a fresh instance via SSM so `deploy.sh` can auth to ECR. Hard-codes both instance IDs and the ECR registry. |
| [`deploy/backup.sh`](../../deploy/backup.sh) | pg_dump + optional R2/rclone upload + 14-day local retention. **Exists but was never installed** (confirmed live — no cron, no rclone). |
| [`deploy/cloudwatch-alerts.sh`](../../deploy/cloudwatch-alerts.sh) | Log group, metric filters, alarms, SNS. **Defaults `AWS_REGION=eu-west-1`** — must always pass `--region us-east-1` explicitly. Never actually run (confirmed: zero alarms exist). |
| [`Dockerfile`](../../Dockerfile), [`docker-compose.yml`](../../docker-compose.yml), [`docker-compose.override.yml`](../../docker-compose.override.yml) | App image + local dev stack (musehub, postgres:16, musehub-runner). Ports bound to `127.0.0.1` only — nginx is the only public entry point. |
| [`docs/infrastructure.md`](../infrastructure.md) | Living infra reference. Its own overview line ("Let's Encrypt TLS") is stale/self-contradictory against the rest of the same document. |
| [`docs/deploy.md`](../deploy.md) | Deploy runbook; states deploys are "image-based via ECR — no SSH, no rsync, no code on the instance" — **contradicted by live host inspection**, see §6. |
| [`docs/musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md) | A much deeper, earlier (2026-08-04/08-08) 19-section readiness sweep covering identity, IaC, architecture, DNS/TLS, backups, CI/CD, observability, security, cost. Most of its Section 2/3 status is now out of date (we've since built the Organization split and Identity Center workforce model it called "deferred"/"not started"). |
| [`docs/production-readiness/runbook-deploy-onboarding.md`](../production-readiness/runbook-deploy-onboarding.md) | Team access tiers: Gabriel/Aaron full access, interns (Sol/Toby/Ben) staging-only, pending individual SSO identities. |
| [`.museagent.md`](../../.museagent.md) | Local dev container restart/schema-drift runbook; also the source of the two instance IDs referenced throughout. |

---

## 3. Nonproduction AWS inventory (992382692655, us-east-1)

| Resource | Detail |
|---|---|
| VPC | Default only, `vpc-0552bdd377de8a9a8`, `172.31.0.0/16` |
| EC2 | `musehub-staging` (t3.medium, running, EIP `23.22.27.39`); `musehub-prod` (t3.small, running but broken, EIP `98.89.99.211`) |
| Security group | `musehub-sg` (`sg-05815872537fcfe76`) — now just port 443 from Cloudflare ranges (port-1337 hole removed, see §6) |
| ECR | `musehub/musehub`, scan-on-push, AES256, mutable tags, ~420 images (lifecycle policy now applied, see §6) |
| S3 | `musehub-releases` (ours, active — muse CLI tarballs); `stori-assets-992382692655-useast2` (not ours) |
| RDS | `musehub-staging-db` (db.t3.micro, 20GB, single-AZ, private, 7-day backups) — **orphaned**, app doesn't reference it |
| IAM users | `musehub-infra`, `musehub-app`, `stori-assets-app` (expected); `s3-migration-temp` (deactivated, see §6) |
| IAM roles | `musehub-ec2-ssm`, `rds-monitoring-role`, `tus-image-processing-1-role-*` (Stori's), plus the new SSO role |
| CloudWatch | One log group `/musehub/staging`, was never-expire retention, 1.9GB stored; zero alarms |
| Secrets/SSM | Secrets Manager empty; SSM has 3 of 5 expected `/musehub/staging/*` parameters — a stalled, abandoned migration off hand-provisioned `.env` files |
| CloudTrail | None configured |
| Other | No Lambda, DynamoDB, SQS, SNS, load balancers, or Route 53 zones |

---

## 4. Production account starting state (672469410277, us-east-1)

Confirmed clean: default VPC only (`vpc-038dc9e464d82a05d`), zero EC2/S3/ECR/IAM-users/EIPs/RDS. Only roles present are the SSO `AdministratorAccess` role and the standard AWS Organizations `OrganizationAccountAccessRole`. This is a genuine from-scratch build — nothing to import, nothing to reconcile.

---

## 5. Staging deployment flow (confirmed live, matches docs)

```
Local machine:
  1. docker build --platform linux/amd64
  2. docker save → crane push → ECR (992382692655.../musehub/musehub:<commit>-<timestamp>)
  3. aws ssm send-command → writes deploy.sh/nginx-cf.conf/set-active-slot.sh to instance, then runs deploy.sh

Instance (via SSM, no SSH):
  4. docker login (ecr get-login-password) → docker pull
  5. Alembic upgrade head (with stamp-and-retry fallback) + schema-gate hard-fail check
  6. docker run new slot (blue↔green, 127.0.0.1 only)
  7. curl /healthz until healthy (30 × 2s)
  8. musehub-set-slot → nginx -s reload (zero-downtime flip)
  9. docker rm old slot; restart worker; prune local images (keep 3)
```

Rollback: `IMAGE_TAG=<previous-tag> bash deploy/push.sh <env>` (skips build/push, redeploys a known-good tag). **Never actually tested**, per the readiness checklist.

Live-verified via SSM (Phase 3): Ubuntu 22.04.5, Docker 29.3.1, Compose v5.1.1, active slot `green`/1338, TLS via Cloudflare Origin Cert, no cron/systemd job runs backups, `.env` is `644`.

---

## 6. Actions already taken during discovery (approved individually, all reversible)

| # | Action | Why | Rollback |
|---|---|---|---|
| 1 | Revoked SG rule allowing TCP 1337 from `98.51.144.179/32` on `sg-05815872537fcfe76` | Confirmed dead — app binds to `127.0.0.1` only; rule was a landmine, not an active exposure | Re-add the same rule if ever needed |
| 2 | Deactivated access key for IAM user `s3-migration-temp` | One-time R2 migration credential (`AmazonS3FullAccess`), used once on 2026-02-10, never since | `aws iam update-access-key --status Active` |
| 3 | Applied ECR lifecycle policy to `musehub/musehub` | ~420 untagged/unpruned images with no cleanup; policy expires untagged >14 days and caps tagged history at 60 | `aws ecr delete-lifecycle-policy` (doesn't restore already-expired images, but nothing was touched immediately) |

**Deferred, documented, not acted on:** the orphaned `musehub-staging-db` RDS instance — holding off until the production data-store setup is confirmed, per your call.

### Other findings surfaced but not acted on (no urgency, flagged for later)

- `sshd` listens on `0.0.0.0:22` on staging; only the (port-22-less) security group prevents exposure. Recommend disabling sshd outright on the new Production host rather than relying solely on the SG.
- `/opt/musehub` on the staging host contains the **entire repository** — `node_modules`, `tests`, `.cursor`, `.mypy_cache`, ~150KB of standalone design docs — a leftover from the pre-image-based (`rsync`) deploy method, contradicting `docs/deploy.md`'s "no code on the instance" claim. Likely contributing to the 70% (14G/20G) disk usage. Cleanup candidate, not a safety issue.
- `/opt/musehub/.env` is `644` (world-readable); should be `600`/`640` on Production.
- The one manual DB backup ever taken (`musehub_staging_manual_20260613_003724.sql.gz`) is **20 bytes** — independently confirms the readiness checklist's top-flagged risk that no usable backup exists.
- A leftover `certbot.timer` fires every ~14h with nothing to renew (Cloudflare handles TLS) — harmless noise.

---

## 7. External dependencies

| Dependency | Provider | Notes |
|---|---|---|
| Database | Self-hosted Postgres 16 (Docker container) | Not the orphaned RDS instance |
| Object storage | Cloudflare R2 | Separate `musehub-staging`/`musehub-prod` buckets; presigned upload/download, instance never buffers full mpacks |
| DNS | Cloudflare | Namecheap is registrar only |
| TLS | Cloudflare Origin Certificate (15-yr) | Full (Strict) mode; not Let's Encrypt, not ACM |
| CDN/Edge/DDoS/WAF | Cloudflare | SG restricts port 443 to published Cloudflare IP ranges |
| CI/CD | None | Every deploy is a human running `push.sh` from a laptop |
| Muse CLI distribution | S3 (`musehub-releases`) served through the app, not directly public | Still-active use of real AWS S3, separate from R2 |
| Secrets | Hand-provisioned `.env` per instance | Not Secrets Manager/SSM (a partial, stalled migration attempt exists in SSM) |
| Cloudflare Worker | `WORKER_INTERNAL_KEY` shared secret, `musehub/worker.py` | Callback path from a Cloudflare Worker into MuseHub; not independently reviewed for SSRF exposure per the prior security pass — flagged there as worth a closer look |

No email, payment, or analytics integrations were identified in this pass (confirmed independently by both this discovery and the prior sweep's `00-inventory-and-decisions.md`).

---

## 8. Risks and blockers

**Blocking or near-blocking for a confident launch:**
1. **No usable database backup exists anywhere** (pre-existing top risk, independently reproduced this pass — the one manual dump is 20 bytes). Should be fixed before or immediately alongside production launch, not after.
2. **Rollback has never been tested.** The command exists; it has not been exercised.
3. **No IaC.** Every existing resource was created by hand or by ad-hoc shell script. Reproducing into Production means either writing Terraform/OpenTofu now (bigger upfront lift, much safer long-term) or scripting it the same ad-hoc way a second time (faster, repeats the same risk).
4. **Canonical production hostname is still undecided** (root `musehub.ai` vs. a subdomain) — blocks Phase 5 (DNS/TLS) planning until you confirm.
5. **Cross-account ECR strategy undecided** — Production needs either its own ECR repo (clean isolation, requires re-pushing images) or cross-account pull access to the existing Nonproduction repo (faster, extends the shared-account blast radius that's already a flagged risk).

**Non-blocking, worth addressing during the build:**
6. Deploys still use one shared, long-lived IAM user credential (`musehub-infra`) rather than per-person/per-environment roles — the SSO work this session is a first step toward fixing this, but `push.sh` itself hasn't been migrated off the shared key yet.
7. Single-instance, single-AZ architecture is a known, accepted SPOF (documented in the readiness checklist).
8. No CloudTrail, no budgets/alerts, no centralized monitoring beyond raw CloudWatch logs.
9. **Single Postgres role for everything** — app runtime, Alembic migrations, and admin `psql` access all share one role. An app-level SQL injection (however unlikely, given no raw-SQL surface was found) would carry migration-level privileges. Flagged in `08-database-backups-migrations.md` as worth closing before launch.
10. **Migration failure-handling could mask a real break**: `deploy.sh`'s fallback on `alembic upgrade head` failure is `alembic stamp --purge head` + retry — intended for stale-revision-ID drift, but indistinguishable from a genuinely broken migration by the script's own logic. The follow-up schema-parity gate is the actual safety net; worth documenting that reliance explicitly rather than assuming the stamp-and-retry itself is safe.
11. **MuseHub-native CI/CD platform features don't exist yet** — no branch protection, no required status checks on `merge_proposal`, no CODEOWNERS-equivalent. This directly affects Phase 6's "repeatable deploys" goal: adding account-ID guards and a production-confirmation step to `push.sh` is achievable now, but gating production deploys on passing checks isn't possible until these are built as product features.
12. **Resource tagging is inconsistent/absent everywhere** (no `Owner`/`Managed-by`/`Cost-center` tags on any resource) — cheap to get right on Production from day one, much harder to retrofit later, and it's the only real path to cost-attributing the shared Stori/MuseHub account.

---

## 9. Recommended target production architecture

**Starting point worth stating plainly: the application itself is already well-hardened**, per the prior code-level security review (`05-production-architecture.md`, `14-application-security.md`) — real dependency-checking health checks, per-route rate limiting, bot throttling that exempts only authenticated traffic, security headers + HSTS + CSP already set, no SQL injection surface, path-traversal rejection, archive-bomb protection, MSign requiring fresh signed auth for privileged calls, CORS failing closed, and deliberate upload quotas. Structured JSON logging with PII/secret scrubbing is already live. The one real finding from that pass (`/_debug/memory` exposed without a debug guard) was already fixed. **This means reproducing Production is primarily an infrastructure exercise, not an application-hardening one** — the risk surface identified in this report is almost entirely in deployment/ops maturity (backups, IaC, CI/CD, monitoring wiring), not the app's own security posture.

Mirror staging's proven shape — don't redesign, reproduce:

```
MuseHub Production (672469410277), us-east-1, default VPC
  ├── EC2 (size TBD — see below), Elastic IP, SSM-only (no SSH)
  │     nginx :443 (Cloudflare Origin Cert) → 127.0.0.1:1337/1338 (blue/green)
  │     containers: musehub-{blue,green}, musehub-worker, postgres, musehub-runner
  ├── Security group: 443 from Cloudflare ranges only, nothing else
  ├── ECR: dedicated `musehub/musehub` repo in this account (recommended — see §8.5) with the same lifecycle policy applied in §6
  ├── IAM: production-specific deploy role/instance profile, distinct from `musehub-infra`
  ├── Secrets: hand-provisioned `.env` (matches current proven pattern) — or take this as the moment to actually wire up the stalled SSM Parameter Store pipeline, since Production is a clean slate
  └── CloudWatch: log group with **explicit 30-day retention set from day one** (the never-expire mistake shouldn't repeat), plus the alarms `cloudwatch-alerts.sh` already defines (5xx rate, p99 latency, disk, DB connections) — remember to pass `--region us-east-1` explicitly
```

**Instance sizing**: staging is `t3.medium`, the old/broken "prod" was `t3.small` — inconsistent with no documented reason. Recommend starting Production at `t3.medium` (match staging, the known-working size) rather than perpetuating the smaller, undersized guess.

**Do not** copy staging's database or R2 bucket contents into production — start with a fresh database and a fresh `musehub-prod` R2 bucket (already exists as a naming convention, per `docs/infrastructure.md`), consistent with the readiness checklist's explicit rule against copying unsanitized staging data into production.

---

## 10. TLS recommendation

**Continue with Cloudflare Origin Certificate — do not introduce Let's Encrypt or ACM.** This is already the proven, working pattern for staging (and was the original "prod" instance's design too), needs no load balancer, and gets a 15-year cert with zero renewal maintenance. Introducing a second TLS mechanism for Production alone would add complexity for no benefit, and ACM specifically doesn't apply here since there's no ALB/CloudFront in this architecture. If a load balancer is ever introduced later, ACM would become the natural fit at that point — not before.

---

## 11. DNS / cutover strategy

**Blocked on one decision from you: the canonical production hostname.** Once that's confirmed:

1. Confirm current DNS records in the **Cloudflare dashboard** (not Namecheap — registrar only).
2. Provision the Production EC2 + Elastic IP first, fully validate over the raw IP (`curl -k https://<EIP>/healthz` with a Host header, or a temporary `/etc/hosts` override) before touching DNS.
3. Add the production hostname's `A` record in Cloudflare pointed at the new Elastic IP, proxied (orange-cloud) — matching staging's and the old prod's existing pattern.
4. Verify propagation, confirm the Cloudflare Origin Certificate covers the exact hostname(s) chosen, confirm SSL mode is Full (Strict).
5. Full smoke test over the final HTTPS hostname before considering it launched.
6. Decide the old, broken `musehub-prod` instance's fate as a separate, explicit follow-up (retire vs. repurpose) — not part of this cutover.

---

## 12. Backup and rollback strategy

**This needs to be real before launch, not deferred again.** Concretely:

1. Wire `deploy/backup.sh` in for real on the new Production instance: install `rclone`, configure an R2 remote, install the cron job, and — critically — **verify a real, non-empty backup lands and can actually be restored**, not just that the script runs.
2. Take a pre-cutover snapshot/backup as the very last step before routing any production traffic.
3. Rehearse rollback at least once before launch: redeploy a prior `IMAGE_TAG`, confirm the blue/green flip and health gate behave as expected, and separately rehearse a database rollback path (`alembic downgrade` against a scratch copy, not production).
4. Document both procedures in `docs/infrastructure.md`'s Production section once real.

---

## 13. Monitoring and health-check strategy

- Reuse `/healthz` as the deploy-time gate (already proven).
- Run `deploy/cloudwatch-alerts.sh --region us-east-1 --instance-id <new-instance-id>` for real this time — it already defines sensible thresholds (5xx rate >1%, p99 latency >2s, disk >80%, DB connections >90%) that have simply never been applied anywhere.
- Set explicit log retention (30 days, matching the script's own default) instead of never-expire.
- Confirm `ALERT_EMAIL` (and optionally `ALERT_PHONE`) so SNS actually notifies someone.

---

## 14. Estimated recurring cost

Based on the existing cost table in `docs/infrastructure.md`, extended to a third environment:

| Item | $/month (approx.) |
|---|---|
| EC2 t3.medium (production, matching staging's proven size) | ~$30 |
| Elastic IP (while associated) | ~$0 |
| EBS gp3 20GB | ~$1.60 |
| CloudWatch logs (30-day retention vs. never-expire) | small, bounded |
| **New Production total** | **~$32/mo** |
| Existing staging + (fixed) prod, unchanged | ~$35/mo |

Decommissioning the orphaned `musehub-staging-db` RDS instance (once confirmed unused) would save a small additional amount and is pure waste today regardless of the production launch timeline.

---

## 15. Exact implementation sequence

Following the launch TODO's phase structure, next steps in order:

1. **You decide**: canonical production hostname; ECR strategy (dedicated repo vs. cross-account); IaC now vs. later.
2. **Phase 3 (provision baseline)**: create Production security group, EC2 + EIP, IAM instance profile/role, ECR repo (+ same lifecycle policy), CloudWatch log group with 30-day retention.
3. **Phase 4 (deploy and validate before DNS)**: bootstrap instance, first deploy of a known-good staging-tested image, validate over raw IP, run and verify migrations against a fresh production database, wire up real backups (§12) before going further.
4. **Phase 5 (DNS/TLS)**: once validated, add the Cloudflare DNS record, confirm HTTPS end-to-end.
5. **Phase 6 (repeatable deploys)**: add account-ID guards to `push.sh`/`deploy.sh` so they fail closed in the wrong account, add a production confirmation step, wire the new production SSO profile in place of the shared key where practical.
6. **Phase 7 (cutover)**: pre-cutover backup, route traffic, monitor, decide the old `musehub-prod` instance's fate.

## 16. Proposed next checkpoint

If you approve the direction in §9–§11, the next concrete action is provisioning the Production security group and EC2 instance (Phase 3) — I'll propose the exact `aws ec2` commands for review before running anything, the same way Phases 2–3 were handled.

## 17. Items requiring your manual action

- Confirm the canonical production hostname.
- Decide dedicated ECR repo vs. cross-account access for Production.
- Decide IaC now vs. after this launch (Terraform/OpenTofu already recommended, tool choice settled — only timing is open).
- Confirm whether the orphaned `musehub-staging-db` RDS instance is truly unused (once you're confident, I can help decommission it).
- Set up billing visibility / an AWS Budget for the Production account (needs Management-account access, which we deliberately haven't set up).
- Decide the old `musehub-prod` instance's eventual fate (fix in place vs. retire).
- Decide whether to split the single Postgres role (app/migrations/admin) before or shortly after production launch.
- Confirm `musehub-releases`' fully-public S3 bucket settings are intentional and that nothing sensitive ever lands there (carried over from `13-security-monitoring.md`, never explicitly confirmed).
- Grant broader IAM/CloudTrail/GuardDuty/budgets read access (requested in the prior sweep, still outstanding) so CloudTrail/GuardDuty/Security Hub/Access Analyzer state can actually be checked instead of guessed at.
- Several items are flatly Gabriel-only regardless of AWS access level (root MFA, Namecheap account hygiene, domain/billing ownership verification) — carried over unresolved from `01-ownership-and-access.md`.

## 18. Material unanswered questions

- Is the shared Nonproduction/Stori account split ever planned, or is that accepted indefinitely (per the readiness checklist's decision log)? The 3-account Organization split we've since built covers MuseHub's own environments, but Nonproduction itself is still shared with Stori.
- Should Production adopt the stalled SSM Parameter Store secrets pipeline from day one, or continue the proven hand-provisioned `.env` pattern? (Open since `00-inventory-and-decisions.md`, still unresolved.)
- What are the actual RPO/RTO targets? Nothing in any doc defines them, and they materially affect backup cadence and architecture decisions (multi-AZ, etc.) that haven't been sized yet.
- What's the actual traffic/growth expectation? Also undefined everywhere, and it affects instance sizing decisions beyond just "match staging's t3.medium."
- Is a Cloudflare Worker (referenced via `WORKER_INTERNAL_KEY`) already provisioned for Production, or does that need standing up alongside the EC2/DNS work?