# Section 0 — Decisions and Inventory

> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).
> Every fact below was verified against either `docs/infrastructure.md`, the scripts in
> `deploy/`, or a live AWS query on 2026-08-04 — not assumed. Items marked **UNVERIFIED**
> need a decision or a check I can't make from here.

---

## Current staging/production architecture

**This is materially simpler than the checklist assumes.** No ECS/EKS, no ALB, no RDS.
Two raw EC2 instances running Docker Compose, fronted by Cloudflare.

| | Production | Staging |
|---|---|---|
| Instance ID | `i-0855d6efe7fa1a49d` (`musehub-prod`) | `i-07547cd20bee2dea5` (`musehub-staging`) |
| Type | `t3.small` | `t3.medium` |
| State | running | running |
| Elastic IP | `98.89.99.211` | `23.22.27.39` |
| VPC | `vpc-0552bdd377de8a9a8` (default VPC, `172.31.0.0/16`) | same (default VPC) |
| IAM instance profile | **none — verified via `describe-instances`** | `musehub-ec2-ssm` |
| SSM reachable | ❌ no (no instance profile → `push.sh prod` fails with `InvalidInstanceId`) | ✅ yes |
| Domain | `musehub.ai`, `www.musehub.ai` | `staging.musehub.ai` |

- **Compute platform:** raw EC2, Docker Compose (musehub uvicorn + postgres:16 + musehub-runner). No load balancer, no autoscaling group.
- **Ingress:** Cloudflare (proxy + TLS termination at edge, Full Strict to nginx origin on 443) → nginx on the instance → uvicorn on 1337/1338 (blue/green).
- **Database:** self-hosted PostgreSQL 16 in a Docker container on the same instance as the app — not RDS. No managed failover, no read replica.
- **Object storage:** ⚠️ **contradiction found, see below.**
- **VPC:** both instances sit in the **default VPC** with no custom subnetting — no private subnet for the database, no NAT gateway. The DB is only reachable from the app container network, not the public internet, but there is no network-layer isolation beyond that.
- **Security group** (`musehub-sg`, `sg-05815872537fcfe76`, verified live): inbound TCP 443 from Cloudflare IP ranges only, all egress open. No port 22, no port 80. Confirmed no direct DB exposure.
- **Secrets and configuration sources:** ⚠️ **contradiction found, see below.**
- **Logging and monitoring:** ⚠️ **correction — I was wrong earlier in this doc.** `aws cloudwatch describe-alarms` and `aws sns list-topics` are indeed empty — no alarms, no notification topics. But I hadn't checked log groups themselves: **`/musehub/staging` exists live in CloudWatch Logs, actively receiving ~1.7 GB of structured JSON application logs in real time**, including per-request access logs (method/path/status/duration_ms/request_id) and app warnings. Retention is unset (**never expires** — a cost/compliance item, not a missing-logging item). So: logging *is* centralized and structured; only alarms/dashboards/metric-filters on top of it are missing. `cloudwatch-alerts.sh` (drafted, never run) also targets the wrong log group name (`/musehub/app`, not the real `/musehub/staging`) — another reason it clearly never executed successfully. See Section 12 for full detail.
- **Current deployment process:** manual, human-triggered. A person runs `bash deploy/push.sh staging` (or `prod`) from a laptop. It builds the image locally, pushes to ECR via `crane`, then SSMs a deploy script to the instance which does blue/green swap + `curl /healthz` + nginx reload. There is **no CI/CD automation** — no GitHub Actions, no MuseHub-native pipeline triggers on push. This is the entire subject of Sections 9–11.

### ✅ Contradiction 1 — object storage — RESOLVED: R2 is live, `docs/infrastructure.md` is stale

`docs/infrastructure.md` states the object store is a Docker volume (`musehub_data`) local to the
instance. This is **wrong for staging today**. Direct proof from this session's own `muse push`
to staging: the push flow's `mpack-presign` step returned a live presigned upload URL at
`https://bed873d46de5273abf843468a7833f09.r2.cloudflarestorage.com/musehub-staging/mpacks/...`,
and the upload succeeded (`HTTP 200`). Staging's object storage is **Cloudflare R2**, bucket
`musehub-staging`, right now — not a Docker volume. `musehub/config.py` also confirms
`blob_storage_bucket`/`blob_storage_endpoint` are first-class, generically S3-compatible settings
(R2, MinIO, or AWS S3), not a one-off script. **`docs/infrastructure.md` needs a correction**, not
just this session's inventory doc.

### ⚠️ Contradiction 2 — secrets and backups

`docs/infrastructure.md` says "All secrets live in `/opt/musehub/.env` on each instance" (no SSM) and "No automated backup is configured yet." But `deploy/secrets.sh` implements SSM Parameter Store-sourced secrets (with a hard `die` if `DB_PASSWORD` isn't in SSM), and `deploy/backup.sh` implements a full pg_dump + R2 + cron backup strategy.

**Update — partially resolved.** `aws ssm get-parameters-by-path --path /musehub/ --recursive` returns only **3** of the 7 parameters `secrets.sh` requires (`PACK_WORKER_URL`, `WEBHOOK_SECRET_KEY`, `WORKER_INTERNAL_KEY` — no `DB_PASSWORD`, no `BLOB_STORAGE_*`, no `RUNNER_TOKEN`). Since `secrets.sh` dies immediately if `DB_PASSWORD` is missing from SSM, **`secrets.sh` cannot be what produced staging's live `.env`.** But we now know (see Contradiction 1, resolved above) that staging's `.env` *does* contain working `BLOB_STORAGE_*` R2 credentials — proven by a successful live push. So the actual current state is:

> Staging's `.env` was **hand-provisioned** (matching `docs/infrastructure.md`'s "generated at provision time" description of `DB_PASSWORD`) but its **contents are richer than `infrastructure.md` documents** — it includes working `BLOB_STORAGE_*` R2 keys that the doc never mentions. `secrets.sh` (SSM-sourced) and `backup.sh` (R2-backup) are drafted-but-not-wired for the *secrets pipeline and backup automation* specifically — that part of my original read still stands, confirmed by the empty CloudWatch/SNS query. But "no R2" was wrong; R2 is live for object storage, just not provisioned via the SSM automation path.

**Still needs Gabriel to confirm:** whether production's `.env` (once the IAM instance profile is fixed and prod deploy works again) is expected to mirror staging's hand-provisioned R2 setup, or whether prod should go through the SSM pipeline from day one.

---

## Manually created AWS resources (inventoried, not yet in IaC)

Everything below was created by hand via the `aws-provision*.sh` scripts or console — none of it exists as code today (Section 4 territory):

- 2× EC2 instances + 2× Elastic IPs (listed above)
- 1× security group (`musehub-sg`)
- IAM user `musehub-infra` (deploy credentials — ECR push, SSM send, scoped SSM param read)
- IAM role/instance-profile `musehub-ec2-ssm` (attached to staging only)
- ECR repository `musehub/musehub` (100 images currently)
- S3 bucket `musehub-releases` (muse CLI release tarballs, public-facing via `/install.sh`)
- 3 SSM SecureString parameters under `/musehub/staging/`
- Cloudflare: DNS proxying, TLS (Full Strict), zone for `musehub.ai`, and — per `secrets.sh`/`worker.py` — likely R2 buckets and a Worker, **unverified**, I have no Cloudflare API access from here

## External dependencies inventoried

| Dependency | Role | Evidence |
|---|---|---|
| Cloudflare | DNS, TLS termination, DDoS/edge proxy, possibly R2 object storage + a Worker | `musehub-sg` rules, `docs/infrastructure.md`, `secrets.sh`, `musehub/worker.py` |
| Namecheap | Domain registrar + DNS host for `musehub.ai` | `docs/infrastructure.md` DNS tables |
| AWS (account `992382692655`) | Compute, ECR, S3, SSM | live queries above |
| MSign (Ed25519) | Auth — **no OAuth provider exists**, confirmed by grep | `docs/infrastructure.md`, no OAuth client code found outside MCP protocol types |
| — | No email provider found (no SendGrid/SES/Postmark/SMTP integration in `musehub/`) | grep, empty |
| — | No payment/billing provider found | grep, empty |
| — | No error-tracking (Sentry or similar) found | grep, empty |
| GitHub | **Backup mirror only** (`cgcardona/muse-backup`, `musehub-backup`) — per longstanding project convention, GitHub is never the VCS, CI, or collaboration platform for this ecosystem | prior project memory |

**No email provider, no payment provider, and no error tracker exist yet.** These aren't hidden — they're just not built. Section 12 (observability) and Section 14 (application security) will need to either add one or explicitly note the gap as deferred.

## Things I could not verify from here (permissions gap)

The `musehub-infra` IAM user — the only credential available in this environment — is **explicitly denied** on `iam:ListUsers`, `iam:ListRoles`, `iam:ListAccountAliases`, `route53:ListHostedZones`, `cloudtrail:DescribeTrails`, `guardduty:ListDetectors`, `budgets:ViewBudget`, `cloudfront:ListDistributions`, `wafv2:ListWebACLs`, and `ecr:DescribeRepositories` (though `ecr describe-images` on the specific repo works). This is actually a **good sign for least-privilege** (Section 3), but it means:

- I can't confirm whether CloudTrail, GuardDuty, Security Hub, or Config are enabled account-wide (Section 13) — needs an admin/root check from you.
- I can't enumerate IAM users/roles/policies account-wide (Section 3) — same.
- I can't see Route 53 zones, budgets, or CloudFront/WAF — same.
- **The AWS account is not MuseHub-dedicated.** `aws s3api list-buckets` surfaced `stori-assets-992382692655-useast2` alongside `musehub-releases` — this AWS account (`992382692655`) is shared with the Stori project. This is directly relevant to Section 1 (ownership) and Section 2 (account structure) — worth being explicit about in the Decision Log rather than assuming MuseHub has a dedicated account.

---

## Stateful / mission-critical data (identified)

1. PostgreSQL database (users, repos, commits, issues, proposals, releases, MSign keys) — self-hosted, **no automated backup currently active** (see Contradiction 2).
2. Object store — pushed repo content, currently either a Docker volume or R2 depending on which contradiction resolves to true. Either way, content-addressed, so integrity is easy to verify but availability depends entirely on whichever medium is real.
3. `musehub-releases` S3 bucket — muse CLI release artifacts, public via `/install.sh`. No backup needed (rebuildable from `~/ecosystem/muse` source), but availability matters (it's a live install path).
4. SSM SecureString parameters (3 currently) — small blast radius today, will grow if secrets migration (Section 7) proceeds.

## Open decisions — UNVERIFIED, need Gabriel

| # | Item | Status |
|---|---|---|
| 1 | Architecture target: harden current EC2/Compose vs. redesign to ECS+ALB+RDS | **Open** — flagged per your answer, revisit after this inventory |
| 2 | IaC tool (Terraform/OpenTofu, CDK, CloudFormation) | **Open** — deferred until now; see recommendation below |
| 3 | AWS account structure | **Decided** — single account accepted as technical debt for now |
| 4 | Object storage ground truth: local Docker volume vs. Cloudflare R2 | **Resolved:** R2 is live on staging, proven by a successful push this session; `docs/infrastructure.md` needs correcting |
| 5 | Secrets/backup ground truth: static `.env` vs. SSM+R2 pipeline | **Partially resolved:** `.env` is hand-provisioned (not SSM-automated) but contains working R2 keys `infrastructure.md` doesn't document; SSM/backup *automation* scripts are still drafted-not-wired. Still needs Gabriel on whether prod should adopt the SSM pipeline from day one |
| 6 | RPO / RTO targets | **Not yet defined** — no numbers exist anywhere in the repo |
| 7 | Expected initial traffic / growth assumptions | **Not yet defined** |
| 8 | Availability expectations, acceptable downtime during deploy | **Not yet defined** |
| 9 | Canonical hostname (root vs `www`) | **Not yet decided** — both currently resolve to the same IP; no redirect logic found in nginx config reference |
| 10 | DNS host: stay on Namecheap vs move to Route 53 | **Not yet decided** |

### IaC recommendation (for when you're ready to decide item 2)

Given the actual footprint (2 EC2 instances, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params — not a sprawling ECS/RDS estate), **Terraform/OpenTofu** is the pragmatic choice: it directly represents EC2/SG/SSM/S3/ECR without AWS-specific ceremony, has the biggest module ecosystem for exactly this shape, and doesn't require picking a programming language the way CDK does. This is a recommendation, not a decision — flagging it here so it's visible next to the other open items rather than buried in Section 4.

---

## Note on Section 1 / 9 / 10 / 11 — GitHub references

The checklist's ownership and CI/CD sections (1, 9, 10, 11) are written assuming GitHub Actions, GitHub environments, GitHub branch protection, and `CODEOWNERS`. Per absolute ecosystem policy, **git and GitHub are never used** for this project — Muse and MuseHub are the only VCS/CI tools, and GitHub exists solely as a passive backup mirror. When those sections are worked, every "GitHub" concept will be translated to its MuseHub equivalent (MuseHub proposals instead of PRs, MuseHub branch protection instead of GitHub branch protection, `musehub-runner`-based CI instead of GitHub Actions, MuseHub collaborator roles instead of GitHub org membership). Flagging this now so it isn't a surprise three sections from now.