Section 0 — Decisions and Inventory

Companion to musehub-production-readiness-checklist.md. Every fact below was verified against either docs/infrastructure.md, the scripts in deploy/, or a live AWS query on 2026-08-04 — not assumed. Items marked UNVERIFIED need a decision or a check I can't make from here.

Current staging/production architecture

This is materially simpler than the checklist assumes. No ECS/EKS, no ALB, no RDS. Two raw EC2 instances running Docker Compose, fronted by Cloudflare.

• Compute platform: raw EC2, Docker Compose (musehub uvicorn + postgres:16 + musehub-runner). No load balancer, no autoscaling group.
• Ingress: Cloudflare (proxy + TLS termination at edge, Full Strict to nginx origin on 443) → nginx on the instance → uvicorn on 1337/1338 (blue/green).
• Database: self-hosted PostgreSQL 16 in a Docker container on the same instance as the app — not RDS. No managed failover, no read replica.
• Object storage: ⚠️ contradiction found, see below.
• VPC: both instances sit in the default VPC with no custom subnetting — no private subnet for the database, no NAT gateway. The DB is only reachable from the app container network, not the public internet, but there is no network-layer isolation beyond that.
• Security group (musehub-sg, sg-05815872537fcfe76, verified live): inbound TCP 443 from Cloudflare IP ranges only, all egress open. No port 22, no port 80. Confirmed no direct DB exposure.
• Secrets and configuration sources: ⚠️ contradiction found, see below.
• Logging and monitoring: ⚠️ correction — I was wrong earlier in this doc. aws cloudwatch describe-alarms and aws sns list-topics are indeed empty — no alarms, no notification topics. But I hadn't checked log groups themselves: /musehub/staging exists live in CloudWatch Logs, actively receiving ~1.7 GB of structured JSON application logs in real time, including per-request access logs (method/path/status/duration_ms/request_id) and app warnings. Retention is unset (never expires — a cost/compliance item, not a missing-logging item). So: logging is centralized and structured; only alarms/dashboards/metric-filters on top of it are missing. cloudwatch-alerts.sh (drafted, never run) also targets the wrong log group name (/musehub/app, not the real /musehub/staging) — another reason it clearly never executed successfully. See Section 12 for full detail.
• Current deployment process: manual, human-triggered. A person runs bash deploy/push.sh staging (or prod) from a laptop. It builds the image locally, pushes to ECR via crane, then SSMs a deploy script to the instance which does blue/green swap + curl /healthz + nginx reload. There is no CI/CD automation — no GitHub Actions, no MuseHub-native pipeline triggers on push. This is the entire subject of Sections 9–11.

✅ Contradiction 1 — object storage — RESOLVED: R2 is live, docs/infrastructure.md is stale

docs/infrastructure.md states the object store is a Docker volume (musehub_data) local to the instance. This is wrong for staging today. Direct proof from this session's own muse push to staging: the push flow's mpack-presign step returned a live presigned upload URL at https://bed873d46de5273abf843468a7833f09.r2.cloudflarestorage.com/musehub-staging/mpacks/..., and the upload succeeded (HTTP 200). Staging's object storage is Cloudflare R2, bucket musehub-staging, right now — not a Docker volume. musehub/config.py also confirms blob_storage_bucket/blob_storage_endpoint are first-class, generically S3-compatible settings (R2, MinIO, or AWS S3), not a one-off script. docs/infrastructure.md needs a correction, not just this session's inventory doc.

⚠️ Contradiction 2 — secrets and backups

docs/infrastructure.md says "All secrets live in /opt/musehub/.env on each instance" (no SSM) and "No automated backup is configured yet." But deploy/secrets.sh implements SSM Parameter Store-sourced secrets (with a hard die if DB_PASSWORD isn't in SSM), and deploy/backup.sh implements a full pg_dump + R2 + cron backup strategy.

Update — partially resolved. aws ssm get-parameters-by-path --path /musehub/ --recursive returns only 3 of the 7 parameters secrets.sh requires (PACK_WORKER_URL, WEBHOOK_SECRET_KEY, WORKER_INTERNAL_KEY — no DB_PASSWORD, no BLOB_STORAGE_*, no RUNNER_TOKEN). Since secrets.sh dies immediately if DB_PASSWORD is missing from SSM, secrets.sh cannot be what produced staging's live .env. But we now know (see Contradiction 1, resolved above) that staging's .env does contain working BLOB_STORAGE_* R2 credentials — proven by a successful live push. So the actual current state is:

Staging's .env was hand-provisioned (matching docs/infrastructure.md's "generated at provision time" description of DB_PASSWORD) but its contents are richer than infrastructure.md documents — it includes working BLOB_STORAGE_* R2 keys that the doc never mentions. secrets.sh (SSM-sourced) and backup.sh (R2-backup) are drafted-but-not-wired for the secrets pipeline and backup automation specifically — that part of my original read still stands, confirmed by the empty CloudWatch/SNS query. But "no R2" was wrong; R2 is live for object storage, just not provisioned via the SSM automation path.

Still needs Gabriel to confirm: whether production's .env (once the IAM instance profile is fixed and prod deploy works again) is expected to mirror staging's hand-provisioned R2 setup, or whether prod should go through the SSM pipeline from day one.

Manually created AWS resources (inventoried, not yet in IaC)

Everything below was created by hand via the aws-provision*.sh scripts or console — none of it exists as code today (Section 4 territory):

• 2× EC2 instances + 2× Elastic IPs (listed above)
• 1× security group (musehub-sg)
• IAM user musehub-infra (deploy credentials — ECR push, SSM send, scoped SSM param read)
• IAM role/instance-profile musehub-ec2-ssm (attached to staging only)
• ECR repository musehub/musehub (100 images currently)
• S3 bucket musehub-releases (muse CLI release tarballs, public-facing via /install.sh)
• 3 SSM SecureString parameters under /musehub/staging/
• Cloudflare: DNS proxying, TLS (Full Strict), zone for musehub.ai, and — per secrets.sh/worker.py — likely R2 buckets and a Worker, unverified, I have no Cloudflare API access from here

External dependencies inventoried

No email provider, no payment provider, and no error tracker exist yet. These aren't hidden — they're just not built. Section 12 (observability) and Section 14 (application security) will need to either add one or explicitly note the gap as deferred.

Things I could not verify from here (permissions gap)

The musehub-infra IAM user — the only credential available in this environment — is explicitly denied on iam:ListUsers, iam:ListRoles, iam:ListAccountAliases, route53:ListHostedZones, cloudtrail:DescribeTrails, guardduty:ListDetectors, budgets:ViewBudget, cloudfront:ListDistributions, wafv2:ListWebACLs, and ecr:DescribeRepositories (though ecr describe-images on the specific repo works). This is actually a good sign for least-privilege (Section 3), but it means:

• I can't confirm whether CloudTrail, GuardDuty, Security Hub, or Config are enabled account-wide (Section 13) — needs an admin/root check from you.
• I can't enumerate IAM users/roles/policies account-wide (Section 3) — same.
• I can't see Route 53 zones, budgets, or CloudFront/WAF — same.
• The AWS account is not MuseHub-dedicated. aws s3api list-buckets surfaced stori-assets-992382692655-useast2 alongside musehub-releases — this AWS account (992382692655) is shared with the Stori project. This is directly relevant to Section 1 (ownership) and Section 2 (account structure) — worth being explicit about in the Decision Log rather than assuming MuseHub has a dedicated account.

Stateful / mission-critical data (identified)

1. PostgreSQL database (users, repos, commits, issues, proposals, releases, MSign keys) — self-hosted, no automated backup currently active (see Contradiction 2).
2. Object store — pushed repo content, currently either a Docker volume or R2 depending on which contradiction resolves to true. Either way, content-addressed, so integrity is easy to verify but availability depends entirely on whichever medium is real.
3. musehub-releases S3 bucket — muse CLI release artifacts, public via /install.sh. No backup needed (rebuildable from ~/ecosystem/muse source), but availability matters (it's a live install path).
4. SSM SecureString parameters (3 currently) — small blast radius today, will grow if secrets migration (Section 7) proceeds.

Open decisions — UNVERIFIED, need Gabriel

IaC recommendation (for when you're ready to decide item 2)

Given the actual footprint (2 EC2 instances, 1 SG, 1 ECR repo, 1 S3 bucket, a handful of SSM params — not a sprawling ECS/RDS estate), Terraform/OpenTofu is the pragmatic choice: it directly represents EC2/SG/SSM/S3/ECR without AWS-specific ceremony, has the biggest module ecosystem for exactly this shape, and doesn't require picking a programming language the way CDK does. This is a recommendation, not a decision — flagging it here so it's visible next to the other open items rather than buried in Section 4.

Note on Section 1 / 9 / 10 / 11 — GitHub references

The checklist's ownership and CI/CD sections (1, 9, 10, 11) are written assuming GitHub Actions, GitHub environments, GitHub branch protection, and CODEOWNERS. Per absolute ecosystem policy, git and GitHub are never used for this project — Muse and MuseHub are the only VCS/CI tools, and GitHub exists solely as a passive backup mirror. When those sections are worked, every "GitHub" concept will be translated to its MuseHub equivalent (MuseHub proposals instead of PRs, MuseHub branch protection instead of GitHub branch protection, musehub-runner-based CI instead of GitHub Actions, MuseHub collaborator roles instead of GitHub org membership). Flagging this now so it isn't a surprise three sections from now.