MuseHub Cloud Infrastructure

Last updated: 2026-08-25 — Production rebuilt from scratch in a dedicated, isolated AWS account (the old musehub-prod instance in the shared Nonproduction account was never fixed; a new instance was built instead). Corrected TLS mechanism (Cloudflare Origin Certificate everywhere, never Let's Encrypt — the prior "Let's Encrypt via Certbot" line in this doc was stale even before today). Production DNS cutover is live.

Overview

MuseHub runs on AWS EC2 (us-east-1) behind nginx with a Cloudflare Origin Certificate (never Let's Encrypt/Certbot — Cloudflare terminates TLS at its edge and connects to the origin using this certificate; nginx never listens on port 80 publicly). The application stack is Docker Compose semantics run as plain docker run containers on a shared user-defined network: musehub (uvicorn) + postgres:16 (+ musehub-runner on staging only — see below). No managed RDS, no ECS, no load balancer — intentionally minimal for this stage.

Two environments, now in two separate, isolated AWS accounts (see musehub-cloud-identity-and-aws-operating-model.md for the full account/identity model):

Legacy, not decommissioned yet: i-0855d6efe7fa1a49d (musehub-prod, 98.89.99.211) still exists inside the shared Nonproduction account. It was never fixed (no IAM instance profile was ever attached, so it was never reachable via SSM) — production was rebuilt from scratch in the new account instead of repairing it. Its retirement is a separate, deliberately deferred decision.

Shared AWS Resources

Resources below are per-account — Production and Nonproduction are fully separate AWS accounts now, each with their own security group, ECR repository, and IAM roles. Nothing is shared between them except the Cloudflare zone/DNS/TLS layer in front of both.

Security group inbound rules (both accounts, identical shape): TCP 443 only, restricted to Cloudflare's published IPv4 + IPv6 ranges. Port 22 (SSH) and port 80 (HTTP) are not open on either security group. All remote access is via AWS SSM Session Manager (Session Manager, not SSH — no key pair exists for the production instance at all, by design).

Cloudflare SSL mode is Full (Strict) for both environments — Cloudflare terminates TLS at the edge using a Cloudflare-issued cert, then connects to the origin on port 443 using a Cloudflare Origin Certificate (/etc/ssl/cloudflare/origin.pem/origin.key on each instance). Staging and Production have separate Origin Certificates — Production's was generated fresh on 2026-08-25 rather than reusing/hunting for staging's original private key, which was never confirmed saved anywhere retrievable (Cloudflare never re-displays a private key after creation; regenerating a fresh one is the normal, low-stakes path, not a workaround). Nginx never needs to listen on port 80.

Instance access for Nonproduction still requires the legacy musehub-infra AWS credentials (default profile in ~/.aws/credentials). Production access uses named AWS SSO profiles (musehub-production, musehub-nonproduction) via IAM Identity Center — see the identity/operating model doc linked above.

Production Environment

Built from scratch 2026-08-25 in the dedicated MuseHub Production AWS account. Not a repair of the old instance — a new one, in a new account, with a correctly-attached IAM instance profile from the start (SSM registered on the very first poll, unlike the legacy instance, which never worked).

Instance

Instance ID : i-043aaed71bef11903
Name        : musehub-production
Account     : MuseHub Production (672469410277)
Elastic IP  : 100.60.14.220
App dir     : /opt/musehub
IMDSv2      : enforced (HttpTokens=required) — a hardening default the legacy instances don't have

DNS (musehub.ai) — hosted at Cloudflare, not Namecheap

musehub.ai's nameservers point at Cloudflare (dig NS musehub.ai → *.ns.cloudflare.com). Namecheap is the registrar only — DNS records are managed in the Cloudflare dashboard, not Namecheap's Advanced DNS tab. The table below shows the origin records configured in Cloudflare (what Cloudflare proxies to), not what a public DNS query returns (which resolves to Cloudflare's anycast IPs for everyone).

musehub.ai (apex) is the canonical hostname, decided 2026-08-25 (matching the github.com/www.github.com precedent). Two Cloudflare Redirect Rules handle the rest, both verified live including query-string preservation:

• www.musehub.ai/* → 301 → https://musehub.ai/* ("Redirect from WWW to root" template)
• http://* → 301 → https://* ("Redirect from HTTP to HTTPS" template) — not optional: the security group doesn't accept port 80 at all, so without this edge-level redirect, plain HTTP requests would fail to connect rather than redirect.

Stack

nginx (host, port 443 only — port 80 not listened on publicly; Cloudflare edge handles the HTTP→HTTPS redirect)
  └─ proxy_pass → 127.0.0.1:1337 (blue) or 127.0.0.1:1338 (green)
       └─ musehub-blue/musehub-green container (uvicorn), network-alias "musehub"
            └─ musehub_postgres container (port 5432 internal), network-alias "postgres"
  musehub-worker container (background job processor)

Containers run as plain docker run on a shared user-defined bridge network (musehub_musehub-internal) rather than via docker compose — production has no repository checkout on the instance at all (purely image + generated config), which is cleaner than staging's setup (staging's /opt/musehub still contains a full legacy rsync-era repo copy, dead weight not worth replicating here).

musehub-runner is deliberately not running in production. Per the CI-pipeline research, this container is scaffolding for a job-execution client that was never actually built — running a privileged docker:24-dind container for zero functional benefit isn't worth it. Revisit once a real runner client exists.

Container memory limits

| musehub_postgres | 1GiB | Not in any tracked script — set live via docker update --memory 1g on 2026-08-28. Postgres has no deploy script of its own (unlike the app/worker containers), so this limit only exists in the running container's live config and will be lost if the container is ever removed and recreated. Tracked as a real gap — see production-readiness #170.

Raised from the original defaults (app/worker: 2GiB, postgres: 256MiB) on 2026-08-28 after both values proved too small for muse (1,492 commits, 11,313 blobs, ~389MB mpack) and crashed with OOM kills — confirmed via dmesg, not inferred. See production-readiness #170 for the full writeup: the underlying issue is that /push/unpack-mpack processes an entire mpack synchronously in memory rather than streaming, so any sufficiently large repo push will eventually hit whatever ceiling these limits are set to. The memory bump is a stopgap, not a fix.

Volumes

Object storage — Cloudflare R2, not a local volume

Pushed repo objects (mpacks) live in Cloudflare R2, not a Docker volume on the instance. musehub/config.py's blob_storage_bucket / blob_storage_endpoint settings point at R2 (https://<account>.r2.cloudflarestorage.com), with separate buckets per environment:

Verified live: the /push/mpack-presign endpoint returns a presigned R2 upload URL, and pushes complete via direct-to-R2 upload — the instance itself never buffers the full mpack to disk.

Secrets — AWS SSM Parameter Store (both environments now use this pipeline for the secret half)

Production secrets are not a hand-provisioned .env — they're sourced from AWS SSM Parameter Store via deploy/secrets.sh, same as the script has always supported but staging never fully adopted (staging still has only 3 of 7 parameters in SSM; production has all 6 required secrets). See the Secrets inventory section near the bottom of this doc for the full parameter list and the non-secret config values (PUBLIC_URL, CORS_ORIGINS, BLOB_STORAGE_BUCKET, etc.) that secrets.sh fills in per-environment.

To regenerate /opt/musehub/.env from SSM at any time (e.g. after a secret rotation):

# Via an interactive SSM session on the instance
sudo MUSEHUB_ENV=production bash /opt/musehub/deploy-secrets.sh

Nginx config

Site config lives at /etc/nginx/sites-available/musehub on the instance (installed from deploy/nginx-cf.conf, with DOMAIN_PLACEHOLDER substituted to musehub.ai www.musehub.ai).

Key timeouts:

• /push and /push/objects — 300 s (large repo push serialization)
• MCP / SSE streaming endpoints — 3600 s
• Everything else — 60 s

TLS

Cloudflare Origin Certificate — RSA 2048, 15-year validity, covering musehub.ai + *.musehub.ai, generated fresh 2026-08-25 specifically for this instance (not reused from staging or the legacy instance). Lives at /etc/ssl/cloudflare/origin.pem / /etc/ssl/cloudflare/origin.key on the instance. Never Let's Encrypt/Certbot — there is no Certbot installed or configured on this instance; Cloudflare's edge is what visitors' browsers actually see, and Full (Strict) mode requires only that the origin presents a cert Cloudflare trusts, which this is.

Instance access (SSM — no SSH, no key pair exists)

# Open an interactive shell on the production instance
aws ssm start-session --profile musehub-production --region us-east-1 --target i-043aaed71bef11903

# Run a one-off command
aws ssm send-command \
  --profile musehub-production \
  --instance-ids i-043aaed71bef11903 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker ps"]' \
  --region us-east-1 \
  --query "Command.CommandId" --output text

Useful commands on the instance

Run via SSM (aws ssm start-session --profile <musehub-production|musehub-nonproduction> --region us-east-1 --target <instance-id>). The active app slot is either musehub-blue (port 1337) or musehub-green (port 1338).

# Which slot is live?
cat /opt/musehub/.active-slot
cat /etc/nginx/musehub-active-port

# View running containers
sudo docker ps

# Tail live app logs (substitute blue/green as needed)
sudo docker logs -f musehub-blue
sudo docker logs -f musehub-green

# Quick health check
curl -s http://127.0.0.1:1337/healthz   # blue slot
curl -s http://127.0.0.1:1338/healthz   # green slot

# Run Alembic migrations manually (against the live DB)
SLOT=$(cat /opt/musehub/.active-slot)
DB_PASSWORD=$(grep ^DB_PASSWORD /opt/musehub/.env | cut -d= -f2)
sudo docker run --rm \
  --network musehub_musehub-internal \
  --env-file /opt/musehub/.env \
  -e "DATABASE_URL=postgresql+asyncpg://musehub:${DB_PASSWORD}@postgres:5432/musehub" \
  <ecr-image>:<tag> alembic upgrade head

# Postgres shell — actual container name is musehub_postgres (the network
# alias "postgres" used in DATABASE_URL is only resolvable *between*
# containers on the shared network, not usable as a `docker exec` target)
sudo docker exec -it musehub_postgres psql -U musehub -d musehub

# View nginx status
sudo systemctl status nginx
sudo nginx -t

Staging Environment

Purpose

Full production mirror with a separate DB, separate object store, and separate domain. Used for smoke tests before every prod deploy. Never exposed to users.

Instance (provisioned by aws-provision-staging.sh)

Instance ID : i-07547cd20bee2dea5
Name        : musehub-staging
Elastic IP  : 23.22.27.39
App dir     : /opt/musehub
Domain      : staging.musehub.ai

DNS (musehub.ai) — Cloudflare-hosted, see the production section above for detail

Provisioning (one-time, run locally)

# 1. Provision EC2 + EIP
chmod +x deploy/aws-provision-staging.sh
./deploy/aws-provision-staging.sh
# Note the instance ID and Elastic IP printed at the end.

# 2. Add staging.musehub.ai A record on Namecheap (see above).
#    Wait for propagation (~5 min with Automatic TTL):
watch -n 10 "dig staging.musehub.ai +short"

# 3. Bootstrap the instance (installs AWS CLI, verifies ECR access)
bash deploy/bootstrap-instance.sh staging

# 4. Run setup script on the instance via SSM
aws ssm send-command \
  --instance-ids <instance-id> \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["chmod +x /opt/musehub/deploy/setup-ec2-staging.sh && /opt/musehub/deploy/setup-ec2-staging.sh"]' \
  --region us-east-1

# 5. Do the first deploy
bash deploy/push.sh staging

Recovering a down staging instance (522 / Bad Gateway)

Symptom: staging.musehub.ai returns Cloudflare 522 or Bad Gateway.

Root cause pattern: The container stopped (either manually or after a reboot) and --restart unless-stopped did not fire because the container was in a stopped (not crashed) state when the instance last rebooted.

Fix — one SSM command, no polling:

CMD_ID=$(aws ssm send-command \
  --region us-east-1 \
  --instance-ids i-07547cd20bee2dea5 \
  --document-name "AWS-RunShellScript" \
  --parameters '{"commands":["sudo docker start musehub-blue musehub-worker 2>&1 && sudo musehub-set-slot blue && echo done"]}' \
  --query "Command.CommandId" --output text)
echo "Command sent: $CMD_ID"
# Wait ~20s then check once:
sleep 20 && aws ssm get-command-invocation \
  --region us-east-1 \
  --command-id "$CMD_ID" \
  --instance-id i-07547cd20bee2dea5 \
  --query "[Status,StandardOutputContent]" --output text

Check staging.musehub.ai in the browser — it should be back.

Critical rules when using SSM to recover staging:

1. Never reboot to fix SSM Pending. A reboot stops containers that were manually started — --restart unless-stopped only auto-starts containers that were running (not stopped) at reboot time. Rebooting to fix SSM will take the site down and require a manual docker start anyway.

2. Never poll SSM in a loop. The shell until/while sleep pattern freezes the terminal and masks whether the command succeeded. Send the command, wait a fixed interval, fetch once.

3. SSM Pending ≠ SSM broken. The agent can show Online but queue commands as Pending for 10–30 seconds after a fresh start. Wait before concluding SSM is broken.

4. InProgress means it will complete. If a command shows InProgress it is executing on the instance — do not cancel or resend. Check back in 30s.

Ongoing code deploys to staging

# Standard — builds image locally, pushes to ECR, triggers blue-green on staging
bash deploy/push.sh staging

Publishing a new muse CLI release

The install.sh script (served at https://staging.musehub.ai/install.sh) downloads muse-{version}.tar.gz from /releases/. The version comes from musehub/protocol/version.py (MUSE_VERSION), which tracks the musehub package version.

To ship a new muse build:

# From ~/ecosystem/musehub — builds sdist, uploads to S3, SSMs to staging,
# cleans up old tarballs (keeps 3), and verifies the URL is live.
bash deploy/publish_muse_release.sh

What it does:

1. Builds muse-{version}.tar.gz from ~/ecosystem/muse
2. Uploads to s3://musehub-releases/muse-{version}.tar.gz
3. SSMs to staging to copy from S3 → /data/releases/ (Docker volume)
4. Deletes stale tarballs from S3 and the server (keeps the 3 newest)
5. Smoke-tests https://staging.musehub.ai/releases/muse-{version}.tar.gz

Note: SSH is blocked on the instance (port 443 only). All server commands go through AWS SSM (musehub-infra IAM user). The staging instance (i-07547cd20bee2dea5) has the required IAM instance profile; no other instance does.

To test the install script end-to-end locally:

curl -fsSL https://staging.musehub.ai/install.sh | sh
# verify
~/.local/bin/muse --version
# cleanup
rm -rf ~/.local/share/muse/venv && rm -f ~/.local/bin/muse

Instance access (SSM — no SSH)

# Interactive shell on staging
aws ssm start-session --target i-07547cd20bee2dea5 --region us-east-1

Deployment Workflow

Deploys are image-based via ECR. No SSH, no rsync, no code on the instance after provisioning. All deploy commands run from the local ~/ecosystem/musehub directory.

Deploy pipeline overview

Local machine (push.sh):
  1. docker build (linux/amd64)
  2. docker save → tar, crane push → ECR (musehub/musehub:<tag>)
  3. aws ssm send-command → sync deploy.sh, then run it

Instance (deploy.sh via SSM):
  4. deploy.sh written from local copy (always current — never stale)
  5. aws ecr get-login-password | docker login
  6. docker pull <ecr>:<tag>
  7. docker run (migrations only, then exit)
  8. docker run -d (new slot — blue or green)
  9. curl /healthz until healthy
  10. nginx -s reload (zero-downtime flip)
  11. docker rm (old slot)

Key invariant: push.sh always writes the current local deploy.sh to the instance via SSM before running it. This means the instance's deploy.sh is always in sync with the local repo — there is no separate "sync the deploy scripts" step.

ECR Push — Use crane (not docker push)

docker push to ECR routes through Docker Desktop's VPNKit proxy (http.docker.internal:3128 / 192.168.65.1:3128 from inside the VM). After a local IP change or a Docker Desktop restart, the VPNKit proxy drops connections mid-upload on large layer pushes, producing broken pipe errors. The fix is crane — Google's container registry tool — which pushes images directly from the macOS host network, bypassing the Docker Desktop VM layer and its proxy entirely.

crane is the standard push method. Never use docker push to ECR.

Install once:

brew install crane

push.sh calls crane internally. If pushing manually outside the script:

# 1. Build the image locally (linux/amd64 target)
docker build --platform linux/amd64 -t musehub/musehub:latest .

# 2. Save to a tar archive on the host
docker save musehub/musehub:latest -o /tmp/musehub-latest.tar

# 3. Authenticate crane against ECR
aws ecr get-login-password --region us-east-1 \
  | crane auth login 992382692655.dkr.ecr.us-east-1.amazonaws.com \
      --username AWS --password-stdin

# 4. Push with crane (runs entirely on the macOS host — no VPNKit involved)
crane push /tmp/musehub-latest.tar \
  992382692655.dkr.ecr.us-east-1.amazonaws.com/musehub/musehub:latest

Standard deploy

# Deploy to staging
bash deploy/push.sh staging

⚠️ bash deploy/push.sh prod does NOT deploy to the new Production instance/account. deploy/push.sh's hardcoded PROD_INSTANCE still points at the legacy i-0855d6efe7fa1a49d in the shared Nonproduction account — it was never updated for the new isolated Production account (672469410277, instance i-043aaed71bef11903, dedicated ECR repo). Adding an explicit production target with account-ID guards to push.sh/deploy.sh is open follow-up work (readiness ticket #159). Until that exists, deploying a new build to Production means repeating the manual SSM-driven sequence used for the initial launch: crane copy (or rebuild) the image into the Production ECR repo, then docker pull + blue/green swap via SSM commands against i-043aaed71bef11903 directly.

Rollback

# List recent ECR image tags (Nonproduction registry — staging only)
aws ecr describe-images \
  --profile musehub-nonproduction \
  --repository-name musehub/musehub \
  --region us-east-1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-10:].imageTags[0]' \
  --output table

# Redeploy a specific tag to staging (skips build+push)
IMAGE_TAG=<previous-tag> bash deploy/push.sh staging

Production rollback (manual, until push.sh supports a real production target): re-run the same manual blue/green docker run sequence used for the initial launch, substituting the previous image tag in the Production ECR repo (672469410277.dkr.ecr.us-east-1.amazonaws.com/musehub/musehub). Not yet rehearsed — this is an open item, not a tested procedure.

Emergency migration rollback (on instance via SSM)

# Nonproduction / staging
aws ssm send-command \
  --profile musehub-nonproduction \
  --instance-ids i-07547cd20bee2dea5 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/musehub && sudo docker run --rm --network musehub_musehub-internal --env-file .env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1

# Production
aws ssm send-command \
  --profile musehub-production \
  --instance-ids i-043aaed71bef11903 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker run --rm --network musehub_musehub-internal --env-file /opt/musehub/.env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1

Backups

Production has real, verified automated backups as of 2026-08-25 (readiness ticket #157, closed — this had been the single highest-priority open item across the entire production-readiness effort). Staging still has none — the one manual dump that ever existed there was 20 bytes (empty/failed), and that gap hasn't been replicated to production.

Production backup pipeline (deploy/backup.sh, run daily at 3 AM via cron):

1. pg_dump against musehub_postgres (not via docker compose — production has no docker-compose.yml; the script was fixed to docker exec directly), compressed, written to /opt/backups/musehub/ (14-day local retention).
2. Synced to a dedicated R2 bucket, musehub-production-backups — deliberately separate from the app's own musehub-prod object-storage bucket, via a separate, narrowly-scoped R2 API token (musehub-production-backups-r2, Object Read & Write on that bucket only) — a compromise of the app's live data credential doesn't grant access to backups, or vice versa. 90-day retention in R2.
3. A real restore was tested (not just "the backup runs"): restored into a throwaway postgres:16-alpine container, confirmed all 74 tables came back correctly, tore the container down. This is what actually proves the pipeline works, not the existence of a file.

Known rclone-on-R2 gotcha, worth knowing if this is ever reconfigured: an R2 API token scoped to "Object Read & Write" on one specific bucket doesn't include bucket-level permissions (HeadBucket/CreateBucket). Older rclone versions (this instance has 1.53.3, from Ubuntu's apt repo) perform a bucket-existence check before uploading by default, which fails with a confusing 403 AccessDenied even though the credentials are completely correct — the fix is no_check_bucket = true in the remote config, not broader token permissions.

Volume snapshot via AWS before every production deploy is still not automated — remains a manual step for now. Object store (R2, content-addressed) is inherently safe to snapshot at any time; that was never the actual gap — the Postgres database was.

Staging backups are the reasonable next follow-up — everything built for production (bucket/token pattern, cron, restore-testing method) is directly reusable, just not yet repeated for staging.

Costs (approximate, us-east-1, 2026 pricing)

Decommissioning the legacy musehub-prod instance (a separate, deliberately deferred decision) would bring this down to roughly the same ~$60/mo as running production + staging alone.

To pause staging when not needed:

aws ec2 stop-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
# Start again with:
aws ec2 start-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>

The Elastic IP stays associated while the instance is stopped — no charge.

Secrets inventory

Production (as of 2026-08-25) sources all 6 secrets from AWS SSM Parameter Store (/musehub/production/*, KMS-encrypted SecureString, default alias/aws/ssm key) via deploy/secrets.sh — the pipeline the script was always built for, now actually the live path. .env on the instance is generated from SSM, not hand-edited; re-running secrets.sh refreshes it from the current SSM values. The instance role has read-only access (ssm:GetParameter/GetParametersByPath + kms:Decrypt, scoped to exactly /musehub/production/*) — no write access, by design; rotation is an operator action from an admin session, not something the instance does to itself.

Staging still only has 3 of 7 parameters in SSM (PACK_WORKER_URL, WEBHOOK_SECRET_KEY, WORKER_INTERNAL_KEY) — its .env remains hand-provisioned. Completing staging's migration to match production is a reasonable follow-up, not yet done.

Never committed to source, either way.

Ed25519 identity keys live in ~/.muse/identity.toml on each client machine. No server-side secret is involved in MSign auth — the public key in the DB is the credential.