# MuseHub Cloud Infrastructure

> Last updated: 2026-08-25 — Production rebuilt from scratch in a dedicated, isolated AWS account (the old `musehub-prod` instance in the shared Nonproduction account was never fixed; a new instance was built instead). Corrected TLS mechanism (Cloudflare Origin Certificate everywhere, never Let's Encrypt — the prior "Let's Encrypt via Certbot" line in this doc was stale even before today). Production DNS cutover is live.

---

## Overview

MuseHub runs on AWS EC2 (us-east-1) behind nginx with a **Cloudflare Origin Certificate** (never
Let's Encrypt/Certbot — Cloudflare terminates TLS at its edge and connects to the origin using this
certificate; nginx never listens on port 80 publicly).
The application stack is Docker Compose semantics run as plain `docker run` containers on a shared
user-defined network: musehub (uvicorn) + postgres:16 (+ musehub-runner on staging only — see below).
No managed RDS, no ECS, no load balancer — intentionally minimal for this stage.

Two environments, now in **two separate, isolated AWS accounts** (see
[`musehub-cloud-identity-and-aws-operating-model.md`](architecture/musehub-cloud-identity-and-aws-operating-model.md)
for the full account/identity model):

| Environment | Domain | AWS account | Instance | Elastic IP | Deploy status |
|-------------|--------|--------------|----------|------------|---------------|
| Production  | `musehub.ai` (canonical), `www.musehub.ai` (redirects to apex) | MuseHub Production (`672469410277`) | `i-043aaed71bef11903` (`musehub-production`) | `100.60.14.220` | ✅ Active — built and cut over live 2026-08-25 |
| Staging     | `staging.musehub.ai` | MuseHub Nonproduction (`992382692655`) | `i-07547cd20bee2dea5` (`musehub-staging`) | `23.22.27.39` | ✅ Active — blue/green deploys working via `push.sh staging` |

**Legacy, not decommissioned yet:** `i-0855d6efe7fa1a49d` (`musehub-prod`, `98.89.99.211`) still
exists inside the shared Nonproduction account. It was never fixed (no IAM instance profile was
ever attached, so it was never reachable via SSM) — production was rebuilt from scratch in the new
account instead of repairing it. Its retirement is a separate, deliberately deferred decision.

---

## Shared AWS Resources

Resources below are **per-account** — Production and Nonproduction are fully separate AWS accounts
now, each with their own security group, ECR repository, and IAM roles. Nothing is shared between
them except the Cloudflare zone/DNS/TLS layer in front of both.

| Resource | Nonproduction (`992382692655`) | Production (`672469410277`) |
|----------|-------------------------------|------------------------------|
| Region | `us-east-1` | `us-east-1` |
| AMI | `ami-0c7217cdde317cfec` (Ubuntu 22.04 LTS, 2024) | `ami-06e78a71af43ef21a` (Ubuntu 22.04 LTS, resolved live via the AWS public SSM parameter at provisioning time — recommended over reusing a hardcoded AMI ID) |
| Instance type | `t3.small` (`musehub-prod`, legacy) / `t3.medium` (`musehub-staging`) | `t3.medium` (matches staging's proven size, not the legacy instance's undersized `t3.small`) |
| Security group | `sg-05815872537fcfe76` (`musehub-sg`) | `sg-02afc8844c9bf135a` (`musehub-production-sg`) |
| ECR registry | `992382692655.dkr.ecr.us-east-1.amazonaws.com` | `672469410277.dkr.ecr.us-east-1.amazonaws.com` — **dedicated repo**, not shared/cross-account with Nonproduction (Nonproduction is shared with the unrelated Stori project; production access must stay independent of it) |
| ECR repository | `musehub/musehub` | `musehub/musehub` |
| ECR lifecycle policy | Applied 2026-08-24: expire untagged >14 days, keep last 60 tagged | Same policy, applied at repo creation |
| IAM deploy user | `musehub-infra` (ECR push + SSM send) — legacy shared laptop credential, still in use for Nonproduction | N/A — deploys use the operator's own SSO session + the instance role, not a shared IAM user |
| IAM instance role | `musehub-ec2-ssm` (ECR pull + SSM receive) | `musehub-production-ec2-ssm` — SSM + scoped ECR pull + scoped CloudWatch Logs write + scoped SSM Parameter Store read (see Secrets inventory below) |
| CloudTrail | `musehub-nonproduction-trail`, multi-region, log-file validation, S3 bucket `musehub-nonproduction-cloudtrail-992382692655` (365-day lifecycle) — enabled 2026-08-25, did not exist before | `musehub-production-trail`, same shape, bucket `musehub-production-cloudtrail-672469410277` — enabled 2026-08-25 |
| GuardDuty | Enabled 2026-08-25 (6h finding frequency) — did not exist before | Enabled 2026-08-25 (6h finding frequency) |

Security group inbound rules (both accounts, identical shape): **TCP 443 only**, restricted to
Cloudflare's published IPv4 + IPv6 ranges. Port 22 (SSH) and port 80 (HTTP) are not open on either
security group. All remote access is via AWS SSM Session Manager (Session Manager, not SSH — no
key pair exists for the production instance at all, by design).

Cloudflare SSL mode is **Full (Strict)** for both environments — Cloudflare terminates TLS at the
edge using a Cloudflare-issued cert, then connects to the origin on port 443 using a **Cloudflare
Origin Certificate** (`/etc/ssl/cloudflare/origin.pem`/`origin.key` on each instance). Staging and
Production have **separate** Origin Certificates — Production's was generated fresh on 2026-08-25
rather than reusing/hunting for staging's original private key, which was never confirmed saved
anywhere retrievable (Cloudflare never re-displays a private key after creation; regenerating a
fresh one is the normal, low-stakes path, not a workaround). Nginx never needs to listen on port 80.

Instance access for Nonproduction still requires the legacy `musehub-infra` AWS credentials
(default profile in `~/.aws/credentials`). Production access uses named AWS SSO profiles
(`musehub-production`, `musehub-nonproduction`) via IAM Identity Center — see the identity/operating
model doc linked above.

---

## Production Environment

Built from scratch 2026-08-25 in the dedicated MuseHub Production AWS account. Not a repair of the
old instance — a new one, in a new account, with a correctly-attached IAM instance profile from the
start (SSM registered on the very first poll, unlike the legacy instance, which never worked).

### Instance

```
Instance ID : i-043aaed71bef11903
Name        : musehub-production
Account     : MuseHub Production (672469410277)
Elastic IP  : 100.60.14.220
App dir     : /opt/musehub
IMDSv2      : enforced (HttpTokens=required) — a hardening default the legacy instances don't have
```

### DNS (musehub.ai) — hosted at Cloudflare, not Namecheap

`musehub.ai`'s nameservers point at Cloudflare (`dig NS musehub.ai` → `*.ns.cloudflare.com`).
Namecheap is the registrar only — DNS records are managed in the Cloudflare dashboard, not
Namecheap's Advanced DNS tab. The table below shows the **origin** records configured in
Cloudflare (what Cloudflare proxies to), not what a public DNS query returns (which resolves to
Cloudflare's anycast IPs for everyone).

| Type | Host | Origin value | Proxy |
|------|------|-------|-----|
| A Record | @ (`musehub.ai`) | `100.60.14.220` | Proxied (orange-cloud) |
| A Record | `www` | `100.60.14.220` | Proxied (orange-cloud) |
| A Record | `staging` | `23.22.27.39` | Proxied (orange-cloud) |

`musehub.ai` (apex) is the **canonical hostname**, decided 2026-08-25 (matching the
`github.com`/`www.github.com` precedent). Two Cloudflare Redirect Rules handle the rest, both
verified live including query-string preservation:
- `www.musehub.ai/*` → 301 → `https://musehub.ai/*` ("Redirect from WWW to root" template)
- `http://*` → 301 → `https://*` ("Redirect from HTTP to HTTPS" template) — **not optional**: the
  security group doesn't accept port 80 at all, so without this edge-level redirect, plain HTTP
  requests would fail to connect rather than redirect.

### Stack

```
nginx (host, port 443 only — port 80 not listened on publicly; Cloudflare edge handles the HTTP→HTTPS redirect)
  └─ proxy_pass → 127.0.0.1:1337 (blue) or 127.0.0.1:1338 (green)
       └─ musehub-blue/musehub-green container (uvicorn), network-alias "musehub"
            └─ musehub_postgres container (port 5432 internal), network-alias "postgres"
  musehub-worker container (background job processor)
```

Containers run as plain `docker run` on a shared user-defined bridge network
(`musehub_musehub-internal`) rather than via `docker compose` — production has no repository
checkout on the instance at all (purely image + generated config), which is cleaner than staging's
setup (staging's `/opt/musehub` still contains a full legacy `rsync`-era repo copy, dead weight not
worth replicating here).

**`musehub-runner` is deliberately not running in production.** Per the CI-pipeline research, this
container is scaffolding for a job-execution client that was never actually built — running a
privileged `docker:24-dind` container for zero functional benefit isn't worth it. Revisit once a
real runner client exists.

### Volumes

| Volume | Contents |
|--------|----------|
| `musehub_postgres_data` | PostgreSQL data directory |
| `musehub_data` | App container `/data` mount (not the object store — that's R2; see below) |

### Object storage — Cloudflare R2, not a local volume

Pushed repo objects (mpacks) live in **Cloudflare R2**, not a Docker volume on the instance.
`musehub/config.py`'s `blob_storage_bucket` / `blob_storage_endpoint` settings point at R2
(`https://<account>.r2.cloudflarestorage.com`), with separate buckets per environment:

| Environment | R2 bucket | R2 API token |
|-------------|-----------|--------------|
| Staging | `musehub-staging` | pre-existing |
| Production | `musehub-prod` | `musehub-production-r2` — created 2026-08-25, scoped to **Object Read & Write on `musehub-prod` only** (least-privilege; the bucket itself was also created fresh 2026-08-25, never shared with staging's bucket) |

Verified live: the `/push/mpack-presign` endpoint returns a presigned R2 upload URL, and pushes
complete via direct-to-R2 upload — the instance itself never buffers the full mpack to disk.

### Secrets — AWS SSM Parameter Store (both environments now use this pipeline for the secret half)

Production secrets are **not** a hand-provisioned `.env` — they're sourced from AWS SSM Parameter
Store via `deploy/secrets.sh`, same as the script has always supported but staging never fully
adopted (staging still has only 3 of 7 parameters in SSM; production has all 6 required secrets).
See the Secrets inventory section near the bottom of this doc for the full parameter list and the
non-secret config values (`PUBLIC_URL`, `CORS_ORIGINS`, `BLOB_STORAGE_BUCKET`, etc.) that
`secrets.sh` fills in per-environment.

To regenerate `/opt/musehub/.env` from SSM at any time (e.g. after a secret rotation):

```bash
# Via an interactive SSM session on the instance
sudo MUSEHUB_ENV=production bash /opt/musehub/deploy-secrets.sh
```

### Nginx config

Site config lives at `/etc/nginx/sites-available/musehub` on the instance (installed from
`deploy/nginx-cf.conf`, with `DOMAIN_PLACEHOLDER` substituted to `musehub.ai www.musehub.ai`).

Key timeouts:
- `/push` and `/push/objects` — 300 s (large repo push serialization)
- MCP / SSE streaming endpoints — 3600 s
- Everything else — 60 s

### TLS

**Cloudflare Origin Certificate** — RSA 2048, 15-year validity, covering `musehub.ai` +
`*.musehub.ai`, generated fresh 2026-08-25 specifically for this instance (not reused from
staging or the legacy instance). Lives at `/etc/ssl/cloudflare/origin.pem` /
`/etc/ssl/cloudflare/origin.key` on the instance. **Never Let's Encrypt/Certbot** — there is no
Certbot installed or configured on this instance; Cloudflare's edge is what visitors' browsers
actually see, and Full (Strict) mode requires only that the origin presents *a* cert Cloudflare
trusts, which this is.

### Instance access (SSM — no SSH, no key pair exists)

```bash
# Open an interactive shell on the production instance
aws ssm start-session --profile musehub-production --region us-east-1 --target i-043aaed71bef11903

# Run a one-off command
aws ssm send-command \
  --profile musehub-production \
  --instance-ids i-043aaed71bef11903 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker ps"]' \
  --region us-east-1 \
  --query "Command.CommandId" --output text
```

### Useful commands on the instance

Run via SSM (`aws ssm start-session --profile <musehub-production|musehub-nonproduction> --region us-east-1 --target <instance-id>`).
The active app slot is either `musehub-blue` (port 1337) or `musehub-green` (port 1338).

```bash
# Which slot is live?
cat /opt/musehub/.active-slot
cat /etc/nginx/musehub-active-port

# View running containers
sudo docker ps

# Tail live app logs (substitute blue/green as needed)
sudo docker logs -f musehub-blue
sudo docker logs -f musehub-green

# Quick health check
curl -s http://127.0.0.1:1337/healthz   # blue slot
curl -s http://127.0.0.1:1338/healthz   # green slot

# Run Alembic migrations manually (against the live DB)
SLOT=$(cat /opt/musehub/.active-slot)
DB_PASSWORD=$(grep ^DB_PASSWORD /opt/musehub/.env | cut -d= -f2)
sudo docker run --rm \
  --network musehub_musehub-internal \
  --env-file /opt/musehub/.env \
  -e "DATABASE_URL=postgresql+asyncpg://musehub:${DB_PASSWORD}@postgres:5432/musehub" \
  <ecr-image>:<tag> alembic upgrade head

# Postgres shell — actual container name is musehub_postgres (the network
# alias "postgres" used in DATABASE_URL is only resolvable *between*
# containers on the shared network, not usable as a `docker exec` target)
sudo docker exec -it musehub_postgres psql -U musehub -d musehub

# View nginx status
sudo systemctl status nginx
sudo nginx -t
```

---

## Staging Environment

### Purpose

Full production mirror with a separate DB, separate object store, and separate domain.
Used for smoke tests before every prod deploy. Never exposed to users.

### Instance (provisioned by aws-provision-staging.sh)

```
Instance ID : i-07547cd20bee2dea5
Name        : musehub-staging
Elastic IP  : 23.22.27.39
App dir     : /opt/musehub
Domain      : staging.musehub.ai
```

### DNS (musehub.ai) — Cloudflare-hosted, see the production section above for detail

| Type | Host | Origin value | Proxy |
|------|------|-------|-----|
| A Record | staging | `23.22.27.39` | Proxied (orange-cloud) |

### Provisioning (one-time, run locally)

```bash
# 1. Provision EC2 + EIP
chmod +x deploy/aws-provision-staging.sh
./deploy/aws-provision-staging.sh
# Note the instance ID and Elastic IP printed at the end.

# 2. Add staging.musehub.ai A record on Namecheap (see above).
#    Wait for propagation (~5 min with Automatic TTL):
watch -n 10 "dig staging.musehub.ai +short"

# 3. Bootstrap the instance (installs AWS CLI, verifies ECR access)
bash deploy/bootstrap-instance.sh staging

# 4. Run setup script on the instance via SSM
aws ssm send-command \
  --instance-ids <instance-id> \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["chmod +x /opt/musehub/deploy/setup-ec2-staging.sh && /opt/musehub/deploy/setup-ec2-staging.sh"]' \
  --region us-east-1

# 5. Do the first deploy
bash deploy/push.sh staging
```

### Recovering a down staging instance (522 / Bad Gateway)

**Symptom:** `staging.musehub.ai` returns Cloudflare 522 or Bad Gateway.

**Root cause pattern:** The container stopped (either manually or after a reboot)
and `--restart unless-stopped` did not fire because the container was in a
stopped (not crashed) state when the instance last rebooted.

**Fix — one SSM command, no polling:**

```bash
CMD_ID=$(aws ssm send-command \
  --region us-east-1 \
  --instance-ids i-07547cd20bee2dea5 \
  --document-name "AWS-RunShellScript" \
  --parameters '{"commands":["sudo docker start musehub-blue musehub-worker 2>&1 && sudo musehub-set-slot blue && echo done"]}' \
  --query "Command.CommandId" --output text)
echo "Command sent: $CMD_ID"
# Wait ~20s then check once:
sleep 20 && aws ssm get-command-invocation \
  --region us-east-1 \
  --command-id "$CMD_ID" \
  --instance-id i-07547cd20bee2dea5 \
  --query "[Status,StandardOutputContent]" --output text
```

Check `staging.musehub.ai` in the browser — it should be back.

**Critical rules when using SSM to recover staging:**

1. **Never reboot to fix SSM Pending.** A reboot stops containers that were
   manually started — `--restart unless-stopped` only auto-starts containers
   that were *running* (not stopped) at reboot time. Rebooting to fix SSM
   will take the site down and require a manual `docker start` anyway.

2. **Never poll SSM in a loop.** The shell `until`/`while sleep` pattern
   freezes the terminal and masks whether the command succeeded. Send the
   command, wait a fixed interval, fetch once.

3. **SSM Pending ≠ SSM broken.** The agent can show `Online` but queue commands
   as `Pending` for 10–30 seconds after a fresh start. Wait before concluding
   SSM is broken.

4. **InProgress means it will complete.** If a command shows `InProgress` it
   is executing on the instance — do not cancel or resend. Check back in 30s.

### Ongoing code deploys to staging

```bash
# Standard — builds image locally, pushes to ECR, triggers blue-green on staging
bash deploy/push.sh staging
```

### Publishing a new muse CLI release

The `install.sh` script (served at `https://staging.musehub.ai/install.sh`) downloads
`muse-{version}.tar.gz` from `/releases/`. The version comes from `musehub/protocol/version.py`
(`MUSE_VERSION`), which tracks the musehub package version.

To ship a new muse build:

```bash
# From ~/ecosystem/musehub — builds sdist, uploads to S3, SSMs to staging,
# cleans up old tarballs (keeps 3), and verifies the URL is live.
bash deploy/publish_muse_release.sh
```

What it does:
1. Builds `muse-{version}.tar.gz` from `~/ecosystem/muse`
2. Uploads to `s3://musehub-releases/muse-{version}.tar.gz`
3. SSMs to staging to copy from S3 → `/data/releases/` (Docker volume)
4. Deletes stale tarballs from S3 and the server (keeps the 3 newest)
5. Smoke-tests `https://staging.musehub.ai/releases/muse-{version}.tar.gz`

**Note:** SSH is blocked on the instance (port 443 only). All server commands go through
AWS SSM (`musehub-infra` IAM user). The staging instance (`i-07547cd20bee2dea5`) has the
required IAM instance profile; no other instance does.

To test the install script end-to-end locally:
```bash
curl -fsSL https://staging.musehub.ai/install.sh | sh
# verify
~/.local/bin/muse --version
# cleanup
rm -rf ~/.local/share/muse/venv && rm -f ~/.local/bin/muse
```

### Instance access (SSM — no SSH)

```bash
# Interactive shell on staging
aws ssm start-session --target i-07547cd20bee2dea5 --region us-east-1
```

---

## Deployment Workflow

Deploys are image-based via ECR. No SSH, no rsync, no code on the instance after provisioning.
All deploy commands run from the local `~/ecosystem/musehub` directory.

### Deploy pipeline overview

```
Local machine (push.sh):
  1. docker build (linux/amd64)
  2. docker save → tar, crane push → ECR (musehub/musehub:<tag>)
  3. aws ssm send-command → sync deploy.sh, then run it

Instance (deploy.sh via SSM):
  4. deploy.sh written from local copy (always current — never stale)
  5. aws ecr get-login-password | docker login
  6. docker pull <ecr>:<tag>
  7. docker run (migrations only, then exit)
  8. docker run -d (new slot — blue or green)
  9. curl /healthz until healthy
  10. nginx -s reload (zero-downtime flip)
  11. docker rm (old slot)
```

**Key invariant:** `push.sh` always writes the current local `deploy.sh` to the instance
via SSM before running it. This means the instance's `deploy.sh` is always in sync with
the local repo — there is no separate "sync the deploy scripts" step.

### ECR Push — Use crane (not docker push)

`docker push` to ECR routes through Docker Desktop's VPNKit proxy
(`http.docker.internal:3128` / `192.168.65.1:3128` from inside the VM). After a local IP
change or a Docker Desktop restart, the VPNKit proxy drops connections mid-upload on large
layer pushes, producing broken pipe errors. The fix is **crane** — Google's container
registry tool — which pushes images directly from the macOS host network, bypassing the
Docker Desktop VM layer and its proxy entirely.

**crane is the standard push method. Never use `docker push` to ECR.**

Install once:

```bash
brew install crane
```

`push.sh` calls crane internally. If pushing manually outside the script:

```bash
# 1. Build the image locally (linux/amd64 target)
docker build --platform linux/amd64 -t musehub/musehub:latest .

# 2. Save to a tar archive on the host
docker save musehub/musehub:latest -o /tmp/musehub-latest.tar

# 3. Authenticate crane against ECR
aws ecr get-login-password --region us-east-1 \
  | crane auth login 992382692655.dkr.ecr.us-east-1.amazonaws.com \
      --username AWS --password-stdin

# 4. Push with crane (runs entirely on the macOS host — no VPNKit involved)
crane push /tmp/musehub-latest.tar \
  992382692655.dkr.ecr.us-east-1.amazonaws.com/musehub/musehub:latest
```

---

### Standard deploy

```bash
# Deploy to staging
bash deploy/push.sh staging
```

> ⚠️ **`bash deploy/push.sh prod` does NOT deploy to the new Production instance/account.**
> `deploy/push.sh`'s hardcoded `PROD_INSTANCE` still points at the legacy `i-0855d6efe7fa1a49d`
> in the shared Nonproduction account — it was never updated for the new isolated Production
> account (`672469410277`, instance `i-043aaed71bef11903`, dedicated ECR repo). Adding an
> explicit `production` target with account-ID guards to `push.sh`/`deploy.sh` is open follow-up
> work (readiness ticket #159). Until that exists, deploying a new build to Production means
> repeating the manual SSM-driven sequence used for the initial launch: `crane copy` (or rebuild)
> the image into the Production ECR repo, then `docker pull` + blue/green swap via SSM commands
> against `i-043aaed71bef11903` directly.

### Rollback

```bash
# List recent ECR image tags (Nonproduction registry — staging only)
aws ecr describe-images \
  --profile musehub-nonproduction \
  --repository-name musehub/musehub \
  --region us-east-1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-10:].imageTags[0]' \
  --output table

# Redeploy a specific tag to staging (skips build+push)
IMAGE_TAG=<previous-tag> bash deploy/push.sh staging
```

Production rollback (manual, until `push.sh` supports a real `production` target): re-run the same
manual blue/green `docker run` sequence used for the initial launch, substituting the previous
image tag in the Production ECR repo (`672469410277.dkr.ecr.us-east-1.amazonaws.com/musehub/musehub`).
**Not yet rehearsed** — this is an open item, not a tested procedure.

### Emergency migration rollback (on instance via SSM)

```bash
# Nonproduction / staging
aws ssm send-command \
  --profile musehub-nonproduction \
  --instance-ids i-07547cd20bee2dea5 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/musehub && sudo docker run --rm --network musehub_musehub-internal --env-file .env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1

# Production
aws ssm send-command \
  --profile musehub-production \
  --instance-ids i-043aaed71bef11903 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker run --rm --network musehub_musehub-internal --env-file /opt/musehub/.env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1
```

---

## Backups

**Production has real, verified automated backups as of 2026-08-25** (readiness ticket #157,
closed — this had been the single highest-priority open item across the entire
production-readiness effort). **Staging still has none** — the one manual dump that ever existed
there was 20 bytes (empty/failed), and that gap hasn't been replicated to production.

Production backup pipeline (`deploy/backup.sh`, run daily at 3 AM via cron):

1. `pg_dump` against `musehub_postgres` (not via `docker compose` — production has no
   `docker-compose.yml`; the script was fixed to `docker exec` directly), compressed, written to
   `/opt/backups/musehub/` (14-day local retention).
2. Synced to a **dedicated** R2 bucket, `musehub-production-backups` — deliberately separate from
   the app's own `musehub-prod` object-storage bucket, via a **separate**, narrowly-scoped R2 API
   token (`musehub-production-backups-r2`, Object Read & Write on that bucket only) — a compromise
   of the app's live data credential doesn't grant access to backups, or vice versa. 90-day
   retention in R2.
3. A real restore was tested (not just "the backup runs"): restored into a throwaway
   `postgres:16-alpine` container, confirmed all 74 tables came back correctly, tore the container
   down. This is what actually proves the pipeline works, not the existence of a file.

**Known `rclone`-on-R2 gotcha, worth knowing if this is ever reconfigured**: an R2 API token scoped
to "Object Read & Write" on one specific bucket doesn't include bucket-level permissions
(`HeadBucket`/`CreateBucket`). Older `rclone` versions (this instance has 1.53.3, from Ubuntu's
apt repo) perform a bucket-existence check before uploading by default, which fails with a
confusing `403 AccessDenied` even though the credentials are completely correct — the fix is
`no_check_bucket = true` in the remote config, not broader token permissions.

Volume snapshot via AWS before every production deploy is still not automated — remains a manual
step for now. Object store (R2, content-addressed) is inherently safe to snapshot at any time;
that was never the actual gap — the Postgres database was.

**Staging backups are the reasonable next follow-up** — everything built for production
(bucket/token pattern, cron, restore-testing method) is directly reusable, just not yet repeated
for staging.

---

## Costs (approximate, us-east-1, 2026 pricing)

| Item | $/month |
|------|---------|
| t3.medium (production) | ~$30 |
| t3.small (legacy `musehub-prod`, not yet decommissioned) | ~$15 |
| t3.medium (staging) | ~$30 |
| Elastic IPs (3) | ~$0 while associated, $3.60/mo each if unassociated |
| EBS gp3 (20GB staging/legacy, 30GB production) | ~$1.60–$2.40 each |
| CloudTrail S3 storage (2 accounts, 365-day lifecycle) | negligible at current volume |
| **Total (all running, including undecommissioned legacy instance)** | **~$80/mo** |

Decommissioning the legacy `musehub-prod` instance (a separate, deliberately deferred decision)
would bring this down to roughly the same ~$60/mo as running production + staging alone.

To pause staging when not needed:

```bash
aws ec2 stop-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
# Start again with:
aws ec2 start-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
```

The Elastic IP stays associated while the instance is stopped — no charge.

---

## Secrets inventory

**Production** (as of 2026-08-25) sources all 6 secrets from **AWS SSM Parameter Store**
(`/musehub/production/*`, KMS-encrypted `SecureString`, default `alias/aws/ssm` key) via
`deploy/secrets.sh` — the pipeline the script was always built for, now actually the live path.
`.env` on the instance is *generated from SSM*, not hand-edited; re-running `secrets.sh` refreshes
it from the current SSM values. The instance role has read-only access
(`ssm:GetParameter`/`GetParametersByPath` + `kms:Decrypt`, scoped to exactly
`/musehub/production/*`) — no write access, by design; rotation is an operator action from an
admin session, not something the instance does to itself.

**Staging** still only has 3 of 7 parameters in SSM (`PACK_WORKER_URL`, `WEBHOOK_SECRET_KEY`,
`WORKER_INTERNAL_KEY`) — its `.env` remains hand-provisioned. Completing staging's migration to
match production is a reasonable follow-up, not yet done.

Never committed to source, either way.

| Secret | How generated | Rotation | SSM path (production) |
|--------|---------------|----------|------------------------|
| `DB_PASSWORD` | `openssl rand -hex 16` | Manual, on compromise | `/musehub/production/DB_PASSWORD` |
| `WEBHOOK_SECRET_KEY` | Fernet-equivalent (stdlib `base64.urlsafe_b64encode(os.urandom(32))` — no extra dependency needed) | Manual, on compromise | `/musehub/production/WEBHOOK_SECRET_KEY` |
| `RUNNER_TOKEN` | `openssl rand -hex 32` | Manual, on compromise | `/musehub/production/RUNNER_TOKEN` |
| `BLOB_STORAGE_ACCESS_KEY_ID` / `BLOB_STORAGE_SECRET_ACCESS_KEY` | Cloudflare R2 API token, scoped to `musehub-prod` bucket only | Manual, on compromise | `/musehub/production/BLOB_STORAGE_ACCESS_KEY_ID`, `.../BLOB_STORAGE_SECRET_ACCESS_KEY` |
| `WORKER_INTERNAL_KEY` | `openssl rand -hex 32` | Manual, on compromise. **Known gap**: no production Cloudflare Worker exists yet to share this key with — the only pack-receiver Worker deployed (`musehub-pack-receiver-staging`, routed from `packs.musehub.ai`) is staging-only. This key won't do anything useful until a production Worker is built and configured with it. | `/musehub/production/WORKER_INTERNAL_KEY` |

Ed25519 identity keys live in `~/.muse/identity.toml` on each client machine.
No server-side secret is involved in MSign auth — the public key in the DB is the credential.