# Section 7 — Secrets and Application Configuration

> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).

## Update (2026-09-07) — staging SSM migration complete, pipeline proven end-to-end

The actual work item this section flagged is done: all 4 missing SSM parameters were populated
for staging, and `deploy/secrets.sh` was run for real against the staging instance and succeeded.

### What was actually involved (more than just "add the parameters")

- **Found a real gap while doing this**: staging's instance role (`musehub-ec2-ssm`) had *no* SSM
  Parameter Store permissions at all — only `AmazonSSMManagedInstanceCore` (agent connectivity) and
  ECR pull. `deploy/secrets.sh` would have failed on staging even with all 7 parameters populated.
  Added a scoped `musehub-ssm-parameter-read` inline policy (`ssm:GetParameter(s)` on
  `/musehub/staging/*` only), matching the least-privilege pattern already used for production's
  role.
- **Migrated the 4 missing values without ever exposing them**: temporarily granted the instance
  role `ssm:PutParameter` (scoped to `/musehub/staging/*`), ran a script entirely server-side that
  read the existing hand-provisioned `.env` and wrote each value straight into SSM as
  `SecureString`, then immediately revoked the write grant — the instance role now has read-only
  access, same as production. No secret value ever appeared in any tool output or log.
- **Ran `deploy/secrets.sh` for real** (`MUSEHUB_ENV=staging`): fetched all 7 parameters, wrote a
  fresh `/opt/musehub/.env`, passed its own weak-password/length sanity check, backed up the
  previous `.env` automatically. Confirmed staging still healthy afterward (`/healthz` green) —
  values are byte-identical to what was already running, so no restart was needed to prove this;
  `deploy.sh` itself doesn't call `secrets.sh` (they're intentionally decoupled — `secrets.sh`
  regenerates `.env`, `deploy.sh` just uses whatever `.env` exists via `--env-file`).

### Full secrets inventory — now identical shape on both environments

| Secret | Staging | Production |
|---|---|---|
| `DB_PASSWORD` | ✅ in SSM (was missing) | ✅ in SSM |
| `WEBHOOK_SECRET_KEY` | ✅ in SSM | ✅ in SSM |
| `RUNNER_TOKEN` | ✅ in SSM (was missing) | ✅ in SSM |
| `BLOB_STORAGE_ACCESS_KEY_ID` / `_SECRET_ACCESS_KEY` | ✅ in SSM (was missing) | ✅ in SSM |
| `WORKER_INTERNAL_KEY` | ✅ in SSM | ✅ in SSM |
| `PACK_WORKER_URL` | ✅ in SSM (non-secret) | ✅ in SSM (non-secret) |
| `BACKUP_R2_BUCKET` | Not set — staging backups aren't built yet (separate future task) | ✅ in SSM |

**Decision on "should production adopt this from day one" — already resolved.** Production was
built with the SSM pipeline from its first deploy, never had a hand-provisioned `.env` at all. This
section's remaining question was purely about backfilling staging to match, which is now done.

## Checklist assessment — updated

- [x] Inventory all secrets.
- [x] Remove secrets from source control and repository history.
- [x] **Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate** — done for
  both environments now, not just drafted.
- [x] **Encrypt secrets with KMS** — all 7 staging parameters and all 7 production parameters are
  `SecureString` (KMS-encrypted by default).
- [x] Use separate staging and production secrets — separate SSM path prefixes, separate R2
  buckets, confirmed.
- [ ] Give workloads access only to the secrets they require — still N/A/likely-moot for this
  single-process-per-instance architecture, not re-litigated in this pass.
- [x] Do not expose production secrets to pull-request workflows — still N/A, no CI exists yet.
- [ ] Do not print secrets in logs — still not verified; needs a log-content review, which itself
  depends on Section 12's centralized logging (that exists now — worth actually doing this review
  as a follow-up, see below).
- [ ] Configure secret rotation where supported — not done.
- [ ] Rotate credentials during the production launch — not done; still deferred to Section 18.
- [ ] Document emergency rotation procedures — not done.
- [x] Separate secrets from nonsecret configuration.
- [x] Validate required configuration at application startup.
- [x] Fail safely when critical configuration is missing — now proven true for staging too, not
  just production (the sanity check ran for real and passed).
- [x] Remove default credentials and sample keys.
- [ ] Audit OAuth callback URLs and API allowlists for production — still N/A, no OAuth exists.
- [ ] Audit every external webhook secret — `WORKER_INTERNAL_KEY` still not independently audited
  against `musehub/worker.py`'s validation logic.
- [ ] Document ownership and rotation frequency for every production secret — still not done;
  same blocker as before (Section 1's ownership-recording work), though Section 1 is now
  substantially further along than when this was first written (see #150/#175).

## Remaining work

1. Log-content review for accidental secret leakage — now actually possible given Section 12's
   logging is live; wasn't possible before (no logs existed to review).
2. `WORKER_INTERNAL_KEY` validation audit in `musehub/worker.py`.
3. Secret rotation policy + emergency rotation procedure — doesn't exist for any secret yet.
4. Staging backups (and its own `BACKUP_R2_BUCKET`) — separate task, not this section's scope.
5. Per-secret ownership/rotation-frequency documentation — blocked on Section 1's per-area
   ownership assignment (#175).