Section 7 — Secrets and Application Configuration

Companion to musehub-production-readiness-checklist.md.

Update (2026-09-07) — staging SSM migration complete, pipeline proven end-to-end

The actual work item this section flagged is done: all 4 missing SSM parameters were populated for staging, and deploy/secrets.sh was run for real against the staging instance and succeeded.

What was actually involved (more than just "add the parameters")

• Found a real gap while doing this: staging's instance role (musehub-ec2-ssm) had no SSM Parameter Store permissions at all — only AmazonSSMManagedInstanceCore (agent connectivity) and ECR pull. deploy/secrets.sh would have failed on staging even with all 7 parameters populated. Added a scoped musehub-ssm-parameter-read inline policy (ssm:GetParameter(s) on /musehub/staging/* only), matching the least-privilege pattern already used for production's role.
• Migrated the 4 missing values without ever exposing them: temporarily granted the instance role ssm:PutParameter (scoped to /musehub/staging/*), ran a script entirely server-side that read the existing hand-provisioned .env and wrote each value straight into SSM as SecureString, then immediately revoked the write grant — the instance role now has read-only access, same as production. No secret value ever appeared in any tool output or log.
• Ran deploy/secrets.sh for real (MUSEHUB_ENV=staging): fetched all 7 parameters, wrote a fresh /opt/musehub/.env, passed its own weak-password/length sanity check, backed up the previous .env automatically. Confirmed staging still healthy afterward (/healthz green) — values are byte-identical to what was already running, so no restart was needed to prove this; deploy.sh itself doesn't call secrets.sh (they're intentionally decoupled — secrets.sh regenerates .env, deploy.sh just uses whatever .env exists via --env-file).

Full secrets inventory — now identical shape on both environments

Decision on "should production adopt this from day one" — already resolved. Production was built with the SSM pipeline from its first deploy, never had a hand-provisioned .env at all. This section's remaining question was purely about backfilling staging to match, which is now done.

Checklist assessment — updated

• [x] Inventory all secrets.
• [x] Remove secrets from source control and repository history.
• [x] Store secrets in AWS Secrets Manager or SSM Parameter Store as appropriate — done for both environments now, not just drafted.
• [x] Encrypt secrets with KMS — all 7 staging parameters and all 7 production parameters are SecureString (KMS-encrypted by default).
• [x] Use separate staging and production secrets — separate SSM path prefixes, separate R2 buckets, confirmed.
• [ ] Give workloads access only to the secrets they require — still N/A/likely-moot for this single-process-per-instance architecture, not re-litigated in this pass.
• [x] Do not expose production secrets to pull-request workflows — still N/A, no CI exists yet.
• [ ] Do not print secrets in logs — still not verified; needs a log-content review, which itself depends on Section 12's centralized logging (that exists now — worth actually doing this review as a follow-up, see below).
• [ ] Configure secret rotation where supported — not done.
• [ ] Rotate credentials during the production launch — not done; still deferred to Section 18.
• [ ] Document emergency rotation procedures — not done.
• [x] Separate secrets from nonsecret configuration.
• [x] Validate required configuration at application startup.
• [x] Fail safely when critical configuration is missing — now proven true for staging too, not just production (the sanity check ran for real and passed).
• [x] Remove default credentials and sample keys.
• [ ] Audit OAuth callback URLs and API allowlists for production — still N/A, no OAuth exists.
• [ ] Audit every external webhook secret — WORKER_INTERNAL_KEY still not independently audited against musehub/worker.py's validation logic.
• [ ] Document ownership and rotation frequency for every production secret — still not done; same blocker as before (Section 1's ownership-recording work), though Section 1 is now substantially further along than when this was first written (see #150/#175).

Remaining work

1. Log-content review for accidental secret leakage — now actually possible given Section 12's logging is live; wasn't possible before (no logs existed to review).
2. WORKER_INTERNAL_KEY validation audit in musehub/worker.py.
3. Secret rotation policy + emergency rotation procedure — doesn't exist for any secret yet.
4. Staging backups (and its own BACKUP_R2_BUCKET) — separate task, not this section's scope.
5. Per-secret ownership/rotation-frequency documentation — blocked on Section 1's per-area ownership assignment (#175).