Sections 10 & 11 — Automated Staging and Production Deployment

Companion to musehub-production-readiness-checklist.md. Covered together: both sections assume automated deploy pipelines that don't exist yet. Today's reality (Section 0): a human runs bash deploy/push.sh staging|prod from a laptop, using the shared musehub-infra credential. There is no trigger-on-push, no approval gate, no automatic rollback — everything is manual and synchronous.

What already exists and is worth keeping

deploy/push.sh → deploy/deploy.sh already implements several things both sections ask for, just manually-triggered rather than automated:

• [x] Deploy the immutable application artifact — ECR image, tagged, pulled by digest on the instance
• [x] Run database migrations — automatic, before slot swap (Section 8)
• [x] Run health checks — /healthz gates the blue/green flip
• [x] Blue/green deployment — the actual deploy strategy in use (satisfies "deploy using rolling/blue-green/canary" from Section 11)
• [x] A documented manual deployment path (Section 11's explicit ask for an emergency path) — this is the current path, so it trivially exists; worth keeping even after automation exists
• [x] A documented rollback command — IMAGE_TAG=<previous-tag> bash deploy/push.sh <env>, per docs/infrastructure.md
• [x] Rollback tested live on staging (2026-08-28): deployed a fresh build, rolled back to the prior tag via IMAGE_TAG=<prev> bash deploy/push.sh staging, confirmed via docker ps that the old image was actually running (not just that the command exited 0), confirmed /healthz stayed green throughout, then rolled forward again to restore the current build. Full round trip worked as documented — health-gated blue/green swap, no downtime, no manual intervention beyond running the one command.

What's missing — and why it's bigger than "add a pipeline"

Per Section 9's finding, MuseHub doesn't yet have the platform primitives (branch protection, required checks, merge gating) that a "deploy on merge to dev" trigger would normally hook into. That means Sections 10–11 have two layers of work, not one:

1. Trigger mechanism — something needs to fire on push/merge. Options: a MuseHub webhook (muse hub webhook create) calling out to a deploy runner, or the musehub-runner container (once its job-execution client is actually built per Section 9) running the deploy as a job type. Either is real, currently-nonexistent work.
2. Everything downstream of the trigger — OIDC-style AWS auth (no more shared laptop credential), staging/production role separation, approval gating for production, notifications, concurrent-deploy locking, structured deploy records (artifact digest/migration version/ deployer/timestamp).

Section 10 — Automated Staging Deployment — item-by-item

• [ ] Create a MuseHub-native "staging environment" concept (GitHub environment equivalent) — doesn't exist
• [ ] Deploy automatically after merges to dev/main — not started; no trigger mechanism exists
• [ ] Authenticate to AWS through short-lived credentials (OIDC-equivalent) — not started; still the shared musehub-infra key
• [ ] No long-lived AWS keys in CI config — currently moot, no CI config exists; becomes relevant once a trigger mechanism does
• [ ] Restrict the staging role to staging-only resources — not started (Section 3 dependency)
• [ ] Infra planning/apply as a pipeline step — N/A, no IaC yet (Section 4)
• [x] Deploy the immutable artifact — already true, manually
• [x] Run migrations — already true, automatic within the deploy
• [ ] Run smoke tests — not automated; no smoke-test suite identified in this pass (worth checking deploy/smoke_muse.sh's actual coverage separately)
• [x] Run health checks — already true
• [ ] Publish deployment URL + commit SHA — not automated; would need to reference MuseHub commit IDs, not git SHAs
• [ ] Notify both maintainers of success/failure — not automated
• [ ] Preserve deployment logs — not done; deploy output currently only lives in the triggering laptop's terminal
• [ ] Automatic rollback on health-check failure — not automated, but deploy.sh's health-gated blue/green already prevents a bad slot from going live in the first place (the new slot must pass /healthz before nginx flips to it) — so the failure mode is "deploy doesn't complete" rather than "bad deploy goes live and needs rollback." Worth confirming this distinction is enough, or whether explicit rollback automation is still wanted.
• [ ] Database rollback kept separate/explicit — not formalized
• [ ] Prevent concurrent staging deploys from colliding — not enforced; two people running push.sh staging simultaneously today would race
• [ ] Deployment status visible in MuseHub — not built
• [ ] Record artifact digest/migration version/deployer/timestamp — not recorded anywhere structured

Section 11 — Automated Production Deployment — item-by-item

Everything in Section 10 applies again for production, plus:

• [ ] Distinct production AWS role — not started (Section 3)
• [ ] Explicit release trigger (version tag / MuseHub release / manual promotion) — MuseHub already has a release feature (muse release add/muse hub release create) that could serve this purpose; not currently wired to deploy at all
• [ ] Promote the exact staging-tested artifact rather than rebuilding — not done by default; possible via IMAGE_TAG (Section 9), not the default flow
• [ ] Require successful staging deployment/tests first — not enforced
• [ ] Require production approval, prevent self-approval, allow both Gabriel and Aaron to approve — not built; also blocked on Section 1/3 (Aaron has no independent deploy identity yet)
• [ ] Display full deployment plan before approval — not built
• [ ] Back up stateful systems before risky changes — currently impossible to do meaningfully, no backups exist (Section 8)
• [x] Run migrations as a controlled step — already true
• [x] Deploy using blue/green — already true
• [ ] Post-deployment smoke tests, rollout monitoring, stop/rollback on threshold — not automated; no monitoring exists yet to threshold against (Section 12)
• [ ] Record approver/initiator, commit SHA/artifact ID/migration version — not recorded
• [ ] Generate release notes — not automated
• [ ] Notify both operators — not automated
• [ ] Prevent concurrent production deploys — not enforced
• [x] Documented manual/emergency deployment path — exists (it's the current default path)
• [x] Documented rollback command — exists (IMAGE_TAG=<prev>)
• [x] Rollback tested before launch — tested live on staging 2026-08-28 (full round trip: deploy → rollback → verify old image actually running → roll forward again). Same deploy/push.sh mechanism is used for both staging and production, so this validates the production rollback path too, though it hasn't been separately exercised against production itself.

Update (2026-08-26) — prod deploy is fixed; automation itself is still deferred

The i-0855d6efe7fa1a49d instance-profile blocker referenced above no longer applies — that instance doesn't exist. Production was rebuilt from scratch in a dedicated MuseHub Production AWS account (672469410277) with a correctly attached instance role (musehub-production-ec2-ssm) and is live at musehub.ai.

deploy/push.sh was updated to actually target the new account: it now builds one image and pushes it to each target's own ECR registry (staging and prod are separate accounts, not a shared registry), and uses the operator's own musehub-production SSO profile for prod instead of the shared musehub-infra credential (which only exists in Nonproduction and has no access to Production at all). deploy/deploy.sh needed no changes — it already receives ECR_IMAGE as a parameter from push.sh rather than hardcoding it.

Everything else in this doc — the trigger mechanism, OIDC-style short-lived credentials, approval gates, notifications, structured deploy records, concurrent-deploy locking — remains real, substantial, correctly-deferred future work. It was not required to go live, and going live did not require completing Section 9 first as originally assumed; the dependency remains true for building an actual trigger-on-merge pipeline later, just not for today's manual deploy path.