Production readiness — Sections 10-11: Automated Staging and Production Deployment

Reviewed with full detail (10-11-automated-deployment.md).

Fixed the actual blocker (2026-08-26): deploy/push.sh was hardcoded to a stale target for "prod" — an old ECR registry, an instance ID (i-0855d6efe7fa1a49d) that no longer exists, and the musehub-infra credential, which lives in Nonproduction and has zero access to the real Production account. Running push.sh prod today would have silently pointed at the wrong account. Fixed:

• push.sh now has per-environment config (instance ID, ECR registry, AWS profile) for staging vs. prod — they're genuinely separate AWS accounts, not a shared registry
• Prod now authenticates via the operator's own musehub-production IAM Identity Center SSO session, matching what docs/infrastructure.md already documented as the intended model (no shared IAM user in Production, by design)
• Added an auth precheck that fails fast with a clear aws sso login --profile musehub-production message if the SSO session is stale, instead of failing deep into the script
• deploy/deploy.sh needed no changes — it already takes ECR_IMAGE as a parameter rather than hardcoding it

Still correctly deferred — this is real, substantial future work, not something needed for launch: the actual CI/CD automation ask (trigger-on-merge, OIDC-style short-lived credentials, approval gates, notifications, structured deploy records, concurrent-deploy locking, smoke tests). None of that blocked going live and none of it should be rushed now. Section 9's branch-protection dependency still holds for whenever an actual trigger-on-merge pipeline gets built.

Leaving open as backlog for the real automation work — but the concrete "prod deploy is currently broken" blocker this ticket flagged is resolved.

Rollback tested live on staging (2026-08-28) — one of the few remaining "not done" checklist items, now closed out:

1. Deployed a fresh build to staging (8e05daa2-20260828183740)
2. Rolled back via IMAGE_TAG=1705df12-20260808183652 bash deploy/push.sh staging
3. Confirmed via docker ps on the instance that the old image was actually running — not just that the command exited 0
4. Confirmed /healthz stayed green throughout (no downtime, health-gated blue/green swap worked as documented)
5. Rolled forward again to restore the current build, confirmed healthy

Same deploy/push.sh mechanism serves both environments, so this validates the production rollback path too, though it hasn't been separately exercised against production itself yet.

Docs updated in 10-11-automated-deployment.md (both the Section 10 and Section 11 checklist items).

Remaining scope on this ticket is still the real CI/CD automation build (trigger-on-merge, OIDC-style credentials, approval gates, notifications, structured deploy records, concurrent-deploy locking) — correctly deferred, not needed for launch.

Closing this out for launch purposes. Recap of what's actually done vs. what remains:

Done:

• Fixed the real blocker: push.sh was silently targeting the old pre-rebuild instance/account for "prod" — now correctly targets the real Production AWS account with its own SSO-based auth
• Fixed container memory limits and the hardcoded staging-only CloudWatch log group (found while debugging the muse/musehub repo pushes to production)
• Rollback tested live end-to-end on staging (deploy → rollback → verify old image actually running → roll forward), not just "the command exists"
• Manual deploy path (deploy/push.sh) is solid: blue/green, health-gated, automatic migrations, documented rollback

Deferred, not abandoned: the remaining scope here (trigger-on-merge, a real job-runner client, OIDC-style short-lived credentials, approval gates, notifications, structured deploy records, concurrent-deploy locking) is the same underlying infrastructure as MuseHub's planned pro-tier CI/CD feature — not just internal ops tooling. Building a one-off internal-only version now would be duplicate work once that feature actually gets built. Tracked separately as a roadmap item rather than production-readiness debt.

Closing as done for what's achievable/necessary pre-launch.