gabriel / musehub public
runbook-deploy-onboarding.md markdown
76 lines 4.6 KB
Raw
sha256:d47368ca2aca9e5eb1af87830daba9201111f8d38fa86b0f4cfa8b0d069f0f3b docs: add deploy onboarding runbook for expanded 5-person t… Sonnet 5 24 days ago

Deploy Runbook — Team Onboarding

Companion to musehub-production-readiness-checklist.md. Written because deploy responsibility is expanding from Gabriel alone to a 5-person team: Gabriel, Aaron (aaronrene), and interns Sol, Toby, and Ben. This changes the risk calculus on several already-flagged items (Sections 1/3) — more people sharing one AWS credential is a bigger problem than two people sharing it.

Access tiers (decided 2026-08-08)

Person Staging Production Notes
Gabriel Full Full Admin — AWS console, secrets, DNS, billing
Aaron Full Full (approval) Operator — can deploy/approve production
Sol, Toby, Ben Full, once Section 3 ships None Interns — staging-only by design (Gabriel's decision)

⚠️ Interim state: do not hand out the current AWS credential to interns

Today, every deploy uses one shared AWS IAM user (musehub-infra) whose access key lives in ~/.aws/credentials on whoever's laptop runs push.sh. That key can reach both staging and production. Handing it to three additional people — especially before any individual-identity system exists — would mean:

  • No way to give interns staging-only access; the key doesn't distinguish environments by user
  • No accountability for who ran what (Section 1's exact concern, now with 5 people instead of 2)
  • Any one person's compromised laptop compromises deploy access for everyone

Recommendation: do not distribute the current musehub-infra key to Sol, Toby, or Ben. Interns should not deploy to staging for real until Section 3's IAM Identity Center work ships with a staging-scoped permission set they can use under their own identity. This is a short-term constraint, not a permanent one — it's the reason Section 3 was rescoped to cover all 5 people from the start rather than just Gabriel and Aaron.

In the meantime, interns can:

  • Read docs/infrastructure.md and this runbook to learn the deploy flow
  • Watch a deploy happen (screen-share or logs) with Gabriel or Aaron
  • Practice against a local dev environment (not staging/production) if one exists

Target-state staging deploy (once Section 3 ships individual identities)

  1. Authenticate as yourself: aws sso login --profile musehub-staging-operator (exact command depends on how Identity Center is configured — placeholder until Section 3 is built).
  2. From ~/ecosystem/musehub: bash deploy/push.sh staging.
  3. Watch the output — it builds the image, pushes to ECR via crane, SSMs the deploy script to the instance, runs migrations, waits for the new slot's /healthz to pass, then flips nginx.
  4. Verify: curl -s https://staging.musehub.ai/healthz and spot-check the site in a browser.
  5. If anything looks wrong, do not try to fix it solo — see Escalation below.

Production deploy (Gabriel and Aaron only, once prod is actually fixed)

⚠️ Production deploy is currently broken — the prod EC2 instance has no IAM instance profile, so push.sh prod fails outright (InvalidInstanceId) regardless of who runs it. This must be fixed (attach the musehub-ec2-ssm role to i-0855d6efe7fa1a49d) before any production deploy, onboarding or not.

Once fixed, the flow is the same as staging (bash deploy/push.sh prod), but per Section 11's still-open items: there is currently no approval gate, no promotion-of-tested-artifact enforcement, and no automatic rollback. Until those exist, manually confirm staging is healthy and has been running the candidate build without issues before deploying the same build to production.

Escalation

  • Staging is down (522/Bad Gateway): docs/infrastructure.md's "Recovering a down staging instance" section has the exact SSM recovery command. Interns: don't attempt this solo yet — pair with Gabriel or Aaron the first few times.
  • Unsure whether something is safe to do: ask first. This applies especially to anything destructive (--force, --hard, deleting resources, rotating secrets, touching DNS/IAM/security groups) — none of that is deploy-related and none of it should be attempted without Gabriel or Aaron's explicit go-ahead, regardless of who's doing it.

What no one should do without Gabriel or Aaron's explicit sign-off

  • Any production action (deploy, rollback, migration, secret rotation)
  • Any AWS console change to IAM, security groups, DNS, or billing
  • Any destructive command against staging (dropping data, force-resetting the instance)
  • Distributing AWS credentials to anyone not already listed in the access-tier table above
File History 1 commit
sha256:d47368ca2aca9e5eb1af87830daba9201111f8d38fa86b0f4cfa8b0d069f0f3b docs: add deploy onboarding runbook for expanded 5-person t… Sonnet 5 24 days ago