gabriel / musehub public
runbook-deploy-onboarding.md markdown
81 lines 5.3 KB
Raw
sha256:cd5c2fcb91a44ac9e38e9c36176c27ce079a074586d9852296da628a29fb01ff Merge 'docs/aws-identity-and-deploy-fixes' into 'dev' — pro… Human 3 days ago

Deploy Runbook — Team Onboarding

Companion to musehub-production-readiness-checklist.md. Written because deploy responsibility is expanding from Gabriel alone to a 5-person team: Gabriel, Aaron (aaronrene), and interns Sol, Toby, and Ben. This changes the risk calculus on several already-flagged items (Sections 1/3) — more people sharing one AWS credential is a bigger problem than two people sharing it.

Access tiers (decided 2026-08-08; Aaron's tier updated 2026-08-26)

Person Staging Production Notes
Gabriel Full Full Admin — AWS console, secrets, DNS, billing
Aaron Full Full, unrestricted AdministratorAccess on all three AWS accounts (Management, Nonproduction, Production) via IAM Identity Center. Originally planned as a scoped Operator role with a production approval gate; Gabriel deliberately upgraded this given Aaron's 50/50 ownership stake and intent to independently own half of infra maintenance.
Sol, Toby, Ben Full, once Section 3 ships None Interns — staging-only by design (Gabriel's decision). Still deferred as of 2026-08-26 — no active need or timeline yet.

⚠️ Interim state: do not hand out the current AWS credential to interns

Today, every deploy uses one shared AWS IAM user (musehub-infra) whose access key lives in ~/.aws/credentials on whoever's laptop runs push.sh. That key can reach both staging and production. Handing it to three additional people — especially before any individual-identity system exists — would mean:

  • No way to give interns staging-only access; the key doesn't distinguish environments by user
  • No accountability for who ran what (Section 1's exact concern, now with 5 people instead of 2)
  • Any one person's compromised laptop compromises deploy access for everyone

Recommendation: do not distribute the current musehub-infra key to Sol, Toby, or Ben. Interns should not deploy to staging for real until Section 3's IAM Identity Center work ships with a staging-scoped permission set they can use under their own identity. This is a short-term constraint, not a permanent one — it's the reason Section 3 was rescoped to cover all 5 people from the start rather than just Gabriel and Aaron.

In the meantime, interns can:

  • Read docs/infrastructure.md and this runbook to learn the deploy flow
  • Watch a deploy happen (screen-share or logs) with Gabriel or Aaron
  • Practice against a local dev environment (not staging/production) if one exists

Target-state staging deploy (once Section 3 ships individual identities)

  1. Authenticate as yourself: aws sso login --profile musehub-staging-operator (exact command depends on how Identity Center is configured — placeholder until Section 3 is built).
  2. From ~/ecosystem/musehub: bash deploy/push.sh staging.
  3. Watch the output — it builds the image, pushes to ECR via crane, SSMs the deploy script to the instance, runs migrations, waits for the new slot's /healthz to pass, then flips nginx.
  4. Verify: curl -s https://staging.musehub.ai/healthz and spot-check the site in a browser.
  5. If anything looks wrong, do not try to fix it solo — see Escalation below.

Production deploy (Gabriel and Aaron only)

Production was rebuilt from scratch in a dedicated AWS account (i-043aaed71bef11903) with a correctly attached IAM instance profile, and is live at musehub.ai as of 2026-08-25. The instance-profile blocker described in earlier drafts of this runbook (i-0855d6efe7fa1a49d, the old shared Nonproduction-hosted "prod") no longer applies — that instance doesn't exist anymore.

deploy/push.sh/deploy/deploy.sh are still hardcoded to the old instance and shared musehub-infra credential and need updating to target the new Production account (tracked separately — see production-readiness Section 10-11, #159). Until that ships, deploys to the new production instance are done manually via the same SSM-based flow used to bootstrap it.

Per Section 11's still-open items: there is currently no approval gate, no promotion-of-tested-artifact enforcement, and no automatic rollback. Until those exist, manually confirm staging is healthy and has been running the candidate build without issues before deploying the same build to production.

Escalation

  • Staging is down (522/Bad Gateway): docs/infrastructure.md's "Recovering a down staging instance" section has the exact SSM recovery command. Interns: don't attempt this solo yet — pair with Gabriel or Aaron the first few times.
  • Unsure whether something is safe to do: ask first. This applies especially to anything destructive (--force, --hard, deleting resources, rotating secrets, touching DNS/IAM/security groups) — none of that is deploy-related and none of it should be attempted without Gabriel or Aaron's explicit go-ahead, regardless of who's doing it.

What no one should do without Gabriel or Aaron's explicit sign-off

  • Any production action (deploy, rollback, migration, secret rotation)
  • Any AWS console change to IAM, security groups, DNS, or billing
  • Any destructive command against staging (dropping data, force-resetting the instance)
  • Distributing AWS credentials to anyone not already listed in the access-tier table above
File History 2 commits
sha256:be1b2fdf6ab1fc652b3b35a945d0277a36fbf5d96e63692020d36cd7ebe03534 Merge 'docs/security-monitoring-verified' into 'dev' — prop… Human 2 days ago
sha256:632c4ee4285eb160a7b4ca316b8769f1df111cbfc89999579c779d13365a5a89 Merge 'feat/opengraph-repo-cards' into 'dev' — proposal: mu… Human 22 days ago