v0.2.0rc11 staging deploy — muse CLI, musehub, R2 nuke and re-push

Overview

Full rc11 deploy to staging. Two parallel tracks: muse CLI tarball publish and musehub Docker image. After the app is healthy on rc11, we nuke the Cloudflare R2 bucket (tens of thousands of stale objects from bench seeds and old test repos), re-create only the canonical repos, re-push from local, and run the mpack index backfill.

Ordering rationale: musehub rc11 must land before the R2 nuke. The old container code may have object-read paths that depend on storage layout assumptions that changed in rc11. Getting the new code live first means the re-push and backfill happen against the correct serving logic.

Phase 1 — Deploy muse CLI v0.2.0rc11

Goal: publish the rc11 tarball to staging so muse on staging resolves to 0.2.0rc11.

Pre-flight: confirm local muse is at rc11.

python3 -c "import re, pathlib; t = pathlib.Path('~/ecosystem/muse/pyproject.toml').read_text(); m = re.search(r'version.*=.*\"([^\"]+)\"', t); print(m.group(1))"
# expected: 0.2.0rc11

Publish:

bash ~/ecosystem/musehub/deploy/publish_muse_release.sh

What it does: builds sdist from ~/ecosystem/muse, uploads to s3://musehub-releases/muse-0.2.0rc11.tar.gz, SSMs to the staging instance to copy it into /data/releases/, verifies HTTP 200 at https://staging.musehub.ai/releases/muse-0.2.0rc11.tar.gz, prunes older tarballs (keeps 3).

Smoke test (18 checks against the published tarball):

bash ~/ecosystem/musehub/deploy/smoke_muse.sh

Installs the tarball in a throwaway venv, runs: version, init, add, commit, status, log, read, ls-files, branch (create/list/delete), diff, checkout, tag, verify. All 18 must pass. Exit 0 = good, exit 1 = failure in checks, exit 2 = setup error.

Done when: smoke_muse.sh exits 0.

Phase 2 — Deploy musehub rc11 (Docker blue-green)

Goal: get the rc11 musehub image running on staging, Alembic migrations applied.

bash ~/ecosystem/musehub/deploy/push.sh staging

What it does: builds a linux/amd64 Docker image, tags it with <commit>-<timestamp>, pushes to ECR (musehub/musehub), SSMs to the staging instance (i-07547cd20bee2dea5) to pull the image and blue-green swap. Alembic upgrade head runs automatically on container start — migrations 0058 through 0070 must apply cleanly.

Verify:

curl -sk https://staging.musehub.ai/healthz
# expected: {"status": "ok", ...}

Key changes landing in rc11 that affect staging data:

• Migration 0068: adds storage_uri column to musehub_commits and musehub_snapshots
• Migration 0069: proposal comment provenance
• Migration 0070: rename merge_strategy aliases to canonical names (state_overlay → overlay, etc.)
• Architecture: commits and snapshots are now DB-canonical only — blobs exclusively in R2 via mpack byte-range indexing (musehub_mpack_index)
• compute_snapshot_id uses typed-object formula (canonical sha256:-prefixed IDs)

Done when: healthz returns 200 and muse -C ~/ecosystem/muse ls-remote staging --json succeeds without 401.

Phase 3 — Nuke R2 and delete stale repos

Goal: wipe tens of thousands of stale R2 objects (bench seeds, test pushes, old wire protocol objects) by deleting all repos, then clearing R2.

3a — Delete all staging repos

These repos are safe to delete. The canonical repos (muse, musehub, agentception, muse-zsh, identity) will be re-created and re-pushed in Phase 4.

# Repos to delete (adjust list after reviewing hub repo list):
# bench-seed-xs, bench-seed-s, bench-seed-m, bench-seed-l, bench-seed-xl
# wire-hello, diag-pull*, 5i18xxyrgunh, 9clq6y68x4pr, ckxsge2f8yco
# muse, musehub, muse-zsh, identity (will be re-created)
muse -C ~/ecosystem/musehub hub repo list --hub https://staging.musehub.ai --json   | python3 -c "import sys,json; [print(r['slug']) for r in json.load(sys.stdin)['repos']]"

# Delete each repo:
muse -C ~/ecosystem/musehub hub repo delete --hub https://staging.musehub.ai --json
# (run from within each repo's context, or use the slug-based API)

Deleting repos cascades in the DB: removes musehub_objects, musehub_object_refs, musehub_commits, musehub_snapshots, musehub_branches, and all intel/index rows for those repos.

3b — Nuke Cloudflare R2 bucket

With all repos deleted, every R2 object is now orphaned. Wipe the bucket:

# Via AWS CLI (R2 exposes an S3-compatible endpoint):
aws s3 rm s3://muse-objects/ --recursive --profile musehub-infra
# or via the Cloudflare dashboard: R2 → muse-objects → Empty bucket

Verify the bucket is empty:

aws s3 ls s3://muse-objects/ --recursive --profile musehub-infra | wc -l
# expected: 0

Done when: bucket is empty and DB has no repo rows.

Phase 4 — Re-create canonical repos and push

Goal: create fresh repo records on staging and push local dev + main from each canonical repo.

# Create repos (run from each repo directory):
muse -C ~/ecosystem/muse     hub repo create --hub https://staging.musehub.ai --name muse      --visibility public --json
muse -C ~/ecosystem/musehub  hub repo create --hub https://staging.musehub.ai --name musehub   --visibility public --json
muse -C ~/ecosystem/agentception hub repo create --hub https://staging.musehub.ai --name agentception --visibility public --json
muse -C ~/ecosystem/muse-zsh hub repo create --hub https://staging.musehub.ai --name muse-zsh  --visibility public --json

# Verify remotes are configured:
muse -C ~/ecosystem/muse     remote --json
muse -C ~/ecosystem/musehub  remote --json

# Push main + dev for each repo:
muse -C ~/ecosystem/muse     push staging main
muse -C ~/ecosystem/muse     push staging dev
muse -C ~/ecosystem/musehub  push staging main
muse -C ~/ecosystem/musehub  push staging dev
muse -C ~/ecosystem/agentception push staging main
muse -C ~/ecosystem/agentception push staging dev
muse -C ~/ecosystem/muse-zsh push staging main
muse -C ~/ecosystem/muse-zsh push staging dev

Each push will create fresh mpacks in R2, populate musehub_mpack_index rows, and create musehub_commits/musehub_snapshots DB rows for all history.

Done when: all pushes exit 0 and muse ls-remote staging --json shows main and dev for each repo.

Phase 5 — Mpack index backfill

Goal: ensure all mpack objects have their byte-range entries in musehub_mpack_index so wire fetch can serve individual objects efficiently.

docker exec musehub-blue python3 /app/deploy/enqueue_mpack_index_backfill.py --dry-run
# Review: how many unindexed mpacks?

docker exec musehub-blue python3 /app/deploy/enqueue_mpack_index_backfill.py
# Processes one mpack at a time, polls until each job completes before moving to next

If the dry-run shows 0 unindexed mpacks (likely after a clean re-push with rc11), this phase is a no-op.

Object store backfill (commits/snapshots — run only if storage_uri IS NULL rows exist):

docker exec musehub-blue python3 /app/deploy/backfill_object_store.py --dry-run
# If > 0 rows need backfilling, run via the loop in batches of 50:
bash ~/ecosystem/musehub/deploy/backfill_loop.sh --batches 10 --limit 50

After a clean re-push, storage_uri is vestigial in rc11 (commits/snapshots are DB-canonical, not S3-served). This backfill may be a no-op — confirm with dry-run first.

Done when: enqueue_mpack_index_backfill.py --dry-run reports 0 unindexed mpacks.

Phase 6 — Final smoke and sign-off

# Muse CLI smoke (runs against staging tarball):
bash ~/ecosystem/musehub/deploy/smoke_muse.sh --url https://staging.musehub.ai

# Wire push/pull round-trip (confirm objects flow end-to-end):
muse -C /tmp/smoke-clone clone https://staging.musehub.ai/gabriel/muse --json
muse -C /tmp/smoke-clone log --json -n 3

# Check staging healthz one final time:
curl -sk https://staging.musehub.ai/healthz

Done when: smoke exits 0, clone succeeds, healthz is green.

Rollback

Notes

• Staging instance: i-07547cd20bee2dea5 (us-east-1)
• Active slot check: cat /opt/musehub/.active-slot on instance
• SSM must be reachable before Phase 2; if stuck as Pending, reboot the instance and wait for SSM ping
• R2 bucket name: muse-objects (S3-compatible endpoint via Cloudflare)
• identity repo: private — recreate after the public repos are verified, confirm key is still registered on staging (muse auth whoami --hub https://staging.musehub.ai)