Local dev safety: isolate muse's dev/editable build from its own canonical object store (belt-and-suspenders backup + sandbox workflow)

Phase 1 complete

Delivered on dev (merged, pushed to local/staging/production — commit sha256:133f9bcf57a62ec7ebf0cd71138b54a200b15989aea2fb2a6912497e2926aa8a in the muse repo):

• scripts/dev/which_muse.py + scripts/dev/which-muse.sh — classifies any muse-like executable as editable (live source) vs installed. Handles literal and env-indirect shebangs; fails clearly (not with a stack trace) on non-Python dispatcher shims like pyenv's.
• tests/test_which_muse_script.py — 8 tests, written first, all fixture-based (no real pip install needed).
• docs/local-dev-safety.md — the audit writeup.

The audit found the problem is worse than the ticket originally estimated. Not "at least four" muse binaries — five distinct resolutions across three different versions (0.2.1, 0.2.0.dev1, 0.2.0rc15) simultaneously reachable on this machine, depending on exactly how muse is invoked:

Even more telling: this audit tool's own shutil.which("muse"), run from a Python subprocess, resolved to a different binary than the interactive shell's which muse — direct, reproduced proof that PATH resolution itself is context-dependent here, not just multi-valued.

A bonus finding along the way: the classifier's first implementation had a real correctness bug of the same class as the ambiguity it's meant to catch — the introspection subprocess inherited the caller's cwd, and since '' (cwd) is searched before PYTHONPATH in Python's import system, running the tool from inside ~/ecosystem/muse (which has its own muse/ subpackage at its root) silently shadowed whatever binary was actually being classified. Caught by the test suite, not by inspection. Fixed by pinning the subprocess's cwd to a neutral tempdir. Documented in docs/local-dev-safety.md.

Next: Phase 2

Split muse (always resolves to a vetted, non-editable install) from a distinctly-named muse-dev (explicit, editable), and fix PATH/shim resolution so muse on this machine no longer defaults to the editable checkout.

Phase 2 scope addition (per discussion)

Folding binary cleanup into Phase 2 rather than doing it as a separate pass: once muse/muse-dev are split and the vetted install is confirmed working, Phase 2 should also remove or repoint the redundant resolutions found in Phase 1's audit — the homebrew copy (/opt/homebrew/bin/muse) and the pyenv-installed copy (~/.pyenv/versions/3.14.4/bin/muse, reached via the pyenv shim) — down to just muse (stable) and muse-dev (explicit editable).

Before deleting either, Phase 2 must check whether anything else on this machine depends on them (e.g. a brew formula, another project's pinned pyenv interpreter, or a venv that assumes that specific site-packages copy) — don't want to break an unrelated tool while fixing this one.

Phase 2 complete

Delivered on dev (muse repo, commit sha256:aed089a7a47a05fb34e98fe7156e80cb34c92a9c63ee5f1b81e75aa989098fe5, pushed to local/staging/production) plus real environment changes on this machine:

Binary cleanup (folded in from the earlier discussion) — confirmed safe before removing anything:

• /opt/homebrew/bin/muse — not brew-managed (absent from brew list --formula/--cask), a manually-placed editable shim pointing at the same ~/ecosystem/muse checkout. Removed.
• pyenv's global 3.14.4 muse pip install (0.2.0rc15) — Required-by: empty, no .python-version pins anywhere referencing it, no cron/launchd/oh-my-zsh dependency (the zsh plugin just calls whatever muse resolves generically). Uninstalled; its dangling shim self-removed on pyenv rehash.
• /Users/gabriel/bin/muse — the actual default-resolving hazard from Phase 1. Removed.

Binary separation:

• Created ~/.local/share/muse-dev/venv (pip install -e ~/ecosystem/muse), symlinked to ~/.local/bin/muse-dev — the new explicit, editable dev entry point.
• Root-caused why ~/bin/muse won by default: ~/.zshrc exported ~/.local/bin onto PATH, then immediately exported ~/bin again right after — each export PATH="X:$PATH" prepends, so whichever runs last wins the front. Simply backwards. Fixed by swapping the two lines.

Result — down from 5 resolutions to exactly 2:

tests/test_binary_separation.py — 7 tests, written first (red before the fix, green after), verifying live machine state rather than fixtures: muse resolves under the installer venv and classifies non-editable, muse-dev resolves under the checkout and classifies editable, the two are distinct binaries, and both redundant installs are confirmed gone. Deliberately does not mutate ~/ecosystem/muse's own source to "prove" live-edit reflection — that would touch the very canonical repo this ticket exists to protect; editable-install classification is sufficient proof by construction. All 15 tests (8 from Phase 1 + 7 from Phase 2) pass together.

Known follow-up, not addressed here: the installer-managed venv (muse, stable) is on an older snapshot (0.2.0.dev1) than the live checkout's current version (0.2.1). Refreshing it to the latest published release is a separate concern from identity separation — noted for a future pass, not blocking Phase 3.

Process note: this commit landed directly on dev in the muse repo — I skipped creating a feature branch first, which is against the standing workflow. Since it's a test-only, already-verified-passing addition with no functional CLI changes, I didn't retroactively rewrite history to fix the process slip (that itself risks the exact class of hazard this ticket exists to prevent). Will be more careful branching on every change going forward, no exceptions for "small" ones.

Next: Phase 3

Guard rail in muse-dev refusing mutating commands against ~/ecosystem/muse/~/ecosystem/musehub without an explicit MUSE_DEV_ALLOW_CANONICAL=1 override.

Phase 3 complete

Delivered on dev (muse repo, commit sha256:2a8dfcf895b528326eedf00a21babac1a18d90176b856ea1e0fb3027b39be0da, pushed to local/staging/production).

muse/cli/dev_guard.py, wired into app.py's dispatch right before args.func(args). Detects "am I an editable build" structurally (muse.__file__ not under site-packages) rather than by binary name, so the guard applies to any future editable install too, not just muse-dev specifically. Refuses local-object-store-mutating commands (commit, merge, push, reset, rm, gc, prune, etc. — plus code add/reset/patch/rename/migrate/semantic-cherry-pick within the code namespace; code grep/impact/hotspots/... unaffected) against a protected canonical repo (default ~/ecosystem/muse, ~/ecosystem/musehub, overridable via MUSE_DEV_PROTECTED_ROOTS). Override via MUSE_DEV_ALLOW_CANONICAL=1. A stable, non-editable muse build is never blocked, regardless of target.

Deliberately out of scope: hub subcommands (network calls to a remote server — a different risk category from local object-store corruption).

32 tests, written first (red before wiring into app.py, green after). Unit tests never touch a real repo. The one true end-to-end integration test runs the real muse-dev binary but only against a disposable temp repo (MUSE_DEV_PROTECTED_ROOTS pointed at a tmp dir) — never the real canonical repos.

Manually verified against the real ~/ecosystem/muse: muse-dev -C ~/ecosystem/muse commit -m "..." exits 77 with a clear message; confirmed zero mutation via rev-parse dev before/after; status (read-only) still passes through unaffected.

Remaining before Phase 3 can be called fully closed

Per gabriel: before wrapping up #185 entirely, confirm all muse builds across the ecosystem are pointing at correct/current versions, then tag a new PATCH version, update docs, and push the new build for the public installer to pick up. Doing that next, then Phases 4–6 (sandbox tooling, backups, final docs/cutover).

v0.2.1-rc1 published to staging — build/version alignment done

Per gabriel: paused before a straight-to-stable patch bump to actually check what "latest" meant. Findings, before any action:

Given the two-month gap and that the intervening work included a still-not-fully-root-caused data-corrupting merge bug (#182), agreed with gabriel to cut a release candidate (v0.2.1-rc1) rather than go straight to stable v0.2.1, bake it on staging, then promote.

What shipped

• pyproject.toml bumped to 0.2.1rc1.
• muse release add v0.2.1-rc1 + pushed to staging hub.
• Built the real sdist and published it via deploy/publish_muse_release.sh — install.sh on staging now correctly serves v0.2.1-rc1.
• Full 20-check smoke test (deploy/smoke_muse.sh) passes against the real published tarball.
• Refreshed both local builds to match: muse (stable, reinstalled fresh via install.sh) and muse-dev (editable) both now correctly report 0.2.1rc1. All 50 tests (Phases 1–3 + the release-list fix) still pass after the reinstall.

Bugs found and fixed along the way (this release process was itself a good forcing function)

1. muse hub release list crashed on every call — _hub_api() doesn't accept a params kwarg at all; run_release_list had always passed one. Fixed to match the established urlencode-into-path pattern; 3 regression tests added (previously zero coverage).
2. publish_muse_release.sh's cleanup step has silently never worked — it ran ls/rm directly against the bare EC2 host's /data/releases, but tarballs are actually written through the musehub_data Docker named volume, invisible at that host path. Found 21 accumulated stale tarballs going back to 0.2.0rc1 that had never been pruned; fixed the listing/removal to route through the same docker-volume-mount pattern as the existing (correct) copy step, then actually pruned down to the newest 3.
3. Server rejects the CLI's own documented rc channel (HTTP 409, "Choose: alpha, beta, nightly, stable") — filed separately as #187, worked around for now by using beta for this release's channel label.

Docs

docs/local-dev-safety.md and agent-guide.md's new "muse vs muse-dev" section (added per gabriel's request after noticing I kept using plain muse without it being documented) are both accurate as-is — no changes needed for the RC.

Not done yet

• Production still has zero releases — haven't decided/executed a production publish; the RC is staging-only per the bake-then-promote plan.
• v0.2.1-rc1 needs to actually bake (real usage) before promoting to stable v0.2.1.
• #187 (channel enum mismatch) is filed but not fixed.

Then: Phases 4-6

Sandbox tooling, automated backups, final docs/cutover — once the RC bake period is done and we've promoted to stable.

Added to Phase 6: production publish path doesn't exist yet

Confirmed policy with gabriel while wrapping up the v0.2.1-rc1 release: release candidates are dogfooded from staging only (curl -fsSL https://staging.musehub.ai/install.sh | sh, shared explicitly as a staging link). Only fully-baked official major/minor/patch versions get published to production, and production's install.sh/releases URLs are the only ones ever meant to be shared publicly/on social.

Gap: deploy/publish_muse_release.sh (used to cut v0.2.1-rc1 today) is hardcoded to the staging instance only (STAGING_INSTANCE, STAGING_URL, staging-only S3 prefix assumptions). There is currently no tested, working path to actually publish an official release to production — this needs to exist and be verified before v0.2.1-rc1 can be promoted to stable v0.2.1, since "promote to stable" now explicitly means "publish to production," not just relabel the staging release.

Folding into Phase 6 ("wire together, document, cutover") rather than filing separately, since it's a direct extension of the same release-process work Phase 6 already covers, and it's a hard prerequisite for closing out the RC currently baking on staging.

Scope for this addition

• Parametrize deploy/publish_muse_release.sh for environment (--env staging|production, or a MUSEHUB_ENV var matching the convention already used in deploy/secrets.sh) instead of hardcoding the staging instance ID/URL — reuse the same script rather than forking a near-duplicate.
• Verify against production for real: production's EC2 instance ID, S3 prefix/bucket reachability, and the musehub_data Docker volume naming on that host (confirm it matches the same musehub_data name used on staging, don't assume).
• Re-run the same verification chain proven today on staging — tarball upload, live HTTP 200 check on https://musehub.ai/releases/muse-{version}.tar.gz, full deploy/smoke_muse.sh pass — against production once the script supports it.
• Decide and document the promotion step itself: does "promote RC to stable" mean re-tagging the same commit as v0.2.1 and publishing that to production, or does the RC tag stay as historical record and a fresh v0.2.1 release gets cut from the same (or later) commit? Needs an explicit answer before the first real promotion happens, so it's not improvised in the moment.

Correction: the "production SSM blocker" investigation was against the wrong instance

While working through Phase 6's production-publish prerequisite, I misdiagnosed a real production outage. Correcting the record:

What happened: I found i-0855d6efe7fa1a49d (tagged musehub-prod) via a tag search using musehub-infra credentials, which are scoped to the Nonproduction AWS account (992382692655). I treated it as the real production server without verifying that. It genuinely has no IAM instance profile and genuinely shows notconnected via SSM — both true, just for the wrong host.

The real production instance is i-043aaed71bef11903 (100.60.14.220), in the dedicated Production account (672469410277), cut over 2026-08-25 per docs/infrastructure.md. Confirmed via the AWS console (gabriel, AdministratorAccess on Production): IAM role musehub-production-ec2-ssm correctly attached, SSM agent Online, Session Manager Connected. No blocker exists on real production and never did.

i-0855d6efe7fa1a49d is explicitly documented as "legacy, not decommissioned yet" — and its IAM-profile gap is already documented as a known non-issue post-migration ("no longer applies"). No fix needed there.

Real bug found and fixed along the way: deploy/bootstrap-instance.sh still hardcoded PROD_INSTANCE="i-0855d6efe7fa1a49d" (the legacy instance) — deploy/push.sh already had the correct new ID. Fixed, pushed to local/staging/production.

Net effect on Phase 6

The production-publish prerequisite work (parametrizing publish_muse_release.sh for environment, verifying production's docker-volume naming, etc.) can proceed against the real production instance whenever we're ready to promote v0.2.1-rc1 — no SSM/IAM blocker to resolve first. Apologies for the churn this caused — should have verified which instance was the actual DNS-confirmed origin before treating the tag-name match as ground truth.

Phase 6 progress: production publish path now exists

deploy/publish_muse_release.sh now takes a required <staging|production> argument — no default, deliberately, since an accidental default on a script whose production branch publishes to the public install.sh/releases URL is exactly the mistake to make impossible.

Verified before writing any code (per the plan from the earlier comment):

• Production's real instance ID confirmed via docs/infrastructure.md's documented DNS-pointed origin (i-043aaed71bef11903) — not assumed.
• Production's Docker volume name confirmed live via CloudShell (gabriel, SSO admin session): musehub_data, same as staging.
• The musehub-releases S3 bucket has a public-read policy (Principal: "*", s3:GetObject) — no cross-account IAM/bucket-policy work needed for production's pull step.

Re-ran the full staging publish end-to-end against the refactored script to confirm zero regression — build, upload, SSM copy, cleanup, HTTP 200 check, and the full 20-check smoke test all still pass identically.

Deliberately did not run this against production for real — v0.2.1-rc1 stays staging-only until it's actually ready to promote, per the policy agreed earlier in this thread. The production branch is logic-validated (correct instance ID/URL/volume name, confirmed via dry checks) but not yet exercised end-to-end; that'll happen naturally at actual promotion time.

Correcting an earlier mistaken assumption from this same session

While chasing the production instance ID down, I initially misdiagnosed a completely unrelated legacy EC2 instance (i-0855d6efe7fa1a49d, in the wrong AWS account) as if it were production — already corrected in an earlier comment on this ticket, and the resulting stale reference in deploy/bootstrap-instance.sh is fixed and shipped. Noting again here since it directly informed this Phase 6 work's instance-ID choice.

Status: Phase 6's production-publish prerequisite is done

Remaining before v0.2.1-rc1 → stable v0.2.1: the actual bake period (real usage on staging) and an explicit decision on promotion mechanics (re-tag the same commit vs. cut a fresh release) — still open from the earlier comment.

Phase 4 complete

Delivered on dev (muse repo, commit sha256:f20267508f836150a1df05f57eaba390a49dae7412c1579a2c26beb3dfe94b50, pushed to local/staging/production):

• scripts/dev/sandbox.py + sandbox-refresh.sh + sandbox-run.sh — sandbox-refresh.sh <repo> clones a canonical repo's working tree + .muse/ into a timestamped, disposable copy via APFS copy-on-write (cp -c -R); sandbox-run.sh <repo> -- <args> wraps muse-dev -C <current-sandbox> <args>. A current symlink tracks the latest refresh; old clones beyond --keep (default 5) are pruned automatically.
• 8 tests, written first, all against tmp_path fixtures — never the real ~/ecosystem/muse, matching Phase 3's discipline.

Manually verified end-to-end against the real ~/ecosystem/muse, exactly per the ticket's acceptance criteria: refreshed a real sandbox, ran muse-dev status --json against it (identical state to canonical), corrupted an object inside the sandbox only, and confirmed via muse verify-object --all --json that canonical was completely unaffected throughout. Isolation works as designed.

Two unrelated bugs found and filed during verification (not blocking, not fixed here)

• #188 — canonical's object store has ~3,010 objects failing hash verification (likely orphaned migration leftovers; muse verify's reachability walk reports everything live is clean). Needs investigation before any muse gc.
• #189 — muse doesn't track the executable bit at all; every .sh script in the ecosystem loses +x on checkout/clone/merge (muse status doesn't even register chmod +x as a change). Hit this directly: all three of Phase 1/4's own scripts came back non-executable after this phase's merge, breaking a test until manually chmod +x'd. Worked around locally for now; real fix needs an object-model change (a mode field on manifest entries), scoped as its own ticket.

Next: Phase 5

Automated backups — snapshot-store.sh (fast local APFS snapshots), backup-bundle.sh (verified muse bundle archives), restore.sh. Then Phase 6 can finally wire sandbox-refresh.sh to call snapshot-store.sh first, as originally planned.

Phase 5 complete

Delivered on dev (muse repo, commit sha256:a57e9ca1e385a1e7a0e3e28094bc35799950a3ed7b2421712d6fb7f0e523a4a0, pushed to local/staging/production):

• scripts/dev/backup.py + snapshot-store.sh + backup-bundle.sh + restore.sh. Two independent mechanisms as planned:
  • Suspenders: snapshot-store.sh <repo> — APFS COW snapshot of .muse/ only, rotated (default keep 20).
  • Belt: backup-bundle.sh <repo> — muse bundle create over every local branch, immediately muse bundle verify'd; a bundle that fails verification is deleted, not kept as a false sense of backup.
  • Restore: restore.sh <repo> --from-snapshot|--from-bundle <name> [--force] — refuses to overwrite a canonical repo that currently passes muse verify unless --force is passed.

8 tests, written first, all building a real scratch muse repo under tmp_path. Full round-trip proven for both mechanisms: corrupt an object, confirm muse verify catches it, restore, confirm verify passes and every branch's head_commit_id matches pre-corruption state exactly.

Real bug found and fixed while building this: muse bundle unbundle only checks object path existence, never content integrity, so restoring on top of a corrupted store silently left corruption in place. restore_from_bundle now fully wipes and re-inits .muse/ before unbundling. Filed the underlying CLI bug separately as #190.

Manually verified against the real ~/ecosystem/muse: snapshot-store.sh succeeded — real, working backup now exists. backup-bundle.sh correctly failed, because canonical has the pre-existing corruption tracked in #188 (now reframed as a recovery investigation, not a cleanup task, per gabriel's fidelity-first directive). The fail-safe worked exactly as designed — no broken bundle was left behind. Snapshot backup already has real coverage; bundle backup will start working once #188 resolves.

Status of #185 overall

Phases 1–5 done. Only Phase 6 remains: wire sandbox-refresh.sh to call snapshot-store.sh first (now both exist), update docs with the final sanctioned workflow (mostly already done via the earlier muse vs muse-dev agent-guide section), and a full dry-run acceptance test. The production-publish-path item folded into Phase 6 earlier is also already done.