# MWP-9 — Spot check #58: deep verification of the wire protocol MVP goal-of-done > **Master tracker:** [muse#58](https://staging.musehub.ai/gabriel/muse/issues/58) — Muse Wire Protocol MVP hardening. MWP-1..MWP-8 are all closed; this is the verification pass before aaronrene onboards. > **Historical artifact:** the pre-phase-refactor version of this doc was published as a mist: https://staging.musehub.ai/gabriel/mists/FVTnZi7e7Q3b (kept live, superseded by this issue as the tracking source of truth). > **Repo target:** cross-repo (`muse` client + `musehub` server); scratch repos only (`gabriel/mwp-qa-*` on staging) — never `gabriel/muse` or `gabriel/musehub` themselves. > **IMPORTANT — this project uses Muse (not git) for version control.** > - Never run `git`, `gh`, or any git subcommand. Repo paths: `~/ecosystem/muse`, `~/ecosystem/musehub`, `~/ecosystem/agentception`. > - Commit with full provenance: `muse commit -m "..." --agent-id claude-code --model-id claude-sonnet-4-6 --sign`. > - Branch first; never commit to `dev`/`main`. **Never merge to `dev`/`main` without gabriel's explicit permission.** > - No `--force` flag, no `reset --hard`, no `checkout --ours/--theirs` — even inside a disposable scratch repo — without gabriel's explicit go-ahead in the conversation (see Safety notes). --- ## Background [Issue #58](https://staging.musehub.ai/gabriel/muse/issues/58) — "Muse Wire Protocol MVP hardening" — lists 8 sub-tickets (MWP-1..MWP-8), all now closed with green test suites: | Sub-ticket | Repo | Fixes | Status | |---|---|---|---| | MWP-1 | musehub#106 | RC-1 (gen-0 fallback corrupts fetch walk) | ✅ closed | | MWP-2 | musehub#107 | RC-2 (`_walk_commit_delta` no fallback) | ✅ closed | | MWP-3 | musehub#108 | RC-3 (job enqueue drops back-to-back pushes) | ✅ closed | | MWP-4 | musehub#109 | RC-4 (prebuild/index ordering race) | ✅ closed | | MWP-5 | muse#59 | RC-5 (client doesn't retry on 503) | ✅ closed | | MWP-6 | musehub+muse | RC-6 (wire suite disabled) | ✅ closed | | MWP-7 | muse#62 | RC-7 (#55 sibling-ref negotiation, #32 dry-run) | ✅ closed | | MWP-8 | musehub#111 | RC-8 (repair/force-push don't bust fetch cache) | ✅ closed | Every sub-ticket shipped with unit/integration tests that pass in isolation (commit-by-commit, phase-by-phase). **What has not been independently proven is the compound claim in #58's "Goal — definition of MVP done":** that a real developer, running the real CLI against real staging, in the order they'd actually use it, gets correct behavior every time — including at the seams between fixes (e.g. does MWP-8's cache invalidation interact correctly with MWP-3/4's job-ordering fix? does MWP-5's retry loop actually fire against a *real* 503 from staging, not a mocked one?). This is the gap this ticket closes. **We are not re-reviewing whether each MWP's unit tests are well-written — they are.** We are asking: *does the system, as a whole, running as a black box against staging, satisfy the five numbered claims in #58's goal section?* Only after this is true do we bring in aaronrene. ### Goal — definition of MVP done (verbatim from #58) > aaronrene (or any dev) can, against staging, reliably: > > 1. `muse clone ` — get a complete, correct working tree every time, > including immediately after a push (with bounded client retry on 503). > 2. `muse push` — upload only the true delta; never re-upload objects > already on the remote via any ref; survive back-to-back pushes. > 3. `muse fetch` / `muse pull` — receive exactly the new commits/blobs and > converge. > 4. Do all of the above repeatedly with no stale clones, no phantom large > uploads, no manual cache-busting. > 5. The wire regression suite is **re-enabled and green** as the proof. --- ## Goal — definition of done for this ticket Each of the 5 claims above is independently exercised against **real staging** (not localhost, not in-process fixtures, not mocks) via a dedicated `/tmp` scratch repo, and the 5 cross-cutting seams between MWP fixes are each explicitly checked or explicitly deferred with a written reason. No claim is marked done on the strength of a sub-ticket's own unit tests alone — every claim here requires fresh, live evidence. --- ## Design — why this phase order Phases are ordered cheapest/most-foundational first, then increasing interaction complexity, mirroring the workspace's load-bearing phase convention: - **Phase 0 first, always** — the wire suite (Claim 5) is already the most rigorously proven claim, but it costs seconds to re-run and must be green *before* we build any scratch-repo evidence on top of it. If Phase 0 is red, every later phase's result is suspect. - **Phases 1–3 are single-claim, single-command-family checks** (clone, then push, then fetch/pull) — each is a superset of what its own MWP tickets proved, but against live staging instead of localhost/mocks. Ordered clone → push → fetch/pull because clone is the simplest black-box surface (no delta negotiation) and push/fetch build on the same scratch repos clone establishes. - **Phase 4 is the composition check** (Claim 4) — deliberately last among the per-claim phases because it *depends on* Phases 1–3 already being individually green. Running the full repeated loop before the individual claims are proven would conflate "which claim failed" if something breaks. - **Phase 5 (cross-cutting seams) comes after all 5 claims are individually green** — these are interaction risks between MWPs that only make sense to chase once each MWP's *own* surface is confirmed working in isolation. - **Phase 6 (cleanup + sign-off) is last** — no scratch repos are deleted until every phase's findings are written back into this file, so a re-run/audit trail survives if aaronrene hits something during onboarding. **Never skip ahead.** Each phase must be fully green (or have its gaps explicitly written up as deferred) before the next phase begins. --- ## Phases > **Execution rule:** work one phase at a time; check in with gabriel after > each phase before starting the next — comprehensive findings over speed. > Test IDs (`SC_00`, `SC_01a`, …) appear in this plan; record results inline > under each recipe as it's executed (pass/fail, staging repo used, and any > deviation from the recipe as written). ### Phase 0 — Baseline: wire suite green, right now, post-merge - [x] `SC_00` — re-run the full `pytest -m wire` suite in both repos, fresh, after the MWP-8 merge-to-main sequence, to confirm nothing in the final merge broke it. This is the cheapest possible check and gates every later phase. **✅ PASS** **Recipe:** ```bash cd ~/ecosystem/musehub python3 -m pytest -m wire -q --tb=short cd ~/ecosystem/muse python3 -m pytest -m wire -q --tb=short ``` **Already proven (context, not a substitute for re-running now):** 55 tests under `pytest -m wire` green (47 in-process + 8 slow subprocess E2E); `MWP6_00` gate enforces no re-introduction of flux-skips. **Result (`SC_00`, run fresh post-MWP-8-merge):** | Repo | Result | Duration | |---|---|---| | musehub | **56 passed**, 0 failed, 0 skipped | 137.85s | | muse | **21 passed**, 0 failed, 0 skipped | 17.07s | musehub's count rose from the previously-documented 55 to 56 — expected, not a regression: MWP-8 added `EX_02` (`@pytest.mark.wire`, the clone-after-repair E2E), which is now part of the wire-marked corpus. `test_mwp6_wire_suite_enabled.py` (the no-flux-skip gate) and `test_mwp8_cache_invalidation.py`'s wire-marked test both appear in the run. **Exit gate:** both suites green. If either is red, STOP — do not proceed to Phase 1 until it's understood whether the break is pre-existing or a genuine regression from the MWP-8 merge. **Gate satisfied — proceeding to Phase 1.** --- ### Phase 1 — Claim 1: `muse clone` is always complete and correct, including immediately after a push **Fixed by:** MWP-1 (gen-0 authority), MWP-2 (DAG-walk fallback), MWP-5 (bounded 503 retry), MWP-8 (cache never stale after repair/force-push). **Already proven (unit/integration level):** - MWP1_14 — push→clone→push→clone E2E gate (musehub, in-process). - MWP2_12 — fallback alone ships a complete clone even with repair patched out. - MWP5 real-urllib E2E against a live localhost server (retry loop fires on real 503). - EX_02 (MWP-8) — clone-after-repair E2E, `@pytest.mark.wire`. **Residual gap this phase closes:** - All of the above run against **localhost** or **in-process fixtures**, not **staging** (real network latency, real R2, real multi-second prebuild time). The 503→retry window on localhost may not exercise the same timing as staging's actual prebuild latency. - No test proves clone correctness *chained* across a push→clone→push→clone cycle that also crosses a *force-push* or *repair* in the middle — i.e. the MWP-1/2 fix and the MWP-8 fix have never been exercised in the same sequence. - No test proves clone works correctly for a **brand-new repo that has never been pushed to before** against staging specifically (first-ever clone, cold cache, real MISS→prebuild→retry→HIT path end to end). - [x] `SC_01a` — cold clone of a never-touched repo (first MISS, real prebuild latency). **✅ PASS (content correctness); ⚠️ see finding below — 503/retry path not exercised.** - [x] `SC_01b` — push then immediate clone (the exact MWP-1/RC-1 regression), against real staging timing. **✅ PASS (content correctness); ⚠️ same finding.** **Recipe (corrected — the original omitted the required repo-create step; `muse push` 404s against a repo that doesn't exist on the hub yet):** ```bash # SC_01a — cold clone of a never-touched repo (first MISS, real prebuild latency) cd /tmp && rm -rf mwp-qa-1a && mkdir mwp-qa-1a && cd mwp-qa-1a muse init --json echo "hello" > README.md && muse code add README.md muse commit -m "init" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse hub repo create --name mwp-qa-1a --no-init --hub https://staging.musehub.ai --json muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-1a muse push staging main --json # first push to a brand-new repo cd /tmp && rm -rf mwp-qa-1a-clone time muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1a-clone --json # expect: complete tree, README.md present, exit 0 — measure how long the # 503-retry loop actually waited (proves MWP-5's bounded poll against real # staging timing, not a mock) # SC_01b — push then immediate clone (the exact MWP-1/RC-1 regression) cd /tmp/mwp-qa-1a echo "v2" >> README.md && muse code add README.md muse commit -m "v2" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push staging main --json cd /tmp && rm -rf mwp-qa-1b-clone muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1b-clone --json diff /tmp/mwp-qa-1a/README.md /tmp/mwp-qa-1b-clone/README.md # must be identical, must contain "v2" ``` **Result — content correctness:** | Test | Result | Detail | |---|---|---| | `SC_01a` | ✅ PASS | Cold clone of brand-new `mwp-qa-1a`: `README.md` content identical byte-for-byte, `commits_received=1`, `blobs_written=1`, exit 0. Wall time 1.63s. | | `SC_01b` | ✅ PASS | Push `v2` then clone immediately (chained in one command, zero artificial delay): both commits present (`commits_received=2`), content identical including `v2` line, exit 0. Wall time 1.34s. | **Finding — the 503/retry path was never actually exercised:** Both clones returned **HTTP 200 on the first `POST /fetch/mpack` call** — no 503, no `Retry-After`, no retry loop observed in either run, even in `SC_01b` where the clone was issued in the same shell chain immediately after the push returned. This means the *content-correctness* half of Claim 1 is proven against real staging, but the *retry-mechanism* half (MWP-5's bounded poll actually firing against real staging prebuild latency, not a mock) was **not exercised** — the scratch repo is too small (1–2 commits, <2KB) for the prebuild job to lag behind the client's request at all, even with near-zero gap between push and clone. This is a real, not cosmetic, gap relative to the phase's stated goal ("measure how long the 503-retry loop actually waited"). MWP-5's own test suite already proves the retry loop fires correctly against a *mocked* 503 — what's still unproven is that it fires (and behaves correctly) against a *real* staging 503 under real prebuild latency. Closing this fully would require a repo large enough, or a push volume high enough, to reliably lag the prebuild job — deferred to Phase 4's repeated-loop test (`SC_04a`) or a dedicated large-payload follow-up, rather than blocking Phase 1's exit gate on it, since content correctness (the primary contract) is proven. **Exit gate:** `SC_01a` and `SC_01b` both pass against staging (content correctness) — **satisfied.** Timing observation recorded: retry loop was not exercised on this scratch repo's scale; carried forward as an explicit follow-up rather than re-run here, since Phase 1's content-correctness contract is the gating requirement and it is fully met. **Proceeding to Phase 2**, with the retry-path gap tracked for revisit in Phase 4 or as a dedicated large-payload check. --- ### Phase 2 — Claim 2: `muse push` uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes **Fixed by:** MWP-3 (job enqueue reflects latest tip), MWP-4 (prebuild/index ordering), MWP-7 (sibling-ref have-negotiation, dry-run parity). **Already proven:** - MWP3_10/11 — two-push full drain, N=10 stress partial drain (in-process job queue simulation). - MWP4_11/12 — E2E claim loop + N=10 stress, zero `FetchNotIndexedError`. - MWP-7 Phase 3 (VS) — real staging push, 17-commit dev + 1-commit main, `commits_sent=1, objects_sent=1`. **Residual gap this phase closes:** - MWP-3/4's stress tests simulate concurrent job claiming in-process — they do **not** prove that a human issuing two real `muse push` commands back to back (i.e. two separate CLI processes, two separate HTTP round-trips) against staging converges correctly. Real back-to-back means real race conditions in the async worker, not simulated ones. - No test proves the **cross-branch dedup** claim ("never re-upload objects already on the remote via any ref") for a *third* sibling branch beyond the two-branch case MWP-7 tested. If `feature-a`, `feature-b`, and `dev` all share a base and `feature-b` is pushed last, does it dedup against **both** siblings or just the most-recently-pushed one? - `--dry-run` parity was fixed for a single scenario (MWP-7 Phase 4/5) — needs re-verification against a *different* topology (e.g. a rebased branch) to rule out a narrow fix. - [x] `SC_02a` — literal back-to-back push (two real CLI invocations, no artificial delay), converges correctly. **✅ PASS.** - [x] `SC_02b` — three-sibling-branch dedup: pushing the third branch excludes blobs already on staging via either sibling. **✅ dedup itself PASS — but surfaced a separate, more severe bug while verifying it (see below). 🔴 BLOCKING FINDING.** - [x] `SC_02c` — dry-run parity holds on a rebased topology, not just the original MWP-7 scenario. **✅ PASS — run against localhost (musehub#113's fix now live, both fixes verified in place), after resuming per gabriel's local-first priority.** **Recipe (corrected — the original `SC_02a` snippet had a broken background-job/echo race that didn't test what it claimed; replaced with what was actually run: two real sequential commits + pushes, zero artificial delay between them):** ```bash # SC_02a — two real sequential CLI pushes, back-to-back, no artificial delay cd /tmp/mwp-qa-1a echo "a" >> README.md && muse code add README.md muse commit -m "a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push staging main --json # push A echo "b" >> README.md && muse code add README.md muse commit -m "b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push staging main --json # push B, immediately after A returns cd /tmp && rm -rf mwp-qa-2a-clone && muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2a-clone --json tail -3 mwp-qa-2a-clone/README.md # must show both "a" and "b" # SC_02b — three-sibling-branch dedup cd /tmp/mwp-qa-1a muse checkout -b feature-a && echo "fa" > fa.txt && muse code add fa.txt muse commit -m "feature a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push staging feature-a --json muse checkout main && muse checkout -b feature-b && echo "fb" > fb.txt && muse code add fb.txt muse commit -m "feature b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push staging feature-b --json | jq '.objects_sent' # should exclude blobs already on staging via feature-a's shared base # SC_02c — dry-run parity on a rebased topology muse rebase main --dry-run --json # compare counts to a real rebase + push ``` **Result — `SC_02a`:** ✅ PASS. 4 commits (`init`, `v2`, `a`, `b`) converged correctly on clone, content in exact order, no staleness despite zero-delay back-to-back pushes. **Result — `SC_02b`:** the dedup claim itself is **✅ confirmed correct, and more thoroughly than planned** — `feature-b`'s push sent exactly 1 object (`fb.txt`'s blob), correctly excluding README.md's blob across the entire 4-commit shared ancestor chain with both `main` and `feature-a`. RC-7/MWP-7's sibling-ref dedup holds up under a 3-branch test, not just the 2-branch case it originally shipped with. **🔴 But verifying that dedup by cloning `feature-b` surfaced a separate, more severe bug:** `muse clone --branch feature-b` returned `exit_code: 0` / `status: "cloned"` while its own log stated the working tree was only **partially restored** — `fb.txt` was completely missing from disk. `muse verify --json` on the resulting clone reported `all_ok: true` with zero failures, giving no signal anything was wrong. This directly contradicts Claim 1 of #58 ("get a complete, correct working tree every time"), not just Claim 2. Full brain-dump, reproduction steps, and open questions filed as its own investigation ticket rather than patched inline: **[musehub#113](https://staging.musehub.ai/gabriel/musehub/issues/113)**. No production code was touched while chasing this down. **Result — `SC_02c`:** ✅ PASS, run against localhost after musehub#113's two fixes landed. Topology: `main` (base commit), `feature` branches off and gets 2 commits, `main` then diverges with its own new commit, `feature` is rebased onto the new `main` tip (rewriting both of `feature`'s commit IDs). `muse rebase main --dry-run --json` correctly predicted `count: 2` before rebasing. After the real rebase, `muse push local2 feature --dry-run --force --json` predicted `commits_sent: 2, objects_sent: 0` (file content unchanged by rebase, already known to the remote) — the **real** forced push produced the exact same `commits_sent: 2, objects_sent: 0`, `blobs_written: 0` server-side. Exact parity, not just close. A fresh clone of the rebased `feature` branch confirmed correct content (`base.txt`, `f1.txt`, `f2.txt`, `main2.txt` all present) — this also exercises musehub#113's multi-tip manifest-union fix, since the repo has multiple branch tips. **Exit gate — Phase 2 now fully closed.** `SC_02a` ✅, `SC_02b`'s dedup claim ✅ (and the correctness bug it surfaced is now root-caused and fixed via musehub#113, not just filed), `SC_02c` ✅. All three green. --- ### Phase 3 — Claim 3: `muse fetch` / `muse pull` receive exactly the new commits/blobs and converge **Fixed by:** MWP-1/2 (correct generation walk), MWP-7 (sibling-ref negotiation applies to fetch's `have` set too). **Already proven:** - Fetch-side coverage is implied by the same MWP-1/2 test suite (fetch and clone share `_walk_commit_delta`). **Residual gap this phase closes:** - No sub-ticket wrote a **dedicated fetch/pull convergence test against staging** — all direct evidence is clone-shaped (`have=[]`). `fetch`/`pull` with a non-empty `have` set exercise a materially different code path (delta walk, not cache HIT) that has *not* been the subject of an explicit E2E in this campaign. - `pull` = fetch + merge/ff — the merge/ff correctness after a fetch has never been explicitly re-tested post-MWP-8 (does a `pull` after a force-pushed remote branch behave correctly, or does it silently diverge?). - [x] `SC_03a` — fetch convergence: two independent clones, one pushes, the other fetches then pulls; non-empty `have` set exercises the delta walk, not the cache HIT path. **✅ PASS — run against localhost.** - [x] `SC_03b` — pull after a force-push (interaction with MWP-8). **✅ PASS — gabriel explicitly approved via AskUserQuestion before running**, run against a localhost scratch repo (not shared staging). **Recipe (updated to target localhost per gabriel's local-first priority — the original recipe targeted staging, corrected here to match what was actually run):** ```bash # SC_03a — fetch convergence: two clones, one pushes, other fetches cd /tmp && rm -rf mwp-qa-local-v4-3a-x mwp-qa-local-v4-3a-y muse clone https://localhost:1337/gabriel/mwp-qa-local-v4 mwp-qa-local-v4-3a-x --json muse clone https://localhost:1337/gabriel/mwp-qa-local-v4 mwp-qa-local-v4-3a-y --json cd /tmp/mwp-qa-local-v4-3a-x echo "from-x" >> main_unique.txt && muse code add main_unique.txt muse commit -m "from x" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push origin main --json cd /tmp/mwp-qa-local-v4-3a-y muse fetch origin --branch main --json # non-empty `have` — exercises delta walk, not cache HIT muse log --json | jq -r '.commits[0].message' # should NOT yet show "from x" (fetch != merge) muse pull origin main --json cat main_unique.txt # NOW should show "from-x" appended — proves pull = fetch + merge converges # SC_03b — pull after a force-push (interaction with MWP-8) # Approved by gabriel before running. New scratch repo, not the SC_03a one. cd /tmp && rm -rf mwp-qa-force mwp-qa-force-y mkdir mwp-qa-force && cd mwp-qa-force muse init --json echo "c1" > file.txt && muse code add file.txt muse commit -m "c1" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse hub repo create --name mwp-qa-force --no-init --hub https://localhost:1337 --json muse remote add local2 https://localhost:1337/gabriel/mwp-qa-force --json muse push local2 main --json echo "c2-original" >> file.txt && muse code add file.txt muse commit -m "c2 original" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push local2 main --json cd /tmp && muse clone https://localhost:1337/gabriel/mwp-qa-force mwp-qa-force-y --json # y captures pre-force-push state cd /tmp/mwp-qa-force muse reset --json # soft reset, rewinds branch pointer only echo "c1" > file.txt && echo "c2-REWRITTEN" >> file.txt && muse code add file.txt muse commit -m "c2 rewritten (force-push scenario)" --agent-id claude-code --model-id claude-sonnet-4-6 --sign muse push local2 main --force --json # rewrites remote history cd /tmp/mwp-qa-force-y muse pull origin main --json # y still has the OLD c2 locally muse conflicts --json # inspect what pull actually did ``` **Result — `SC_03a`:** ✅ PASS. `muse fetch origin --branch main` sent `want=1 have=6` (non-empty `have`, exercising the delta walk / the `have_oids` fix, not the cache-HIT path). Fetch alone left `y`'s working tree and local HEAD unchanged (`main_unique.txt` still showed the old content, `muse log` still showed `y`'s own last commit) — confirming fetch ≠ merge. `muse pull` then correctly fast-forwarded (`status: "fast_forward"`, `commits_received: 0, blobs_written: 0` since the prior fetch already had everything) and the working tree converged to include `x`'s pushed content. **Result — `SC_03b`:** ✅ PASS, gabriel's explicit approval obtained via AskUserQuestion before running (recorded here per the safety note), run against a fresh localhost scratch repo (`gabriel/mwp-qa-force`, not the `SC_03a` repo, not shared staging). Sequence: pushed `c1`→`c2-original`; cloned `y` (captures pre-force-push state); on the original clone, soft-reset back to `c1` and committed a genuinely different `c2-rewritten`, then force-pushed. Pulling into `y` (which still had the old `c2-original` locally) produced: ```json {"status": "conflict", "exit_code": 1, "commits_received": 1, "blobs_written": 1, "head": null, "conflict_paths": ["file.txt"]} ``` `muse conflicts --json` confirmed `theirs_commit` was the genuinely new force-pushed commit (not a stale cached one — MWP-8's cache invalidation correctly delivered the real new content) and `ours_commit`/`base_commit` correctly identified `y`'s own history and the true common ancestor. **Pull did not silently pick a side, did not corrupt data, and did not serve stale content — it safely halted with a surfaced conflict**, exactly matching this workspace's own "never blindly `--ours`/`--theirs`" convention. The merge was left unresolved intentionally (the test's purpose — proving no silent divergence and no stale-cache interaction — was already fully served); the scratch directories are disposable and were not committed to anything. **Exit gate — Phase 3 fully closed.** Both `SC_03a` and `SC_03b` pass. --- ### Phase 4 — Claim 4: repeatable with no stale clones, no phantom large uploads, no manual cache-busting **Fixed by:** all of MWP-1..MWP-8 in combination; MWP-8 specifically removes the last known manual-cache-bust scenario (repair). **Already proven:** each sub-ticket proves its own slice is real-time self-healing (no `UPDATE musehub_fetch_mpack_cache ...` needed). **Residual gap this phase closes (the important one):** - **No test runs the full loop N times in sequence** (push, clone, push, clone, repair, clone, force-push, clone) and asserts *zero* manual intervention across the whole sequence. Each MWP proved its *own* trigger in isolation; nothing proves the *composition*. This phase depends on Phases 1–3 already being individually green (see Design above). - "No phantom large uploads" has only been checked at the byte-count level for two-branch topologies (MWP-7). A repeated-push loop (this phase) should also confirm upload size stays flat/small across many iterations, not just correct on push #2. - [x] `SC_04a` — the full repeated loop, one scratch repo, 10 iterations: `objects_sent`/`blobs_sent` stay flat (not growing) per iteration, and every clone's tail content matches the latest push with zero manual intervention. **✅ PASS — run against localhost.** **Recipe (retargeted to localhost per gabriel's local-first priority):** ```bash # SC_04a — the full repeated loop, one repo, ~10 iterations, watch for growth cd /tmp && rm -rf mwp-qa-4a && mkdir mwp-qa-4a && cd mwp-qa-4a muse init --json muse hub repo create --name mwp-qa-4a --no-init --hub https://localhost:1337 --json muse remote add local2 https://localhost:1337/gabriel/mwp-qa-4a --json for i in $(seq 1 10); do echo "iteration $i" >> log.txt muse code add log.txt muse commit -m "iter $i" --agent-id claude-code --model-id claude-sonnet-4-6 --sign push_out=$(muse push local2 main --json) echo "$push_out" | jq '{i: '"$i"', commits_sent, objects_sent, force}' rm -rf /tmp/mwp-qa-4a-clone-$i muse clone https://localhost:1337/gabriel/mwp-qa-4a /tmp/mwp-qa-4a-clone-$i --json tail -1 /tmp/mwp-qa-4a-clone-$i/log.txt # must equal "iteration $i" every time, no staleness done ``` **Result:** ✅ PASS, all 10 iterations. `objects_sent` stayed exactly `1` on every single iteration (no growth trend at all — not just "small," flat). `commits_received` on each clone correctly accumulated (1, 2, 3, ... 10) — every clone always got the complete history, never a truncated one. **Every one of the 10 clones' tail content matched exactly** (`"iteration 1"` through `"iteration 10"`, zero mismatches, zero staleness). Zero manual cache-busting, zero retries needed, zero intervention across the entire loop. `exit_code: 0` on every push and every clone. **Exit gate:** `SC_04a` passes across all 10 iterations with flat upload size and zero staleness — confirmed, no growth trend or stale-tail iteration found. **Phase 4 fully closed.** --- ### Phase 5 — Cross-cutting seams between MWP fixes These are the interaction risks that only make sense to chase once each MWP's *own* surface is confirmed working in isolation (Phases 1–4 green). This is the "deeper than surface" part of the review — the reason re-reading each ticket's own tests is not sufficient. - [x] `SC_05_01` — **MWP-3/4 (job ordering/idempotency) × MWP-8 (cache invalidation enqueues its own prebuild job).** `_enqueue_repair_prebuild` calls the same `enqueue_job` that MWP-3 hardened for push-triggered prebuilds. Check: has the repair-triggered enqueue path been checked against MWP-3's dedup-by-`mpack_key` logic? A repair followed immediately by a push could enqueue two prebuild jobs — does MWP-3's fix correctly coalesce them, or does one get silently dropped (regressing RC-3 in a new trigger)? **✅ VERIFIED CORRECT BY DESIGN — checked by direct code inspection, not inference.** `musehub_jobs.py:172-174`: `fetch.mpack.prebuild` is explicitly excluded from the per-`mpack_key` dedup (`coarse_jobs = [... if jt != "mpack.index"]`) — only `mpack.index` gets that treatment. `fetch.mpack.prebuild` intentionally uses the coarse `(repo_id, job_type, pending)` key, because `process_fetch_mpack_prebuild_job`'s own docstring confirms it's "self-coalescing": it re-reads live `MusehubBranch` tips at run time rather than trusting its enqueue-time payload (`payload["tip_commit_ids"]` is "retained only for observability"). A repair's `_enqueue_repair_prebuild` call colliding with an existing pending push-triggered prebuild is not a bug — whichever single job survives dedup still rebuilds against current (already-repaired) state when it eventually runs, because the repair's cache invalidation is already durably committed before the enqueue call happens. No RC-3 regression possible here. - [x] `SC_05_02` — **MWP-1/2 (generation authority) × MWP-7 (sibling-ref dedup).** MWP-7's dedup subtracts a sibling tip's full manifest. If that sibling's own generation was ever backfilled by MWP-1's on-demand repair, does the manifest lookup still resolve correctly, or could a repaired-generation commit be mis-scoped out of a dedup calculation? **✅ VERIFIED — architecturally independent, checked by direct code inspection.** `muse/cli/commands/push.py:830-838`: the client's sibling-ref dedup boundary (`branch_have`) is built entirely from `remote_branch_heads` (actual commit IDs fetched via `GET /refs`) and the client's own local `walk_commits()` — a parent-chain walk over the client's local repo, using no server-side `generation` data whatsoever. `MusehubCommitGraph.generation` is a purely server-side fetch-path optimization (used only in `_walk_commit_delta`'s range scan); push-side dedup never reads it. These two features share no state and cannot interact. - [x] `SC_05_03` — **MWP-5 (client retry) × MWP-4 (prebuild ordering).** MWP-5's bounded retry budget (120s default) was tuned against MWP-4's fixed ordering guarantee. If a repo is large enough that index+prebuild together exceed 120s, does the client fail with a clear, actionable error, or a confusing timeout? **✅ VERIFIED — real evidence already collected earlier in this investigation**, not simulated for this check. During the musehub#113 investigation, a genuinely stale/stuck server produced exactly this 120s-exhaustion condition (four retries, 30s apart). The client's actual output: `⏳ remote preparing fetch data (server busy, attempt 4); retrying in 30s … (waited 90s / 120s budget)` followed by `❌ Remote still preparing clone data after 120s — try again shortly.`, `exit_code: 3`, and the JSON envelope carries `"retryable": true`. This is a clear, actionable message with a machine-readable retry hint — not a confusing raw timeout or stack trace, regardless of *why* the server took too long. - [ ] `SC_05_04` — **Real R2 vs local MinIO.** Several sub-tickets' "best-effort R2 delete" paths (MWP-8's `invalidate_fetch_mpack_cache`) were only tested against a mocked backend. Staging uses real Cloudflare R2. Confirm the delete call shape matches R2's actual API (auth headers, error codes on 404-already-gone) rather than the mock's simplified contract. **Explicitly deferred, not skipped — written reason:** this specifically requires real R2 access to verify, which is out of scope while local-first is the explicit priority (no staging until local is airtight — gabriel's direction). Partial risk assessment done: both MinIO and R2 are accessed through the identical boto3 S3-compatible client code path (only the endpoint/credentials config differs), so the DELETE call shape itself is not expected to diverge — the open risk is specifically R2's response behavior on a delete-of-already-deleted object (a 404), which cannot be confirmed without hitting real R2. Revisit when this investigation resumes against staging. - [x] `SC_05_05` — **Concurrent agents on the same repo.** aaronrene will not be the only person pushing to shared repos. No MWP ticket tested two *different client machines* (not just two CLI invocations from one machine) pushing to the same branch concurrently. This is the actual onboarding scenario. **✅ VERIFIED — actually run, not inferred.** Two independent clones (`gabriel/mwp-qa-race`, simulating two agents), each committing independently from the same base, both pushes fired via genuine shell backgrounding (`(...&); (...&); wait`) to guarantee real OS-level overlap, not sequential back-to-back calls. Result: one push landed cleanly (`commits_sent: 1, objects_sent: 1`, exit 0); the other was correctly rejected by the atomic CAS branch-pointer update with a clean `409 diverged` error (`"push rejected: ... is not a fast-forward of ... — use --force to override"`) — no data loss (the losing commit is still safely in its own local repo, recoverable via pull+rebase), no corruption, no silent overwrite, no crash. One honest caveat: this was two truly concurrent *processes* from one physical machine, not two literal separate machines — but the server cannot distinguish the two cases (HTTP requests are indistinguishable by origin machine), so this is a faithful test of the actual concurrency-safety mechanism. **Exit gate:** each of `SC_05_01`..`SC_05_05` is either explicitly checked (with evidence) or explicitly deferred with a written reason — no silent skips. **4 of 5 checked with direct evidence (code inspection or live test); 1 explicitly deferred pending staging/R2 access, with the specific open risk narrowed down.** Phase 5 exit gate satisfied. --- ### Phase 6 — Cleanup & sign-off - [x] `SC_06_01` — all scratch repos (`gabriel/mwp-qa-*`) cleaned up / deleted from staging. **✅ DONE.** Five staging scratch repos existed (`mwp-qa-1a`, `mwp-qa-atomic-1..4`, all created before the local-first pivot — everything from Phase 2 onward ran against localhost instead, so no further staging scratch repos accumulated). Before deleting, gabriel requested a full precautionary backup given this session's own findings about subtle scoping bugs: cascade-delete FK scoping for issues/proposals/mists was independently verified safe by code inspection (all FK to `repo_id` with correct `ondelete="CASCADE"`; mists are structurally independent repos, unreachable by another repo's delete), AND a full `pg_dump` of the staging DB was taken via SSM, uploaded to a temporary private/encrypted S3 bucket, downloaded to gabriel's Desktop, verified as a complete valid dump (correct pg_dump header/trailer, 69 `COPY` statements), before all temporary cloud artifacts were cleaned up. Gabriel then independently confirmed via the staging UI that none of the 5 repos had any attached proposals or issues. All 5 repos deleted (`muse hub repo delete --yes`), each returning `deleted: true`. Post-deletion `hub repo list` confirmed only `muse`, `musehub`, `muse-zsh`, `identity`, and mist repos remain — nothing else touched. - [x] `SC_06_02` — findings from every phase written back into this file (this file is the permanent record, not a scratch pad — update it as phases complete, per the workspace body-copy-quality convention). **✅ DONE** — every phase (0 through 5) has its results, evidence, and any deferred items recorded inline as they completed, pushed to staging muse#63 after each phase. - [x] `SC_06_03` — any new bug discovered during any phase filed as its own musehub/muse ticket (not silently patched in place). **✅ DONE** — [musehub#113](https://staging.musehub.ai/gabriel/musehub/issues/113) (the silent partial clone / multi-tip manifest union bug, root-caused and fixed with two TDD'd commits), [musehub#114](https://staging.musehub.ai/gabriel/musehub/issues/114) (dead `process_mpack_gc_job`/`purge_stale_mpack_index_entries` code, correction history included), [musehub#115](https://staging.musehub.ai/gabriel/musehub/issues/115) (Docker container restart/rebuild guidance for the agent-config docs). - [ ] `SC_06_04` — gabriel gives explicit go-ahead to send aaronrene the staging URL. **Pending** — this is the final sign-off step; not yet given. **Exit gate:** all of Phase 0–5's exit gates are satisfied — confirmed. This issue closes only after gabriel's explicit sign-off in `SC_06_04`. --- ## Safety notes for this QA pass - **All QA repos are scratch repos under `gabriel/mwp-qa-*`** on staging — never reuse `gabriel/muse` or `gabriel/musehub` themselves for destructive operations (force-push, reset --hard, repair-*). - Per workspace hard rules: no `--force` flag, no `reset --hard`, no `checkout --ours/--theirs` without gabriel's explicit go-ahead in the conversation, even inside a scratch repo — ask first if a recipe step needs one. This specifically gates `SC_03b`. - Every `/tmp` directory created here is disposable; clean up with a plain `rm -rf` when done (not a muse operation, so no special permission needed). - If any staging repair endpoint needs to be exercised (Phase 1's repair × clone interaction, or `SC_05_01`), coordinate the exact repo/commit with gabriel before running it — repair mutates content-addressed rows in place. --- ## Acceptance criteria - [x] Phase 0 — wire suite green in both repos, verified fresh (not from memory of a prior run). ✅ 56 (musehub) + 21 (muse) passed, 0 failed. - [x] Phase 1 — `SC_01a`, `SC_01b` pass against real staging. ✅ Content correctness proven; 503/retry-path timing not exercised (repo too small) — tracked as a follow-up, not blocking. - [x] Phase 2 — `SC_02a` ✅, `SC_02b` ✅ (dedup confirmed; the blocking bug it surfaced is now root-caused and fixed via musehub#113 — two fixes, `all_oids` and `have_oids`, both TDD'd and verified), `SC_02c` ✅ (rebase dry-run parity, exact match). **Fully green.** - [x] Phase 3 — `SC_03a` ✅ (fetch ≠ merge, pull converges via fast-forward). `SC_03b` ✅ (gabriel's sign-off obtained; pull after force-push safely surfaces a conflict, no silent divergence, no stale cache content). **Fully green.** - [x] Phase 4 — `SC_04a` (10-iteration loop) shows flat upload size (exactly 1 object/iteration, no growth), zero staleness (10/10 clone tails matched exactly). **Fully green.** - [x] Phase 5 — all 5 cross-cutting seams checked or explicitly deferred. 4/5 verified with direct evidence (`SC_05_01` code inspection, `SC_05_02` code inspection, `SC_05_03` real prior evidence, `SC_05_05` live concurrent-push test); `SC_05_04` explicitly deferred pending real R2/staging access. **Fully green.** - [ ] Phase 6 — scratch repos cleaned up ✅ (5 staging repos deleted, backup taken first, cascade-scoping verified safe), findings recorded ✅, new bugs filed as their own tickets ✅ (musehub#113/114/115). **Only `SC_06_04` (gabriel's final go-ahead to send aaronrene the staging URL) remains before this ticket can close.** ## Out of scope - Re-litigating whether each MWP-1..MWP-8 sub-ticket's own unit tests are well-designed — they are; this ticket only adds live-staging, black-box, and cross-cutting evidence on top. - Any new correctness fix discovered here is **filed as its own ticket**, not patched inline as part of this spot-check (see `SC_06_03`). - Load/perf testing beyond the 10-iteration loop in Phase 4 — this ticket proves correctness under realistic single-developer usage, not high-concurrency stress (that would be a separate ticket if `SC_05_05` surfaces a real gap).