MWP-8 / #58 Spot Check — deep verification of wire protocol MVP goal-of-done

Task: Spot Check MWP — deep verification of #58's "definition of MVP done"

Background

Issue #58 — "Muse Wire Protocol MVP hardening" — lists 8 sub-tickets (MWP-1..MWP-8), all now closed with green test suites:

Every sub-ticket shipped with unit/integration tests that pass in isolation (commit-by-commit, phase-by-phase). What has not been independently proven is the compound claim in #58's "Goal — definition of MVP done": that a real developer, running the real CLI against real staging, in the order they'd actually use it, gets correct behavior every time — including at the seams between fixes (e.g. does MWP-8's cache invalidation interact correctly with MWP-3/4's job-ordering fix? does MWP-5's retry loop actually fire against a real 503 from staging, not a mocked one?).

This is the gap this spot-check closes. We are not re-reviewing whether each MWP's unit tests are well-written — they are. We are asking: does the system, as a whole, running as a black box against staging, satisfy the five numbered claims in #58's goal section? Only after this is true do we bring in aaronrene.

Goal — definition of MVP done (verbatim from #58)

aaronrene (or any dev) can, against staging, reliably:

1. muse clone <url> — get a complete, correct working tree every time, including immediately after a push (with bounded client retry on 503).
2. muse push — upload only the true delta; never re-upload objects already on the remote via any ref; survive back-to-back pushes.
3. muse fetch / muse pull — receive exactly the new commits/blobs and converge.
4. Do all of the above repeatedly with no stale clones, no phantom large uploads, no manual cache-busting.
5. The wire regression suite is re-enabled and green as the proof.

Each claim below is broken into: what fixed it, what's already proven (from the sub-ticket's own test suite), the residual gap (what unit tests can't tell us), and the concrete /tmp CLI recipe that closes the gap against live staging.

Claim 1 — muse clone is always complete and correct, including immediately after a push

Fixed by: MWP-1 (gen-0 authority), MWP-2 (DAG-walk fallback), MWP-5 (bounded 503 retry), MWP-8 (cache never stale after repair/force-push).

Already proven (unit/integration level):

• MWP1_14 — push→clone→push→clone E2E gate (musehub, in-process).
• MWP2_12 — fallback alone ships a complete clone even with repair patched out.
• MWP5 real-urllib E2E against a live localhost server (retry loop fires on real 503).
• EX_02 (MWP-8) — clone-after-repair E2E, @pytest.mark.wire.

Residual gap:

• All of the above run against localhost or in-process fixtures, not staging (real network latency, real R2, real multi-second prebuild time). The 503→retry window on localhost may not exercise the same timing as staging's actual prebuild latency.
• No test proves clone correctness chained across a push→clone→push→clone cycle that also crosses a force-push or repair in the middle — i.e. the MWP-1/2 fix and the MWP-8 fix have never been exercised in the same sequence.
• No test proves clone works correctly for a brand-new repo that has never been pushed to before against staging specifically (first-ever clone, cold cache, real MISS→prebuild→retry→HIT path end to end).

/tmp recipe:

# 1a — cold clone of a never-touched repo (first MISS, real prebuild latency)
cd /tmp && rm -rf mwp-qa-1a && mkdir mwp-qa-1a && cd mwp-qa-1a
muse init --json
echo "hello" > README.md && muse code add README.md
muse commit -m "init" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-1a
muse push staging main --json   # first push to a brand-new repo

cd /tmp && rm -rf mwp-qa-1a-clone
time muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1a-clone --json
# expect: complete tree, README.md present, exit 0 — measure how long the
# 503-retry loop actually waited (proves MWP-5's bounded poll against real
# staging timing, not a mock)

# 1b — push then immediate clone (the exact MWP-1/RC-1 regression)
cd /tmp/mwp-qa-1a
echo "v2" >> README.md && muse code add README.md
muse commit -m "v2" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json
cd /tmp && rm -rf mwp-qa-1b-clone
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1b-clone --json
diff /tmp/mwp-qa-1a/README.md /tmp/mwp-qa-1b-clone/README.md   # must be identical, must contain "v2"

Claim 2 — muse push uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes

Fixed by: MWP-3 (job enqueue reflects latest tip), MWP-4 (prebuild/index ordering), MWP-7 (sibling-ref have-negotiation, dry-run parity).

Already proven:

• MWP3_10/11 — two-push full drain, N=10 stress partial drain (in-process job queue simulation).
• MWP4_11/12 — E2E claim loop + N=10 stress, zero FetchNotIndexedError.
• MWP-7 Phase 3 (VS) — real staging push, 17-commit dev + 1-commit main, commits_sent=1, objects_sent=1.

Residual gap:

• MWP-3/4's stress tests simulate concurrent job claiming in-process — they do not prove that a human issuing two real muse push commands back to back (i.e. two separate CLI processes, two separate HTTP round-trips) against staging converges correctly. Real back-to-back means real race conditions in the async worker, not simulated ones.
• No test proves the cross-branch dedup claim ("never re-upload objects already on the remote via any ref") for a third sibling branch beyond the two-branch case MWP-7 tested. If feature-a, feature-b, and dev all share a base and feature-b is pushed last, does it dedup against both siblings or just the most-recently-pushed one?
• --dry-run parity was fixed for a single scenario (MWP-7 Phase 4/5) — needs re-verification against a different topology (e.g. a rebased branch) to rule out a narrow fix.

/tmp recipe:

# 2a — literal back-to-back push (two real CLI invocations, no artificial delay)
cd /tmp/mwp-qa-1a
echo "a" >> README.md && muse code add README.md
muse commit -m "a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json &
echo "b" >> README.md && muse code add README.md    # NOTE: run after push A's mpack-presign step ideally overlaps; simplest form below
wait
echo "b" >> README.md && muse code add README.md
muse commit -m "b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json    # push B immediately after A returns
cd /tmp && rm -rf mwp-qa-2a-clone && muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2a-clone --json
tail -3 mwp-qa-2a-clone/README.md    # must show both "a" and "b"

# 2b — three-sibling-branch dedup
cd /tmp/mwp-qa-1a
muse checkout -b feature-a && echo "fa" > fa.txt && muse code add fa.txt
muse commit -m "feature a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-a --json
muse checkout main && muse checkout -b feature-b && echo "fb" > fb.txt && muse code add fb.txt
muse commit -m "feature b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-b --json --json | jq '.objects_sent'   # should exclude blobs already on staging via feature-a's shared base

# 2c — dry-run parity on a rebased topology
muse rebase main --dry-run --json    # compare counts to a real rebase + push

Claim 3 — muse fetch / muse pull receive exactly the new commits/blobs and converge

Fixed by: MWP-1/2 (correct generation walk), MWP-7 (sibling-ref negotiation applies to fetch's have set too).

Already proven:

• Fetch-side coverage is implied by the same MWP-1/2 test suite (fetch and clone share _walk_commit_delta).

Residual gap:

• No sub-ticket wrote a dedicated fetch/pull convergence test against staging — all direct evidence is clone-shaped (have=[]). fetch/pull with a non-empty have set exercise a materially different code path (delta walk, not cache HIT) that has not been the subject of an explicit E2E in this campaign.
• pull = fetch + merge/ff — the merge/ff correctness after a fetch has never been explicitly re-tested post-MWP-8 (does a pull after a force-pushed remote branch behave correctly, or does it silently diverge?).

/tmp recipe:

# 3a — fetch convergence: two clones, one pushes, other fetches
cd /tmp && rm -rf mwp-qa-3a-x mwp-qa-3a-y
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-3a-x --json
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-3a-y --json
cd /tmp/mwp-qa-3a-x
echo "from-x" >> README.md && muse code add README.md
muse commit -m "from x" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json
cd /tmp/mwp-qa-3a-y
muse fetch staging main --json    # non-empty `have` — exercises delta walk, not cache HIT
muse log --json | jq '.commits[0].message'   # should NOT yet show "from x" (fetch != merge)
muse pull staging main --json
grep "from-x" README.md    # NOW should be present — proves pull = fetch + merge converges

# 3b — pull after a force-push (interaction with MWP-8)
cd /tmp/mwp-qa-1a
muse reset HEAD~1 --json   # local-only rewrite, still on main — DO NOT run --hard without asking gabriel first
# (coordinate with gabriel before any reset/force-push against a shared staging repo — see Safety notes)

Claim 4 — Repeatable with no stale clones, no phantom large uploads, no manual cache-busting

Fixed by: all of MWP-1..MWP-8 in combination; MWP-8 specifically removes the last known manual-cache-bust scenario (repair).

Already proven: each sub-ticket proves its own slice is real-time self-healing (no UPDATE musehub_fetch_mpack_cache ... needed).

Residual gap — this is the important one:

• No test runs the full loop N times in sequence (push, clone, push, clone, repair, clone, force-push, clone) and asserts zero manual intervention across the whole sequence. Each MWP proved its own trigger in isolation; nothing proves the composition.
• "No phantom large uploads" has only been checked at the byte-count level for two-branch topologies (MWP-7). A repeated-push loop (claim 4) should also confirm upload size stays flat/small across many iterations, not just correct on push #2.

/tmp recipe:

# 4a — the full repeated loop, one repo, ~10 iterations, watch for growth
cd /tmp && rm -rf mwp-qa-4a && mkdir mwp-qa-4a && cd mwp-qa-4a
muse init --json
muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-4a
for i in $(seq 1 10); do
  echo "iteration $i" >> log.txt
  muse code add log.txt
  muse commit -m "iter $i" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
  push_out=$(muse push staging main --json)
  echo "$push_out" | jq '{i: '"$i"', objects_sent, blobs_sent, size_bytes}'
  rm -rf /tmp/mwp-qa-4a-clone-$i
  muse clone https://staging.musehub.ai/gabriel/mwp-qa-4a /tmp/mwp-qa-4a-clone-$i --json
  tail -1 /tmp/mwp-qa-4a-clone-$i/log.txt   # must equal "iteration $i" every time, no staleness
done
# expect: objects_sent/blobs_sent stay ~1 per iteration (not growing), every clone's tail matches

Claim 5 — Wire regression suite is re-enabled and green

Fixed by: MWP-6.

Already proven: 55 tests under pytest -m wire green (47 in-process + 8 slow subprocess E2E); MWP6_00 gate enforces no re-introduction of flux-skips.

Residual gap:

• This claim is the most rigorously proven of the five — but it should be re-run right now, post-MWP-8-merge-to-main, to confirm nothing in the final merge sequence broke it. This is cheap and should be step 0 of the spot-check, not step 5.

Recipe:

cd ~/ecosystem/musehub
python3 -m pytest -m wire -q --tb=short
cd ~/ecosystem/muse
python3 -m pytest -m wire -q --tb=short

Cross-cutting risks not captured by any single MWP's test suite

These are the seams between fixes — the reason a "deeper than surface" review matters more than re-reading each ticket's own tests:

1. MWP-3/4 (job ordering/idempotency) × MWP-8 (cache invalidation enqueues its own prebuild job) — _enqueue_repair_prebuild calls the same enqueue_job that MWP-3 hardened for push-triggered prebuilds. Has the repair-triggered enqueue path been checked against MWP-3's dedup-by- mpack_key logic? A repair followed immediately by a push could enqueue two prebuild jobs — does MWP-3's fix correctly coalesce them, or does one get silently dropped (regressing RC-3 in a new trigger)?
2. MWP-1/2 (generation authority) × MWP-7 (sibling-ref dedup) — MWP-7's dedup subtracts a sibling tip's full manifest. If that sibling's own generation was ever backfilled by MWP-1's on-demand repair, does the manifest lookup still resolve correctly, or could a repaired-generation commit be mis-scoped out of a dedup calculation?
3. MWP-5 (client retry) × MWP-4 (prebuild ordering) — MWP-5's bounded retry budget (120s default) was tuned against MWP-4's fixed ordering guarantee. If a repo is large enough that index+prebuild together exceed 120s, does the client fail with a clear, actionable error, or a confusing timeout?
4. Real R2 vs local MinIO — several sub-tickets' "best-effort R2 delete" paths (MWP-8's invalidate_fetch_mpack_cache) were only tested against a mocked backend. Staging uses real Cloudflare R2. Confirm the delete call shape matches R2's actual API (auth headers, error codes on 404-already- gone) rather than the mock's simplified contract.
5. Concurrent agents on the same repo — aaronrene will not be the only person pushing to shared repos. No MWP ticket tested two different client machines (not just two CLI invocations from one machine) pushing to the same branch concurrently. This is the actual onboarding scenario.

Safety notes for this QA pass

• All QA repos are scratch repos under gabriel/mwp-qa-* on staging — never reuse gabriel/muse or gabriel/musehub themselves for destructive operations (force-push, reset --hard, repair-*).
• Per workspace hard rules: no --force flag, no reset --hard, no checkout --ours/--theirs without gabriel's explicit go-ahead in the conversation, even inside a scratch repo — ask first if a recipe step needs one.
• Every /tmp directory created here is disposable; clean up with a plain rm -rf when done (not a muse operation, so no special permission needed).
• If any staging repair endpoint needs to be exercised (Claim 1's repair× clone interaction), coordinate the exact repo/commit with gabriel before running it — repair mutates content-addressed rows in place.

Sign-off checklist before aaronrene onboards

• Claim 1 recipes (1a, 1b) run clean against staging
• Claim 2 recipes (2a, 2b, 2c) run clean against staging
• Claim 3 recipes (3a; 3b coordinated with gabriel) run clean
• Claim 4 recipe (4a, 10-iteration loop) shows flat upload size, zero staleness
• Claim 5 — pytest -m wire green in both repos, run fresh today
• Cross-cutting risks 1–5 above each explicitly checked or explicitly deferred with a written reason
• All scratch repos (gabriel/mwp-qa-*) cleaned up / deleted from staging
• Findings written back into this file; any new bugs filed as their own musehub/muse tickets (not silently patched)
• gabriel gives explicit go-ahead to send aaronrene the staging URL