Open #63

filed by gabriel human · 15 days ago

Spot check #58: deep verification of the wire protocol MVP goal-of-done before aaronrene onboards

0 Anchors

— Blast radius

— Churn 30d

0 Proposals

MWP-9 — Spot check #58: deep verification of the wire protocol MVP goal-of-done

Master tracker: muse#58 — Muse Wire Protocol MVP hardening. MWP-1..MWP-8 are all closed; this is the verification pass before aaronrene onboards. Historical artifact: the pre-phase-refactor version of this doc was published as a mist: https://staging.musehub.ai/gabriel/mists/FVTnZi7e7Q3b (kept live, superseded by this issue as the tracking source of truth). Repo target: cross-repo (muse client + musehub server); scratch repos only (gabriel/mwp-qa-* on staging) — never gabriel/muse or gabriel/musehub themselves.

IMPORTANT — this project uses Muse (not git) for version control.

Never run git, gh, or any git subcommand. Repo paths: ~/ecosystem/muse, ~/ecosystem/musehub, ~/ecosystem/agentception.

Commit with full provenance: muse commit -m "..." --agent-id claude-code --model-id claude-sonnet-4-6 --sign.

Branch first; never commit to dev/main. Never merge to dev/main without gabriel's explicit permission.

No --force flag, no reset --hard, no checkout --ours/--theirs — even inside a disposable scratch repo — without gabriel's explicit go-ahead in the conversation (see Safety notes).

Background

Issue #58 — "Muse Wire Protocol MVP hardening" — lists 8 sub-tickets (MWP-1..MWP-8), all now closed with green test suites:

Sub-ticket	Repo	Fixes	Status
MWP-1	musehub#106	RC-1 (gen-0 fallback corrupts fetch walk)	✅ closed
MWP-2	musehub#107	RC-2 (`_walk_commit_delta` no fallback)	✅ closed
MWP-3	musehub#108	RC-3 (job enqueue drops back-to-back pushes)	✅ closed
MWP-4	musehub#109	RC-4 (prebuild/index ordering race)	✅ closed
MWP-5	muse#59	RC-5 (client doesn't retry on 503)	✅ closed
MWP-6	musehub+muse	RC-6 (wire suite disabled)	✅ closed
MWP-7	muse#62	RC-7 (#55 sibling-ref negotiation, #32 dry-run)	✅ closed
MWP-8	musehub#111	RC-8 (repair/force-push don't bust fetch cache)	✅ closed

Every sub-ticket shipped with unit/integration tests that pass in isolation (commit-by-commit, phase-by-phase). What has not been independently proven is the compound claim in #58's "Goal — definition of MVP done": that a real developer, running the real CLI against real staging, in the order they'd actually use it, gets correct behavior every time — including at the seams between fixes (e.g. does MWP-8's cache invalidation interact correctly with MWP-3/4's job-ordering fix? does MWP-5's retry loop actually fire against a real 503 from staging, not a mocked one?).

This is the gap this ticket closes. We are not re-reviewing whether each MWP's unit tests are well-written — they are. We are asking: does the system, as a whole, running as a black box against staging, satisfy the five numbered claims in #58's goal section? Only after this is true do we bring in aaronrene.

Goal — definition of MVP done (verbatim from #58)

aaronrene (or any dev) can, against staging, reliably:

muse clone <url> — get a complete, correct working tree every time, including immediately after a push (with bounded client retry on 503).

muse push — upload only the true delta; never re-upload objects already on the remote via any ref; survive back-to-back pushes.

muse fetch / muse pull — receive exactly the new commits/blobs and converge.

Do all of the above repeatedly with no stale clones, no phantom large uploads, no manual cache-busting.

The wire regression suite is re-enabled and green as the proof.

Goal — definition of done for this ticket

Each of the 5 claims above is independently exercised against real staging (not localhost, not in-process fixtures, not mocks) via a dedicated /tmp scratch repo, and the 5 cross-cutting seams between MWP fixes are each explicitly checked or explicitly deferred with a written reason. No claim is marked done on the strength of a sub-ticket's own unit tests alone — every claim here requires fresh, live evidence.

Design — why this phase order

Phases are ordered cheapest/most-foundational first, then increasing interaction complexity, mirroring the workspace's load-bearing phase convention:

Phase 0 first, always — the wire suite (Claim 5) is already the most rigorously proven claim, but it costs seconds to re-run and must be green before we build any scratch-repo evidence on top of it. If Phase 0 is red, every later phase's result is suspect.
Phases 1–3 are single-claim, single-command-family checks (clone, then push, then fetch/pull) — each is a superset of what its own MWP tickets proved, but against live staging instead of localhost/mocks. Ordered clone → push → fetch/pull because clone is the simplest black-box surface (no delta negotiation) and push/fetch build on the same scratch repos clone establishes.
Phase 4 is the composition check (Claim 4) — deliberately last among the per-claim phases because it depends on Phases 1–3 already being individually green. Running the full repeated loop before the individual claims are proven would conflate "which claim failed" if something breaks.
Phase 5 (cross-cutting seams) comes after all 5 claims are individually green — these are interaction risks between MWPs that only make sense to chase once each MWP's own surface is confirmed working in isolation.
Phase 6 (cleanup + sign-off) is last — no scratch repos are deleted until every phase's findings are written back into this file, so a re-run/audit trail survives if aaronrene hits something during onboarding.

Never skip ahead. Each phase must be fully green (or have its gaps explicitly written up as deferred) before the next phase begins.

Phases

Execution rule: work one phase at a time; check in with gabriel after each phase before starting the next — comprehensive findings over speed. Test IDs (SC_00, SC_01a, …) appear in this plan; record results inline under each recipe as it's executed (pass/fail, staging repo used, and any deviation from the recipe as written).

Phase 0 — Baseline: wire suite green, right now, post-merge

SC_00 — re-run the full pytest -m wire suite in both repos, fresh, after the MWP-8 merge-to-main sequence, to confirm nothing in the final merge broke it. This is the cheapest possible check and gates every later phase. ✅ PASS

Recipe:

cd ~/ecosystem/musehub
python3 -m pytest -m wire -q --tb=short
cd ~/ecosystem/muse
python3 -m pytest -m wire -q --tb=short

Already proven (context, not a substitute for re-running now): 55 tests under pytest -m wire green (47 in-process + 8 slow subprocess E2E); MWP6_00 gate enforces no re-introduction of flux-skips.

Result (SC_00, run fresh post-MWP-8-merge):

Repo	Result	Duration
musehub	56 passed, 0 failed, 0 skipped	137.85s
muse	21 passed, 0 failed, 0 skipped	17.07s

musehub's count rose from the previously-documented 55 to 56 — expected, not a regression: MWP-8 added EX_02 (@pytest.mark.wire, the clone-after-repair E2E), which is now part of the wire-marked corpus. test_mwp6_wire_suite_enabled.py (the no-flux-skip gate) and test_mwp8_cache_invalidation.py's wire-marked test both appear in the run.

Exit gate: both suites green. If either is red, STOP — do not proceed to Phase 1 until it's understood whether the break is pre-existing or a genuine regression from the MWP-8 merge. Gate satisfied — proceeding to Phase 1.

Phase 1 — Claim 1: `muse clone` is always complete and correct, including immediately after a push

Fixed by: MWP-1 (gen-0 authority), MWP-2 (DAG-walk fallback), MWP-5 (bounded 503 retry), MWP-8 (cache never stale after repair/force-push).

Already proven (unit/integration level):

MWP1_14 — push→clone→push→clone E2E gate (musehub, in-process).
MWP2_12 — fallback alone ships a complete clone even with repair patched out.
MWP5 real-urllib E2E against a live localhost server (retry loop fires on real 503).
EX_02 (MWP-8) — clone-after-repair E2E, @pytest.mark.wire.

Residual gap this phase closes:

All of the above run against localhost or in-process fixtures, not staging (real network latency, real R2, real multi-second prebuild time). The 503→retry window on localhost may not exercise the same timing as staging's actual prebuild latency.
No test proves clone correctness chained across a push→clone→push→clone cycle that also crosses a force-push or repair in the middle — i.e. the MWP-1/2 fix and the MWP-8 fix have never been exercised in the same sequence.
No test proves clone works correctly for a brand-new repo that has never been pushed to before against staging specifically (first-ever clone, cold cache, real MISS→prebuild→retry→HIT path end to end).
SC_01a — cold clone of a never-touched repo (first MISS, real prebuild latency). ✅ PASS (content correctness); ⚠️ see finding below — 503/retry path not exercised.
SC_01b — push then immediate clone (the exact MWP-1/RC-1 regression), against real staging timing. ✅ PASS (content correctness); ⚠️ same finding.

Recipe (corrected — the original omitted the required repo-create step; muse push 404s against a repo that doesn't exist on the hub yet):

# SC_01a — cold clone of a never-touched repo (first MISS, real prebuild latency)
cd /tmp && rm -rf mwp-qa-1a && mkdir mwp-qa-1a && cd mwp-qa-1a
muse init --json
echo "hello" > README.md && muse code add README.md
muse commit -m "init" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse hub repo create --name mwp-qa-1a --no-init --hub https://staging.musehub.ai --json
muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-1a
muse push staging main --json   # first push to a brand-new repo

cd /tmp && rm -rf mwp-qa-1a-clone
time muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1a-clone --json
# expect: complete tree, README.md present, exit 0 — measure how long the
# 503-retry loop actually waited (proves MWP-5's bounded poll against real
# staging timing, not a mock)

# SC_01b — push then immediate clone (the exact MWP-1/RC-1 regression)
cd /tmp/mwp-qa-1a
echo "v2" >> README.md && muse code add README.md
muse commit -m "v2" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json
cd /tmp && rm -rf mwp-qa-1b-clone
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1b-clone --json
diff /tmp/mwp-qa-1a/README.md /tmp/mwp-qa-1b-clone/README.md   # must be identical, must contain "v2"

Result — content correctness:

Test	Result	Detail
`SC_01a`	✅ PASS	Cold clone of brand-new `mwp-qa-1a`: `README.md` content identical byte-for-byte, `commits_received=1`, `blobs_written=1`, exit 0. Wall time 1.63s.
`SC_01b`	✅ PASS	Push `v2` then clone immediately (chained in one command, zero artificial delay): both commits present (`commits_received=2`), content identical including `v2` line, exit 0. Wall time 1.34s.

Finding — the 503/retry path was never actually exercised:

Both clones returned HTTP 200 on the first POST /fetch/mpack call — no 503, no Retry-After, no retry loop observed in either run, even in SC_01b where the clone was issued in the same shell chain immediately after the push returned. This means the content-correctness half of Claim 1 is proven against real staging, but the retry-mechanism half (MWP-5's bounded poll actually firing against real staging prebuild latency, not a mock) was not exercised — the scratch repo is too small (1–2 commits, <2KB) for the prebuild job to lag behind the client's request at all, even with near-zero gap between push and clone.

This is a real, not cosmetic, gap relative to the phase's stated goal ("measure how long the 503-retry loop actually waited"). MWP-5's own test suite already proves the retry loop fires correctly against a mocked 503 — what's still unproven is that it fires (and behaves correctly) against a real staging 503 under real prebuild latency. Closing this fully would require a repo large enough, or a push volume high enough, to reliably lag the prebuild job — deferred to Phase 4's repeated-loop test (SC_04a) or a dedicated large-payload follow-up, rather than blocking Phase 1's exit gate on it, since content correctness (the primary contract) is proven.

Exit gate: SC_01a and SC_01b both pass against staging (content correctness) — satisfied. Timing observation recorded: retry loop was not exercised on this scratch repo's scale; carried forward as an explicit follow-up rather than re-run here, since Phase 1's content-correctness contract is the gating requirement and it is fully met. Proceeding to Phase 2, with the retry-path gap tracked for revisit in Phase 4 or as a dedicated large-payload check.

Phase 2 — Claim 2: `muse push` uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes

Fixed by: MWP-3 (job enqueue reflects latest tip), MWP-4 (prebuild/index ordering), MWP-7 (sibling-ref have-negotiation, dry-run parity).

Already proven:

MWP3_10/11 — two-push full drain, N=10 stress partial drain (in-process job queue simulation).
MWP4_11/12 — E2E claim loop + N=10 stress, zero FetchNotIndexedError.
MWP-7 Phase 3 (VS) — real staging push, 17-commit dev + 1-commit main, commits_sent=1, objects_sent=1.

Residual gap this phase closes:

MWP-3/4's stress tests simulate concurrent job claiming in-process — they do not prove that a human issuing two real muse push commands back to back (i.e. two separate CLI processes, two separate HTTP round-trips) against staging converges correctly. Real back-to-back means real race conditions in the async worker, not simulated ones.
No test proves the cross-branch dedup claim ("never re-upload objects already on the remote via any ref") for a third sibling branch beyond the two-branch case MWP-7 tested. If feature-a, feature-b, and dev all share a base and feature-b is pushed last, does it dedup against both siblings or just the most-recently-pushed one?
--dry-run parity was fixed for a single scenario (MWP-7 Phase 4/5) — needs re-verification against a different topology (e.g. a rebased branch) to rule out a narrow fix.
SC_02a — literal back-to-back push (two real CLI invocations, no artificial delay), converges correctly. ✅ PASS.
SC_02b — three-sibling-branch dedup: pushing the third branch excludes blobs already on staging via either sibling. ✅ dedup itself PASS — but surfaced a separate, more severe bug while verifying it (see below). 🔴 BLOCKING FINDING.
SC_02c — dry-run parity holds on a rebased topology, not just the original MWP-7 scenario. ✅ PASS — run against localhost (musehub#113's fix now live, both fixes verified in place), after resuming per gabriel's local-first priority.

Recipe (corrected — the original SC_02a snippet had a broken background-job/echo race that didn't test what it claimed; replaced with what was actually run: two real sequential commits + pushes, zero artificial delay between them):

# SC_02a — two real sequential CLI pushes, back-to-back, no artificial delay
cd /tmp/mwp-qa-1a
echo "a" >> README.md && muse code add README.md
muse commit -m "a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json                          # push A
echo "b" >> README.md && muse code add README.md
muse commit -m "b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json                           # push B, immediately after A returns
cd /tmp && rm -rf mwp-qa-2a-clone && muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2a-clone --json
tail -3 mwp-qa-2a-clone/README.md    # must show both "a" and "b"

# SC_02b — three-sibling-branch dedup
cd /tmp/mwp-qa-1a
muse checkout -b feature-a && echo "fa" > fa.txt && muse code add fa.txt
muse commit -m "feature a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-a --json
muse checkout main && muse checkout -b feature-b && echo "fb" > fb.txt && muse code add fb.txt
muse commit -m "feature b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-b --json | jq '.objects_sent'   # should exclude blobs already on staging via feature-a's shared base

# SC_02c — dry-run parity on a rebased topology
muse rebase main --dry-run --json    # compare counts to a real rebase + push

Result — SC_02a: ✅ PASS. 4 commits (init, v2, a, b) converged correctly on clone, content in exact order, no staleness despite zero-delay back-to-back pushes.

Result — SC_02b: the dedup claim itself is ✅ confirmed correct, and more thoroughly than planned — feature-b's push sent exactly 1 object (fb.txt's blob), correctly excluding README.md's blob across the entire 4-commit shared ancestor chain with both main and feature-a. RC-7/MWP-7's sibling-ref dedup holds up under a 3-branch test, not just the 2-branch case it originally shipped with.

🔴 But verifying that dedup by cloning feature-b surfaced a separate, more severe bug: muse clone <url> --branch feature-b returned exit_code: 0 / status: "cloned" while its own log stated the working tree was only partially restored — fb.txt was completely missing from disk. muse verify --json on the resulting clone reported all_ok: true with zero failures, giving no signal anything was wrong. This directly contradicts Claim 1 of #58 ("get a complete, correct working tree every time"), not just Claim 2. Full brain-dump, reproduction steps, and open questions filed as its own investigation ticket rather than patched inline: musehub#113. No production code was touched while chasing this down.

Result — SC_02c: ✅ PASS, run against localhost after musehub#113's two fixes landed. Topology: main (base commit), feature branches off and gets 2 commits, main then diverges with its own new commit, feature is rebased onto the new main tip (rewriting both of feature's commit IDs). muse rebase main --dry-run --json correctly predicted count: 2 before rebasing. After the real rebase, muse push local2 feature --dry-run --force --json predicted commits_sent: 2, objects_sent: 0 (file content unchanged by rebase, already known to the remote) — the real forced push produced the exact same commits_sent: 2, objects_sent: 0, blobs_written: 0 server-side. Exact parity, not just close. A fresh clone of the rebased feature branch confirmed correct content (base.txt, f1.txt, f2.txt, main2.txt all present) — this also exercises musehub#113's multi-tip manifest-union fix, since the repo has multiple branch tips.

Exit gate — Phase 2 now fully closed. SC_02a ✅, SC_02b's dedup claim ✅ (and the correctness bug it surfaced is now root-caused and fixed via musehub#113, not just filed), SC_02c ✅. All three green.

Phase 3 — Claim 3: `muse fetch` / `muse pull` receive exactly the new commits/blobs and converge

Fixed by: MWP-1/2 (correct generation walk), MWP-7 (sibling-ref negotiation applies to fetch's have set too).

Already proven:

Fetch-side coverage is implied by the same MWP-1/2 test suite (fetch and clone share _walk_commit_delta).

Residual gap this phase closes:

No sub-ticket wrote a dedicated fetch/pull convergence test against staging — all direct evidence is clone-shaped (have=[]). fetch/pull with a non-empty have set exercise a materially different code path (delta walk, not cache HIT) that has not been the subject of an explicit E2E in this campaign.
pull = fetch + merge/ff — the merge/ff correctness after a fetch has never been explicitly re-tested post-MWP-8 (does a pull after a force-pushed remote branch behave correctly, or does it silently diverge?).
SC_03a — fetch convergence: two independent clones, one pushes, the other fetches then pulls; non-empty have set exercises the delta walk, not the cache HIT path. ✅ PASS — run against localhost.
SC_03b — pull after a force-push (interaction with MWP-8). ✅ PASS — gabriel explicitly approved via AskUserQuestion before running, run against a localhost scratch repo (not shared staging).

Recipe (updated to target localhost per gabriel's local-first priority — the original recipe targeted staging, corrected here to match what was actually run):

# SC_03a — fetch convergence: two clones, one pushes, other fetches
cd /tmp && rm -rf mwp-qa-local-v4-3a-x mwp-qa-local-v4-3a-y
muse clone https://localhost:1337/gabriel/mwp-qa-local-v4 mwp-qa-local-v4-3a-x --json
muse clone https://localhost:1337/gabriel/mwp-qa-local-v4 mwp-qa-local-v4-3a-y --json
cd /tmp/mwp-qa-local-v4-3a-x
echo "from-x" >> main_unique.txt && muse code add main_unique.txt
muse commit -m "from x" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push origin main --json
cd /tmp/mwp-qa-local-v4-3a-y
muse fetch origin --branch main --json    # non-empty `have` — exercises delta walk, not cache HIT
muse log --json | jq -r '.commits[0].message'   # should NOT yet show "from x" (fetch != merge)
muse pull origin main --json
cat main_unique.txt    # NOW should show "from-x" appended — proves pull = fetch + merge converges

# SC_03b — pull after a force-push (interaction with MWP-8)
# Approved by gabriel before running. New scratch repo, not the SC_03a one.
cd /tmp && rm -rf mwp-qa-force mwp-qa-force-y
mkdir mwp-qa-force && cd mwp-qa-force
muse init --json
echo "c1" > file.txt && muse code add file.txt
muse commit -m "c1" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse hub repo create --name mwp-qa-force --no-init --hub https://localhost:1337 --json
muse remote add local2 https://localhost:1337/gabriel/mwp-qa-force --json
muse push local2 main --json
echo "c2-original" >> file.txt && muse code add file.txt
muse commit -m "c2 original" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push local2 main --json
cd /tmp && muse clone https://localhost:1337/gabriel/mwp-qa-force mwp-qa-force-y --json   # y captures pre-force-push state
cd /tmp/mwp-qa-force
muse reset <c1-id> --json                          # soft reset, rewinds branch pointer only
echo "c1" > file.txt && echo "c2-REWRITTEN" >> file.txt && muse code add file.txt
muse commit -m "c2 rewritten (force-push scenario)" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push local2 main --force --json               # rewrites remote history
cd /tmp/mwp-qa-force-y
muse pull origin main --json                        # y still has the OLD c2 locally
muse conflicts --json                               # inspect what pull actually did

Result — SC_03a: ✅ PASS. muse fetch origin --branch main sent want=1 have=6 (non-empty have, exercising the delta walk / the have_oids fix, not the cache-HIT path). Fetch alone left y's working tree and local HEAD unchanged (main_unique.txt still showed the old content, muse log still showed y's own last commit) — confirming fetch ≠ merge. muse pull then correctly fast-forwarded (status: "fast_forward", commits_received: 0, blobs_written: 0 since the prior fetch already had everything) and the working tree converged to include x's pushed content.

Result — SC_03b: ✅ PASS, gabriel's explicit approval obtained via AskUserQuestion before running (recorded here per the safety note), run against a fresh localhost scratch repo (gabriel/mwp-qa-force, not the SC_03a repo, not shared staging). Sequence: pushed c1→c2-original; cloned y (captures pre-force-push state); on the original clone, soft-reset back to c1 and committed a genuinely different c2-rewritten, then force-pushed. Pulling into y (which still had the old c2-original locally) produced:

{"status": "conflict", "exit_code": 1, "commits_received": 1, "blobs_written": 1,
 "head": null, "conflict_paths": ["file.txt"]}

muse conflicts --json confirmed theirs_commit was the genuinely new force-pushed commit (not a stale cached one — MWP-8's cache invalidation correctly delivered the real new content) and ours_commit/base_commit correctly identified y's own history and the true common ancestor. Pull did not silently pick a side, did not corrupt data, and did not serve stale content — it safely halted with a surfaced conflict, exactly matching this workspace's own "never blindly --ours/--theirs" convention. The merge was left unresolved intentionally (the test's purpose — proving no silent divergence and no stale-cache interaction — was already fully served); the scratch directories are disposable and were not committed to anything.

Exit gate — Phase 3 fully closed. Both SC_03a and SC_03b pass.

Phase 4 — Claim 4: repeatable with no stale clones, no phantom large uploads, no manual cache-busting

Fixed by: all of MWP-1..MWP-8 in combination; MWP-8 specifically removes the last known manual-cache-bust scenario (repair).

Already proven: each sub-ticket proves its own slice is real-time self-healing (no UPDATE musehub_fetch_mpack_cache ... needed).

Residual gap this phase closes (the important one):

No test runs the full loop N times in sequence (push, clone, push, clone, repair, clone, force-push, clone) and asserts zero manual intervention across the whole sequence. Each MWP proved its own trigger in isolation; nothing proves the composition. This phase depends on Phases 1–3 already being individually green (see Design above).
"No phantom large uploads" has only been checked at the byte-count level for two-branch topologies (MWP-7). A repeated-push loop (this phase) should also confirm upload size stays flat/small across many iterations, not just correct on push #2.
SC_04a — the full repeated loop, one scratch repo, 10 iterations: objects_sent/blobs_sent stay flat (not growing) per iteration, and every clone's tail content matches the latest push with zero manual intervention. ✅ PASS — run against localhost.

Recipe (retargeted to localhost per gabriel's local-first priority):

# SC_04a — the full repeated loop, one repo, ~10 iterations, watch for growth
cd /tmp && rm -rf mwp-qa-4a && mkdir mwp-qa-4a && cd mwp-qa-4a
muse init --json
muse hub repo create --name mwp-qa-4a --no-init --hub https://localhost:1337 --json
muse remote add local2 https://localhost:1337/gabriel/mwp-qa-4a --json
for i in $(seq 1 10); do
  echo "iteration $i" >> log.txt
  muse code add log.txt
  muse commit -m "iter $i" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
  push_out=$(muse push local2 main --json)
  echo "$push_out" | jq '{i: '"$i"', commits_sent, objects_sent, force}'
  rm -rf /tmp/mwp-qa-4a-clone-$i
  muse clone https://localhost:1337/gabriel/mwp-qa-4a /tmp/mwp-qa-4a-clone-$i --json
  tail -1 /tmp/mwp-qa-4a-clone-$i/log.txt   # must equal "iteration $i" every time, no staleness
done

Result: ✅ PASS, all 10 iterations. objects_sent stayed exactly 1 on every single iteration (no growth trend at all — not just "small," flat). commits_received on each clone correctly accumulated (1, 2, 3, ... 10) — every clone always got the complete history, never a truncated one. Every one of the 10 clones' tail content matched exactly ("iteration 1" through "iteration 10", zero mismatches, zero staleness). Zero manual cache-busting, zero retries needed, zero intervention across the entire loop. exit_code: 0 on every push and every clone.

Exit gate: SC_04a passes across all 10 iterations with flat upload size and zero staleness — confirmed, no growth trend or stale-tail iteration found. Phase 4 fully closed.

Phase 5 — Cross-cutting seams between MWP fixes

These are the interaction risks that only make sense to chase once each MWP's own surface is confirmed working in isolation (Phases 1–4 green). This is the "deeper than surface" part of the review — the reason re-reading each ticket's own tests is not sufficient.

SC_05_01 — MWP-3/4 (job ordering/idempotency) × MWP-8 (cache invalidation enqueues its own prebuild job). _enqueue_repair_prebuild calls the same enqueue_job that MWP-3 hardened for push-triggered prebuilds. Check: has the repair-triggered enqueue path been checked against MWP-3's dedup-by-mpack_key logic? A repair followed immediately by a push could enqueue two prebuild jobs — does MWP-3's fix correctly coalesce them, or does one get silently dropped (regressing RC-3 in a new trigger)?

✅ VERIFIED CORRECT BY DESIGN — checked by direct code inspection, not inference. musehub_jobs.py:172-174: fetch.mpack.prebuild is explicitly excluded from the per-mpack_key dedup (coarse_jobs = [... if jt != "mpack.index"]) — only mpack.index gets that treatment. fetch.mpack.prebuild intentionally uses the coarse (repo_id, job_type, pending) key, because process_fetch_mpack_prebuild_job's own docstring confirms it's "self-coalescing": it re-reads live MusehubBranch tips at run time rather than trusting its enqueue-time payload (payload["tip_commit_ids"] is "retained only for observability"). A repair's _enqueue_repair_prebuild call colliding with an existing pending push-triggered prebuild is not a bug — whichever single job survives dedup still rebuilds against current (already-repaired) state when it eventually runs, because the repair's cache invalidation is already durably committed before the enqueue call happens. No RC-3 regression possible here.
SC_05_02 — MWP-1/2 (generation authority) × MWP-7 (sibling-ref dedup). MWP-7's dedup subtracts a sibling tip's full manifest. If that sibling's own generation was ever backfilled by MWP-1's on-demand repair, does the manifest lookup still resolve correctly, or could a repaired-generation commit be mis-scoped out of a dedup calculation?

✅ VERIFIED — architecturally independent, checked by direct code inspection. muse/cli/commands/push.py:830-838: the client's sibling-ref dedup boundary (branch_have) is built entirely from remote_branch_heads (actual commit IDs fetched via GET /refs) and the client's own local walk_commits() — a parent-chain walk over the client's local repo, using no server-side generation data whatsoever. MusehubCommitGraph.generation is a purely server-side fetch-path optimization (used only in _walk_commit_delta's range scan); push-side dedup never reads it. These two features share no state and cannot interact.
SC_05_03 — MWP-5 (client retry) × MWP-4 (prebuild ordering). MWP-5's bounded retry budget (120s default) was tuned against MWP-4's fixed ordering guarantee. If a repo is large enough that index+prebuild together exceed 120s, does the client fail with a clear, actionable error, or a confusing timeout?

✅ VERIFIED — real evidence already collected earlier in this investigation, not simulated for this check. During the musehub#113 investigation, a genuinely stale/stuck server produced exactly this 120s-exhaustion condition (four retries, 30s apart). The client's actual output: ⏳ remote preparing fetch data (server busy, attempt 4); retrying in 30s … (waited 90s / 120s budget) followed by ❌ Remote still preparing clone data after 120s — try again shortly., exit_code: 3, and the JSON envelope carries "retryable": true. This is a clear, actionable message with a machine-readable retry hint — not a confusing raw timeout or stack trace, regardless of why the server took too long.
SC_05_04 — Real R2 vs local MinIO. Several sub-tickets' "best-effort R2 delete" paths (MWP-8's invalidate_fetch_mpack_cache) were only tested against a mocked backend. Staging uses real Cloudflare R2. Confirm the delete call shape matches R2's actual API (auth headers, error codes on 404-already-gone) rather than the mock's simplified contract.

Explicitly deferred, not skipped — written reason: this specifically requires real R2 access to verify, which is out of scope while local-first is the explicit priority (no staging until local is airtight — gabriel's direction). Partial risk assessment done: both MinIO and R2 are accessed through the identical boto3 S3-compatible client code path (only the endpoint/credentials config differs), so the DELETE call shape itself is not expected to diverge — the open risk is specifically R2's response behavior on a delete-of-already-deleted object (a 404), which cannot be confirmed without hitting real R2. Revisit when this investigation resumes against staging.
SC_05_05 — Concurrent agents on the same repo. aaronrene will not be the only person pushing to shared repos. No MWP ticket tested two different client machines (not just two CLI invocations from one machine) pushing to the same branch concurrently. This is the actual onboarding scenario.

✅ VERIFIED — actually run, not inferred. Two independent clones (gabriel/mwp-qa-race, simulating two agents), each committing independently from the same base, both pushes fired via genuine shell backgrounding ((...&); (...&); wait) to guarantee real OS-level overlap, not sequential back-to-back calls. Result: one push landed cleanly (commits_sent: 1, objects_sent: 1, exit 0); the other was correctly rejected by the atomic CAS branch-pointer update with a clean 409 diverged error ("push rejected: ... is not a fast-forward of ... — use --force to override") — no data loss (the losing commit is still safely in its own local repo, recoverable via pull+rebase), no corruption, no silent overwrite, no crash. One honest caveat: this was two truly concurrent processes from one physical machine, not two literal separate machines — but the server cannot distinguish the two cases (HTTP requests are indistinguishable by origin machine), so this is a faithful test of the actual concurrency-safety mechanism.

Exit gate: each of SC_05_01..SC_05_05 is either explicitly checked (with evidence) or explicitly deferred with a written reason — no silent skips. 4 of 5 checked with direct evidence (code inspection or live test); 1 explicitly deferred pending staging/R2 access, with the specific open risk narrowed down. Phase 5 exit gate satisfied.

Phase 6 — Cleanup & sign-off

SC_06_01 — all scratch repos (gabriel/mwp-qa-*) cleaned up / deleted from staging. ✅ DONE. Five staging scratch repos existed (mwp-qa-1a, mwp-qa-atomic-1..4, all created before the local-first pivot — everything from Phase 2 onward ran against localhost instead, so no further staging scratch repos accumulated). Before deleting, gabriel requested a full precautionary backup given this session's own findings about subtle scoping bugs: cascade-delete FK scoping for issues/proposals/mists was independently verified safe by code inspection (all FK to repo_id with correct ondelete="CASCADE"; mists are structurally independent repos, unreachable by another repo's delete), AND a full pg_dump of the staging DB was taken via SSM, uploaded to a temporary private/encrypted S3 bucket, downloaded to gabriel's Desktop, verified as a complete valid dump (correct pg_dump header/trailer, 69 COPY statements), before all temporary cloud artifacts were cleaned up. Gabriel then independently confirmed via the staging UI that none of the 5 repos had any attached proposals or issues. All 5 repos deleted (muse hub repo delete --yes), each returning deleted: true. Post-deletion hub repo list confirmed only muse, musehub, muse-zsh, identity, and mist repos remain — nothing else touched.
SC_06_02 — findings from every phase written back into this file (this file is the permanent record, not a scratch pad — update it as phases complete, per the workspace body-copy-quality convention). ✅ DONE — every phase (0 through 5) has its results, evidence, and any deferred items recorded inline as they completed, pushed to staging muse#63 after each phase.
SC_06_03 — any new bug discovered during any phase filed as its own musehub/muse ticket (not silently patched in place). ✅ DONE — musehub#113 (the silent partial clone / multi-tip manifest union bug, root-caused and fixed with two TDD'd commits), musehub#114 (dead process_mpack_gc_job/purge_stale_mpack_index_entries code, correction history included), musehub#115 (Docker container restart/rebuild guidance for the agent-config docs).
SC_06_04 — gabriel gives explicit go-ahead to send aaronrene the staging URL. Pending — this is the final sign-off step; not yet given.

Exit gate: all of Phase 0–5's exit gates are satisfied — confirmed. This issue closes only after gabriel's explicit sign-off in SC_06_04.

Safety notes for this QA pass

All QA repos are scratch repos under gabriel/mwp-qa-* on staging — never reuse gabriel/muse or gabriel/musehub themselves for destructive operations (force-push, reset --hard, repair-*).
Per workspace hard rules: no --force flag, no reset --hard, no checkout --ours/--theirs without gabriel's explicit go-ahead in the conversation, even inside a scratch repo — ask first if a recipe step needs one. This specifically gates SC_03b.
Every /tmp directory created here is disposable; clean up with a plain rm -rf when done (not a muse operation, so no special permission needed).
If any staging repair endpoint needs to be exercised (Phase 1's repair × clone interaction, or SC_05_01), coordinate the exact repo/commit with gabriel before running it — repair mutates content-addressed rows in place.

Acceptance criteria

Phase 0 — wire suite green in both repos, verified fresh (not from memory of a prior run). ✅ 56 (musehub) + 21 (muse) passed, 0 failed.
Phase 1 — SC_01a, SC_01b pass against real staging. ✅ Content correctness proven; 503/retry-path timing not exercised (repo too small) — tracked as a follow-up, not blocking.
Phase 2 — SC_02a ✅, SC_02b ✅ (dedup confirmed; the blocking bug it surfaced is now root-caused and fixed via musehub#113 — two fixes, all_oids and have_oids, both TDD'd and verified), SC_02c ✅ (rebase dry-run parity, exact match). Fully green.
Phase 3 — SC_03a ✅ (fetch ≠ merge, pull converges via fast-forward). SC_03b ✅ (gabriel's sign-off obtained; pull after force-push safely surfaces a conflict, no silent divergence, no stale cache content). Fully green.
Phase 4 — SC_04a (10-iteration loop) shows flat upload size (exactly 1 object/iteration, no growth), zero staleness (10/10 clone tails matched exactly). Fully green.
Phase 5 — all 5 cross-cutting seams checked or explicitly deferred. 4/5 verified with direct evidence (SC_05_01 code inspection, SC_05_02 code inspection, SC_05_03 real prior evidence, SC_05_05 live concurrent-push test); SC_05_04 explicitly deferred pending real R2/staging access. Fully green.
Phase 6 — scratch repos cleaned up ✅ (5 staging repos deleted, backup taken first, cascade-scoping verified safe), findings recorded ✅, new bugs filed as their own tickets ✅ (musehub#113/114/115). Only SC_06_04 (gabriel's final go-ahead to send aaronrene the staging URL) remains before this ticket can close.

Out of scope

Re-litigating whether each MWP-1..MWP-8 sub-ticket's own unit tests are well-designed — they are; this ticket only adds live-staging, black-box, and cross-cutting evidence on top.
Any new correctness fix discovered here is filed as its own ticket, not patched inline as part of this spot-check (see SC_06_03).
Load/perf testing beyond the 10-iteration loop in Phase 4 — this ticket proves correctness under realistic single-developer usage, not high-concurrency stress (that would be a separate ticket if SC_05_05 surfaces a real gap).

◎ Activity

●

gabriel opened this issue 15 days ago

◎

No activity yet. Use the CLI to comment.

Assignee

○ unassigned

Release

no commits linked to this issue

create

muse hub issue create \
  --title "..." \
  --body "..." \
  --label bug \
  --anchor path/to/file.py::Symbol \
  --commit-anchor <sha> \
  --repo gabriel/muse

read

muse hub issue get 63 --json
muse hub issue list --state open --json

update

muse hub issue edit 63 \
  --anchor path/to/file.py::Symbol \
  --repo gabriel/muse

comment

muse hub issue comment 63 \
  --body "Fixed in <sha>" \
  --repo gabriel/muse

muse hub issue close 63 \
  --repo gabriel/muse

create

create_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  title: "...",
  body: "...",
  labels: ["bug"],
  symbol_anchors: [
    "path/to/file.py::Symbol"
  ],
  commit_anchors: ["<sha>"]
})

read

get_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 63
})

list_issues({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  state: "open"
})

update

edit_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 63,
  symbol_anchors: ["path/to/file.py::Symbol"]
})

comment

create_issue_comment({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 63,
  body: "..."
})

close_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 63
})

create

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{
    "title": "...",
    "body": "...",
    "labels": ["bug"],
    "symbol_anchors": ["path/to/file.py::Symbol"]
  }'

read

# get one issue
curl http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/63

# list open issues
curl "http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues?state=open"

update

curl -X PATCH \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/63 \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"title": "...", "body": "..."}'

comment

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/63/comments \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"body": "Fixed in <sha>"}'

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/63/close \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\""

Spot check #58: deep verification of the wire protocol MVP goal-of-done before aaronrene onboards

MWP-9 — Spot check #58: deep verification of the wire protocol MVP goal-of-done

Background

Goal — definition of MVP done (verbatim from #58)

Goal — definition of done for this ticket

Design — why this phase order

Phases

Phase 0 — Baseline: wire suite green, right now, post-merge

Phase 1 — Claim 1: muse clone is always complete and correct, including immediately after a push

Phase 2 — Claim 2: muse push uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes

Phase 3 — Claim 3: muse fetch / muse pull receive exactly the new commits/blobs and converge

Phase 4 — Claim 4: repeatable with no stale clones, no phantom large uploads, no manual cache-busting

Phase 5 — Cross-cutting seams between MWP fixes

Phase 6 — Cleanup & sign-off

Safety notes for this QA pass

Acceptance criteria

Out of scope

Phase 1 — Claim 1: `muse clone` is always complete and correct, including immediately after a push

Phase 2 — Claim 2: `muse push` uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes

Phase 3 — Claim 3: `muse fetch` / `muse pull` receive exactly the new commits/blobs and converge