Muse Wire Protocol — MVP hardening (master tracking issue)

Background

The Muse Wire Protocol (MWP) — clone, push, fetch, pull — is the single hardest, most time-consuming subsystem in the workspace. It is functionally equivalent to git smart-HTTP, rebuilt from scratch on an mpack packfile format with content-addressed objects, presigned R2/MinIO uploads, and an async server-side prebuild/index pipeline.

It mostly works. The blocker to declaring it MVP-ready — so aaronrene can clone/push/pull/fetch with confidence and start picking up tickets — is a small number of correctness gaps in the prebuild/index/negotiation pipeline plus a largely-disabled test suite. This issue is the result of a full read-through of both sides of the protocol. It records the architecture, the confirmed root causes (grounded in file:line), and the proposed sub-tickets.

This is a hardening pass, not a rewrite. The architecture is sound. The fixes are targeted.

Proximate triggers

1. Clone-after-push staleness — push a repo, clone it (works), push another commit or two, clone again → the new commits/blobs are missing from the clone, yet clone exits 0.
2. #55 — push have-negotiation ignored sibling refs, causing redundant multi-hundred-MB uploads and broken pipes when a branch lags a sibling already on the remote.

Architecture recap (for onboarding)

All four commands move data as a single binary mpack (b"MUSE" magic, section table, 32-byte SHA-256 footer). Objects are content-addressed: sha256:<hex> is the integrity check.

Server push pipeline (musehub_wire_push.py::wire_push_unpack_mpack): reads mpack from R2 → inserts objects/snapshots/commits → computes MusehubCommitGraph generations (step 9, synchronous) → advances branch pointer (step 10, atomic CAS) → enqueues async jobs (step 11): mpack.index, fetch.mpack.prebuild, intel.*, gc.

Server fetch pipeline (musehub_wire_fetch.py::wire_fetch_mpack): on have=[] (clone), looks up musehub_fetch_mpack_cache — HIT requires every want tip cached AND all pointing to one mpack_id; MISS raises MPackNotReadyError (HTTP 503 + Retry-After). Delta builds walk the commit graph via _walk_commit_delta.

Prebuild (process_fetch_mpack_prebuild_job): after each push, builds one combined mpack covering all branch tips and writes a cache row per tip → next clone is a sub-second cache HIT.

Confirmed root causes

RC-1 — Generation fallback to 0 silently corrupts the fetch walk (CRITICAL — the clone staleness bug)

musehub_wire_push.py:926:

_gen = (max(_parent_gens) + 1) if _parent_gens else 0

When a pushed commit's parent generation cannot be resolved — the parent is not in this mpack (it is on the remote, excluded by the have set) and its MusehubCommitGraph row is not returned by the lookup at musehub_wire_push.py:870-874 — the new commit is written with generation 0 instead of parent_gen + 1.

That single wrong generation poisons the fetch-side walk. _walk_commit_delta (musehub_wire_fetch.py:102-111) is a generation-bounded range scan:

.where(MusehubCommitGraph.generation > min_have_gen)
.where(MusehubCommitGraph.generation <= max_want_gen)

With the new tip at gen 0, max_want_gen = 0, so the scan excludes every real ancestor (gens 1..N). The BFS truncates, and the tip-snapshot resolver (musehub_wire_fetch.py:484-490, ORDER BY generation DESC LIMIT 1) selects an older commit's snapshot. The assembled mpack therefore carries the old tip's blobs but is cached against the new tip id. Clone returns 200 with a working tree missing the latest commits — exactly the reported symptom.

The fetch code already half-knows this — see the [BLOB-DEBUG] block at musehub_wire_fetch.py:495-509: "A mismatch means CommitGraph is stale for the newest push — the old snapshot's manifest_blob would be missing new blobs."

RC-2 — _walk_commit_delta has no fallback when the graph is inconsistent

musehub_wire_fetch.py:66 — if True: # fast path always active — makes the real parent-pointer DAG walk (the "legacy fallback" at lines 159-184) dead code. The generation range-scan is the only path. There is no safety net when generations are wrong or missing (RC-1), so a single bad row produces a silently truncated mpack instead of a correct (if slower) walk.

RC-3 — Job idempotency drops fresh payloads on back-to-back pushes

musehub_jobs.py:72-84 (enqueue_job) and :158-164 (enqueue_push_intel) dedup on (repo_id, job_type, status="pending"). If a fetch.mpack.prebuild (or mpack.index) job from push A is still pending when push B lands, push B's job is skipped. The pending job still carries push A's payload / tip set, so push B's new tip never gets a prebuild or index until yet another push displaces it. Rapid pushes ⇒ permanently stale clone cache.

RC-4 — Prebuild can run before index (foundation-job ordering race)

musehub_jobs.py:170-189: fetch.mpack.prebuild and mpack.index are both _FOUNDATION_TYPES and share the same created_at. claim_next_job (musehub_jobs.py:206-231) orders by created_at with no tiebreaker, so the prebuild may run first. wire_fetch_mpack then hits the index-coverage check (musehub_wire_fetch.py:580-593), raises FetchNotIndexedError, the prebuild swallows it (musehub_wire_fetch.py:1041), and no cache row is written. The clone then 503s until a retry/another push.

RC-5 — Client does not retry on 503 (clone-immediately-after-push fails)

The fetch route correctly returns 503 + Retry-After while the prebuild/index is in flight (wire.py:694-705). But the client (transport.py::fetch_mpack) raises TransportError(503), and clone.py:454-464 catches it, rmtrees the target, and exits INTERNAL_ERROR. There is no honor of Retry-After, no poll loop. A clone issued seconds after a push fails hard even though the server self-heals within a second or two.

RC-6 — The wire test suite is largely disabled ✅ Fixed by MWP-6

~19 wire/mpack test files are skipped, 11 with reason="muse wire protocol in flux" — including test_issue_61_clone_after_push.py, the exact clone-after-push contract. The protocol's own regression net is off, which is why these regressions reach manual testing. MVP cannot be declared with the suite dark.

Fixed: MWP-6 revived all flux-skipped wire files across 6 phases. 55 tests under pytest -m wire now run green (47 in-process + 8 slow subprocess E2E). The MWP6_00 gate (test_mwp6_wire_suite_enabled.py) enforces no-flux-skip forever.

RC-8 — Repair endpoints and force-push never invalidate musehub_fetch_mpack_cache

wire_repair_object, wire_repair_snapshot, and wire_repair_commit (musehub_wire_push.py) each end with await session.commit() → return {"repaired": True} and never touch musehub_fetch_mpack_cache. Likewise, wire_push_unpack_mpack with force=True advances the branch pointer and enqueues intel jobs but never deletes old-tip cache rows.

The cache is keyed by (repo_id, tip_commit_id) with a 7-day TTL. After a repair, the prebuilt mpack cached against the tip still carries the pre-repair bytes — the repair fixed the DB row but the fetch path presigns whatever mpack_id the cache row addresses and exits 0. A clone issued within 7 days of the repair receives corrupted content.

This was the mechanism behind the gabriel/muse rc10 incident: two main-line merge commits had their signer_public_key field stamped by the rc10 migration. repair-commit fixed the musehub_commits rows — but staging clones stayed broken because the prebuilt mpack kept being served. The manual workaround was:

UPDATE musehub_fetch_mpack_cache SET expires_at = NOW() - INTERVAL '1 second' WHERE repo_id = ...

A correct fix requires active invalidation in the same DB transaction as the repair, so a cache row can never outlive its corresponding object/snapshot/commit content.

RC-7 — #55 (sibling-ref push negotiation): main path fixed, needs verification + dry-run alignment

The live-push path already uses all sibling refs as the walk boundary (push.py:819-822, branch_have = remote_branch_heads.values()), and blob dedup subtracts the sibling tip's full manifest (mpack.py:330-336, 362). Client tests exist (tests/test_push_branch_have.py, tests/test_push_have_filter.py). What remains: (a) confirm #55 is actually resolved end-to-end against staging and close it, and (b) the dry-run path still scopes have to a single branch (push.py:668-674) — this is #32 and should be aligned so --dry-run counts match the real push.

Goal — definition of MVP done

aaronrene (or any dev) can, against staging, reliably:

1. muse clone <url> — get a complete, correct working tree every time, including immediately after a push (with bounded client retry on 503).
2. muse push — upload only the true delta; never re-upload objects already on the remote via any ref; survive back-to-back pushes.
3. muse fetch / muse pull — receive exactly the new commits/blobs and converge.
4. Do all of the above repeatedly with no stale clones, no phantom large uploads, no manual cache-busting.
5. The wire regression suite is re-enabled and green as the proof.

Proposed sub-tickets

Create as needed; each is independently shippable. Repo target in brackets.

• MWP-1 [musehub] — Make commit-graph generations authoritative (fixes RC-1). ✅ COMPLETE → filed as musehub#106 (closed): https://staging.musehub.ai/gabriel/musehub/issues/106 Never write generation 0 for a commit that has parents. If a parent generation is unresolved at push time, resolve it (recursive lookup / on-demand backfill) or fail the push loudly. Add an invariant check + regression test for the push-child-of-remote-commit case. Shipped in 6 phases (13 tests, tests/test_mwp1_generation_authority.py): _resolve_generation_with_backfill (on-demand bottom-up backfill from musehub_commits), repair_corrupt_commit_generations (idempotent repair of existing artefacts), check_commit_graph_invariant (no gen-0-with-parents), the [MWP1_13] fetch-side repair guard, and the MWP1_14 end-to-end push→clone→push→clone regression gate. Merged to dev + main (sha256:da49a05fd62cda…).
• MWP-2 [musehub] — Add a correctness fallback to _walk_commit_delta (fixes RC-2). ✅ CLOSED → musehub#107 closed: https://staging.musehub.ai/gabriel/musehub/issues/107 When the generation range-scan truncates (a wrong/missing generation makes the BFS dead-end), fall back to the authoritative MusehubCommit.parent_ids DAG walk so the returned commit set is never incomplete. Removed the if True: dead-code guard (line 66); promoted the legacy walk to a tested helper (_walk_commit_delta_dag) with Kahn's topo-sort (deque.popleft(), O(n)) and max_nodes=100_000 bound. 23 tests GREEN (10 MWP2 + 13 MWP1). All 6 acceptance criteria met. MWP2_12 (acceptance gate) proves fallback alone ships a complete clone even with repair patched out.
• MWP-3 [musehub] — Fix job enqueue for back-to-back pushes (fixes RC-3). ✅ COMPLETE → musehub#108 (closed): https://staging.musehub.ai/gabriel/musehub/issues/108 fetch.mpack.prebuild / mpack.index must always reflect the latest tip set. Two-face fix shipped: (1) process_fetch_mpack_prebuild_job re-reads live MusehubBranch tips at runtime — coarse dedup becomes self-coalescing; (2) enqueue_job deduped per mpack_key via JSONB predicate — distinct mpacks are non-substitutable work and can never be dropped by a coarse pending-job check. RC-4 ordering is explicitly out of scope (→ MWP-4). 36 tests GREEN across 5 suites (14 MWP-3 + adjacent regressions fixed). MWP3_10 (two-push full drain) + MWP3_11 (N=10 stress partial drain) prove E2E correctness end-to-end.
• MWP-4 [musehub] — Order prebuild strictly after index (fixes RC-4). ✅ COMPLETE → musehub#109 (closed): https://staging.musehub.ai/gabriel/musehub/issues/109 Three-layer fix shipped in 5 phases (12 tests, tests/test_mwp4_prebuild_ordering.py): (1) created_at stagger — prebuild gets t+1s, index stays at t+0s (defense-in-depth); (2) correlated NOT EXISTS barrier in claim_next_job — prebuild un-claimable while any mpack.index for the same repo_id is pending/running (load-bearing); (3) narrow except FetchNotIndexedError: raise in process_fetch_mpack_prebuild_job — index gaps surface as retryable fail_job failures instead of silent empty-cache. MWP4_11 (E2E claim loop) + MWP4_12 (N=10 stress) prove zero FetchNotIndexedError and a correct cache at the latest tip after any burst of back-to-back pushes. Commit range: sha256:2c523da453513 → sha256:46854101acdde on task/mwp-4-prebuild-after-index.
• MWP-5 [muse] — Clone/fetch honor Retry-After with a bounded poll (fixes RC-5). ✅ COMPLETE → muse#59 (closed): https://staging.musehub.ai/gabriel/muse/issues/59 Extracted HttpTransport._fetch_mpack_once; added bounded retry loop in fetch_mpack tracking total_waited against a configurable budget (default 120s). `--retry-timeout` / `--no-retry` flags on `clone`, `fetch`, `pull`. Terminal 503 message improved; JSON envelope carries `retryable: true`. 19 tests GREEN (MWP5_00–MWP5_18 including real-urllib E2E on a live localhost server). Zero regressions across adjacent suites.
• MWP-6 [musehub + muse] — Re-enable the wire regression suite (fixes RC-6). Un-skip test_issue_61_clone_after_push.py and the ..._flux mpack/wire tests; fix or rewrite each against the current protocol; gate MVP on green.
• MWP-7 [muse] — Verify and close #55; align --dry-run negotiation (RC-7 / #32). ✅ COMPLETE → muse#62 (closed): https://staging.musehub.ai/gabriel/muse/issues/62 Five-phase E2E + TDD campaign on task/mwp-7-sibling-negotiation. Phase 1/2 (GT/LF): 9 unit tests confirm the live-push sibling-ref negotiation path is correct (push.py:819-822, branch_have = remote_branch_heads.values()). Phase 3 (VS): real staging push with a 17-commit dev + 1-commit main topology — commits_sent=1, objects_sent=1; clone verified integrity all_ok:true. #55 closed. Phase 4 (DR): fixed dry-run have-set: replaced get_remote_head(push_branch) with _all_known_have_anchors(root) (all sibling tracking refs) — matches live-path logic. Phase 5 (RG): removed c != local_head filter so up-to-date dry-runs report 0 correctly. 21 tests GREEN (pytest -m wire); 85 adjacent push-corpus tests GREEN. #32 closed. Commit: sha256:6ec37c93009f7c30451ea8db62a0a1b764bb79fa93231b5506bb6855251b3e5d.
• MWP-8 [musehub] — Cache invalidation for repair and force-push (fixes RC-8). ✅ COMPLETE → musehub#111 (closed): https://staging.musehub.ai/gabriel/musehub/issues/111 Five-phase TDD campaign on task/mwp-8-cache-invalidation. Phase 0 (AU): proved the stale-HIT gap — AU_01..AU_05 go RED before any fix. Phase 1 (IV): added invalidate_fetch_mpack_cache(session, repo_id, tips=None) to musehub_gc.py; deletes rows AND best-effort-deletes their R2 objects regardless of TTL; tips=None → repo-wide (repair path), tips=[...] → tip-scoped (force-push). Does NOT commit — caller owns the transaction (D2). 6 unit tests GREEN (IV_01..IV_06). Phase 2 (RP): wired invalidation into wire_repair_object, wire_repair_snapshot, wire_repair_commit; each calls invalidate_fetch_mpack_cache then _enqueue_repair_prebuild in the same transaction before session.commit(). 4 contract tests GREEN (RP_04..RP_07 incl. atomicity rollback guard). Phase 3 (FP): wired tip-scoped invalidation into wire_push_unpack_mpack force-push path; only old+new tips busted — sibling cache rows survive (D1/D3 design decisions). 4 tests GREEN (FP_02..FP_05 incl. blast-radius and prebuild-still-enqueued proofs). Phase 4 (EX): added _enqueue_repair_prebuild helper; verified cache-bust logged via caplog; EX_02 is the E2E narrative (@pytest.mark.wire): repair → MISS → prebuild → HIT serves fresh content. 3 tests GREEN (EX_01..EX_03). Phase 5 (RG): 54 adjacent corpus tests GREEN; 16 MWP-8 tests collected + GREEN; master tracker (muse#58) updated; musehub#111 closed with audit verdict. 16 tests total in tests/test_mwp8_cache_invalidation.py.

Acceptance criteria

• [x] Push C1→C2→C3, clone, push C4, clone again → second clone contains C4 and its blobs (RC-1, RC-2). Codified as a non-skipped regression test. ✅ Fixed by MWP-1 + MWP-2
• [x] Two pushes within one prebuild interval both end with a correct, fresh clone cache (RC-3 ✅ fixed by MWP-3, RC-4 ✅ fixed by MWP-4).
• [x] muse clone issued immediately after muse push succeeds via bounded retry, never hard-fails on the first 503 (RC-5). ✅ Fixed by MWP-5 (muse#59)
• [x] Pushing main when it lags an already-pushed dev sends only the delta (one merge commit, ~zero new blobs); #55 closed with evidence (RC-7). ✅ Fixed by MWP-7
• [x] --dry-run commit/object counts equal the real push counts (#32). ✅ Fixed by MWP-7
• [x] ✅ The previously-skipped wire/mpack test suite is re-enabled and green (RC-6). Fixed by MWP-6.
• [x] A repaired object/snapshot/commit is never masked by a stale prebuilt mpack — cache invalidated in the same transaction as the repair; clone issued immediately after repair returns the corrected content (RC-8). ✅ Fixed by MWP-8 (musehub#111)

Out of scope

• Any redesign of the mpack binary format or the presigned-upload model — both are sound.
• Per-object presign / fetch/objects fallback paths beyond what clone/fetch need for MVP.
• Shallow-clone (--depth) hardening beyond current behavior.
• Code-intel / structural-intel jobs (independent of correctness of the wire data path).