Silent partial clone: required blob missing from working tree, clone exits 0, verify reports all_ok: true

Parent: muse#63 (MWP-9 — spot check of #58's goal-of-done). Discovered during Phase 2 (SC_02b, three-sibling-branch dedup) against the scratch repo gabriel/mwp-qa-1a on staging. Status: investigation, not yet root-caused. No production code has been touched. This ticket is a brain dump of everything observed plus the open questions that block a fix. The plan is to reproduce at the most atomic level possible, narrowing one variable at a time, until we hit either a confirmed root cause or a wall that tells us exactly what to instrument next.

---

Background — what we were actually testing

Phase 2 of MWP-9 was verifying Claim 2 of #58 ("push uploads only the true delta, never re-uploads objects already on the remote via any ref"). The specific sub-test (SC_02b) was: push two sibling branches (feature-a, feature-b) off a shared main, and confirm the second branch's push dedups correctly against the first.

That part passed, and passed more thoroughly than planned: feature-b's push sent exactly 1 object (its own fb.txt blob), correctly excluding README.md's blob despite 4 incremental versions of it existing across the shared ancestor chain with both main and feature-a. RC-7 / MWP-7's sibling-ref dedup fix holds up under a stronger test than the original two-branch case it shipped with.

What broke instead was cloning the branch we'd just proven the push logic handled correctly.

---

What we know (chronological brain dump)

1. The exact repro sequence

All against https://staging.musehub.ai/gabriel/mwp-qa-1a (scratch repo, created fresh for this QA campaign):

commit 1 (main): "init"       — README.md = "hello"
commit 2 (main): "v2"         — README.md = "hello\nv2"
commit 3 (main): "a"          — README.md = "hello\nv2\na"
commit 4 (main): "b"          — README.md = "hello\nv2\na\nb"
  push staging main   → commits_sent=1, objects_sent=1 (per commit, each pushed individually as they were made)
branch feature-a from main tip (commit 4)
commit 5 (feature-a): "feature a"  — adds fa.txt = "fa\n"
  push staging feature-a → commits_sent=1, objects_sent=1  ✅ correct dedup
checkout main (back to commit 4), branch feature-b from main tip
commit 6 (feature-b): "feature b"  — adds fb.txt = "fb\n"
  push staging feature-b → commits_sent=1, objects_sent=1  ✅ correct dedup

rm -rf /tmp/mwp-qa-2b-clone-retry
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2b-clone-retry --branch feature-b --json

2. The failure, verbatim

[transport] unpackb  commits=6  snaps=6  blobs=2  t=0ms
[clone] apply_mpack DONE  t=12ms  blobs_written=2 blobs_skipped=0  commits_written=6
[mpack] fetch/mpack: 0.78s  blobs: 2  commits: 6
[clone] apply_manifest START  (working tree restore)
⚠️ clone: working tree partially restored — apply_manifest: 1 object(s) missing
   from the local store ('fb.txt'). The working tree cannot be fully restored.
   Fetch the missing objects from the remote before retrying.
[clone] apply_manifest DONE  t=7ms
{"...", "exit_code": 0, "status": "cloned", "commits_received": 6, "blobs_written": 2,
 "skipped_blobs": 0, "head": "sha256:005bc9e6...", "domain": "code", "dry_run": false}

fb.txt is completely absent from disk (confirmed via ls, not just empty). exit_code is 0. status is "cloned", not an error state. 100% reproducible — ran it twice in separate fresh directories, identical result both times.

3. Ruled out: manifest correctness, tool misuse, and (probably) content-store corruption

• The commit's manifest correctly maps fb.txt → sha256:b566176258f9d8.... Confirmed via muse hash-object fb.txt on the original working copy — the manifest's declared object ID is exactly right. This is not a metadata/manifest bug.
• --branch feature-b is the documented, correct clone flag (checked muse clone --help).
• Initial alarm: running muse cat-object sha256:<id> --json on 6 of the 13 objects in the local store returned SHA-256 integrity-check failures ("expected X, got Y") — this first read as store corruption. This was almost certainly a false alarm from misusing cat-object on non-blob objects. Running the actual sanctioned integrity tool, muse verify --json, on the same clone reports all_ok: true, failures: [], and critically promised_objects: 5 — i.e., those 6 "corrupt-looking" objects are very likely placeholder/stub entries for a lazy/partial object model (cat-object appears to hash the placeholder bytes directly rather than recognizing the stub and reporting it as not-yet-materialized, which is itself worth separate scrutiny — see open question 4 below — but is not the primary bug).

4. The actual anomaly: a consistent "2 eager blobs" ceiling regardless of target

Every single clone/fetch run during this QA session — regardless of which branch was requested or how many files that branch's manifest actually needed — reported exactly blobs=2 materialized eagerly, with the remainder tracked as promised_objects:

Clone target	Blobs actually needed by manifest	Blobs eagerly materialized	Promised
main (4 commits, README only)	1 (README latest)	2	— (not observed to fail, README happened to be eager)
feature-b (adds fb.txt)	2 (README latest + fb.txt)	2	5 — but fb.txt was among the promised, not the eager, set

This strongly suggests the server's combined-mpack / prebuild mechanism has a fixed or coincidentally-fixed eager-blob inclusion count, and nothing guarantees the blob(s) required by the specifically requested want tip's own manifest are always in that eager set. When they're not, the client's apply_manifest step correctly detects the gap and prints a warning — but the overall clone command still reports success.

5. muse verify --json output on the broken clone (full)

{
  "repo_id": "sha256:7720f84c823f423f1de706cd9dddefc2e82e98f3301eced36c35ddd8414abc5d",
  "refs_checked": 3,
  "commits_checked": 6,
  "snapshots_checked": 6,
  "objects_checked": 1,
  "signatures_checked": 6,
  "shallow_commits": 0,
  "promised_objects": 5,
  "is_shallow": false,
  "promisor_remotes": ["origin"],
  "all_ok": true,
  "nothing_checked": false,
  "check_objects": true,
  "strict": false,
  "branch": null,
  "failures": []
}

Notable: promisor_remotes: ["origin"] — but the remote we configured and pushed/cloned through was named staging, not origin. Either "origin" is a hardcoded/generic label unrelated to the actual remote name (cosmetic), or there's a mismatch worth checking (see open question 7).

all_ok: true despite feature-b's own checked-out HEAD requiring an object that is not present and is only "promised." This means verify's notion of "ok" does not currently check whether promised objects are needed by the actively-checked-out branch — it appears to only check structural consistency of commits/snapshots/signatures, not working-tree completeness.

6. Confirmed NOT the bug

• Sibling-ref dedup (RC-7/MWP-7): correct, verified independently — see §1, feature-b's push sent exactly 1 object.
• Manifest content: correct — fb.txt's declared object ID matches its real content hash.
• CLI flag usage: correct — --branch feature-b is the documented flag.

---

What we don't know — open questions to close before any fix is attempted

1. What determines the eager-blob cap? Is "2" a hardcoded constant, a config value, or an artifact specific to this session's exact push history shape (e.g., "however many new blobs were added since the last successful prebuild")? Needs a code-level answer, not just observed behavior.

2. Where does the "promised object" concept live in the codebase ✅ RESOLVED (see Code trace §1-2). ObjectState.PROMISED (muse/core/object_availability.py) is a generic, purely local, reactive label — not a decision point. build_mpack's promisor-skip logic (muse/core/mpack.py:781-794) is confirmed client-push-only, never invoked server-side. The actual server-side blob-selection logic is wire_fetch_mpack's own three-strategy resolution (musehub_wire_fetch.py:697-820), which silently drops any oid that fails all three — that is where the real decision (or non-decision) happens.

3. Is the promisor/lazy-object model intentional ✅ RESOLVED. Yes, intentional and documented — object_availability.py's own docstring explicitly describes it as mirroring git's promisor-remote concept for partial clones, with the opted-in-unless-opted-out default explained as the right choice for an agent-first, trusted-hub platform. It is not an accidental byproduct of the MWP campaign. The actual bug is not the existence of this model — it's that a promised object's bytes should be fetchable on demand from the promisor remote, and nothing in apply_manifest currently attempts that on-demand fetch before giving up (this sharpens open question 5 below into the primary remaining lead, alongside the server-side indexing puzzle in the Code trace).

4. Why does apply_manifest correctly detect the missing object and print a clear warning, yet the parent clone command still returns exit_code: 0 and "status": "cloned"? Where exactly does the warning get swallowed instead of propagating to a non-zero exit / error status? This is the most actionable, narrowly-scoped part of the whole investigation — a bug in exit-code propagation is a much smaller fix than a redesign of the promisor model, if that's all this turns out to be.

5. Why does muse verify --json report all_ok: true even though the actively checked-out branch requires a promised (not-yet-materialized) object? Does verify deliberately treat promised objects as fine by definition (correct for a lazy clone that will fetch on demand), and is the actual gap that on-demand fetch never happens when the working tree is restored? I.e., is the real bug not in verify at all, but in apply_manifest failing to trigger a promised-object fetch before giving up?

6. Is a sibling branch a necessary precondition, or would any clone requesting a blob added since the last successful prebuild reproduce this, even on a single-branch repo? (main's own clone in this session happened not to need any blob outside the eager set — but that could be coincidence, not evidence the single-branch case is safe.)

7. Is promisor_remotes: ["origin"] a hardcoded/generic label, or does it reveal a real mismatch (our remote was named staging, not origin) that's worth checking as a contributing factor?

8. Does an explicit muse fetch (delta path, have != []) into an existing checkout correctly retrieve a promised object that a clone left missing, or does the delta path hit the identical gap? (Started testing this during the investigation but didn't reach a clean from-scratch delta-fetch scenario — needs a dedicated atomic repro.)

9. Does muse checkout <branch> (switching branches within an existing clone, rather than clone --branch at creation time) hit the same promised-object gap, or does that code path differ from clone's built-in apply_manifest?

10. Is cat-object's integrity-check failure on promised/stub objects itself a separate, smaller bug (i.e., should cat-object recognize a promised-but-unmaterialized object and say so clearly, rather than reporting a misleading "SHA-256 integrity check failed / store may be corrupt" message that reads as far more alarming than "not fetched yet")? This caused real confusion during triage and could confuse anyone else who reaches for cat-object as a debugging tool.

---

Plan — reproduce at the most atomic level possible

The three-sibling-branch, six-commit, two-feature-branch scratch repo used to discover this has too many moving parts to isolate the trigger. Before touching any production code, narrow the repro one variable at a time:

• [x] Atomic repro 1 — single branch, single push, single clone. muse init → one commit with one file → push → clone (no branches at all). Does this ever produce a promised classification for the only blob that exists? (Expected: no, since this worked for the main clone in the original repro — but confirm on a truly minimal repo, not one with 4 prior commits.)

  **✅ RESULT: no bug.** Fresh repo `gabriel/mwp-qa-atomic-1`, exactly
  one commit, one blob (`file.txt`). Push: `commits_sent=1,
  objects_sent=1`, clean. Clone: `blobs: 1`, `blobs_written: 1`, zero
  warnings, exit 0, content byte-identical. `muse verify --json`:
  `objects_checked: 1`, **`promised_objects: 0`**, `all_ok: true`.
  
  **Two findings from this baseline:**
  1. The "promised" mechanism does not activate at all for a
     single-blob repo — it isn't always-on infrastructure that's simply
     broken; something specific to a multi-blob history triggers it.
     This narrows the search space for atomic repro 2/3.
  2. `promisor_remotes: ["origin"]` appeared in `verify`'s output even
     with `promised_objects: 0` and even though our remote is named
     `staging`, not `origin`. This answers open question 7:
     `"origin"` is very likely a hardcoded/static label in `verify`'s
     output, unrelated to the actual configured remote name — not a
     real naming-mismatch bug. Worth a quick code-read confirmation
     later, not worth its own atomic repro.

• [x] Atomic repro 2 — two commits, two distinct blobs, single branch, immediate clone. Does the second (newest) blob ever get marked promised on a branch with no siblings at all? This isolates whether sibling branches are required to trigger the bug, or whether any "just-added blob" can be miscategorized (open question 6).

  **✅ RESULT: no bug.** Fresh repo `gabriel/mwp-qa-atomic-2`, two
  commits each adding a distinct file (`file1.txt`, `file2.txt`), no
  branches at all. Push: `commits_sent=2, objects_sent=2`. Immediate
  clone: `blobs: 2`, `blobs_written: 2`, zero warnings, exit 0, both
  files present with correct content. `verify`: `objects_checked: 2`,
  `promised_objects: 0`, `all_ok: true`.
  
  **Finding:** sibling branches are *not* required to reach the "2
  eager blobs" ceiling — 2 blobs on a single branch stayed fully eager.
  Combined with atomic repro 1 (1 blob, also fully eager), this narrows
  the eager-cap boundary to **at or above 2** — consistent with the
  original bug, where exactly 2 blobs were eager and the rest (4 more,
  6 total across the whole multi-branch repo) were promised. Atomic
  repro 3 (incrementing blob count on a single branch) should find the
  exact N where promised objects start appearing, without needing any
  branch structure at all.

• [x] Atomic repro 3 — reproduce with a controlled N-blob history, incrementing N one at a time (3, 4, 5, 6, 7 distinct blobs across commits) on a single branch, to determine whether "2 eager blobs" is truly a fixed ceiling (independent of N) or scales with something else (open question 1).

  **✅ RESULT: no bug at any N from 2 through 7.** Fresh repo
  `gabriel/mwp-qa-atomic-3`, one commit per iteration each adding a new
  distinct file, single branch throughout. Every single iteration:
  clean push, clean clone, `promised_objects: 0`, `all_ok: true`, every
  file present with correct content.
  
  | N (blobs) | `promised_objects` | Files on disk |
  |---|---|---|
  | 2 | 0 | 2/2 |
  | 3 | 0 | 3/3 |
  | 4 | 0 | 4/4 |
  | 5 | 0 | 5/5 |
  | 6 | 0 | 6/6 |
  | 7 | 0 | 7/7 |
  
  One noteworthy tangent, recorded for completeness rather than
  pursued: the N=7 iteration hit a 2-minute timeout mid-script during
  the clone step. Retrying the identical push+clone immediately
  afterward completed cleanly in under a second. Most likely cause is
  staging's own rate limiting after 6 rapid consecutive pushes in one
  script (the wire suite has a dedicated
  `test_mpack_rate_limiting_phase4.py`, so this is plausible existing
  infrastructure) rather than anything related to this bug — it did not
  reproduce on retry, so not chased further.
  
  **This falsifies the "blob count on a single branch" hypothesis
  entirely** — a lone branch stays fully eager regardless of how many
  blobs it accumulates, at least up to 7. Combined with atomic repro 2
  (siblings not required to reach 2 blobs cleanly) and the original
  bug (which *did* involve sibling branches), **sibling/multi-branch
  structure now looks like the necessary trigger, not raw blob count.**
  Atomic repro 4 is the direct next test.

• [x] Atomic repro 4 — two sibling branches, minimal (2 commits total: 1 shared + 1 per branch), to isolate whether the sibling-branch structure itself (as opposed to blob count) is the triggering factor.

  **✅ REPRODUCED at the minimal possible scale.** Fresh repo
  `gabriel/mwp-qa-atomic-4`: 1 shared commit on `main` (`shared.txt`),
  branch `feature-a` off it (+`a.txt`), pushed; branch `feature-b` off
  `main` (not off `feature-a`) (+`b.txt`), pushed. Three commits, three
  distinct blobs total — the same blob count that stayed 100% eager on
  a single branch in atomic repro 3.
  
  Cloning `feature-b` (`--branch feature-b`):
  ```
  [transport] want=3 have=0
  [transport] unpackb  commits=3  snaps=3  blobs=2
  ⚠️ clone: working tree partially restored — apply_manifest: 1 object(s)
     missing from the local store ('b.txt').
  exit_code: 0, status: "cloned"
  ```
  
  **This confirms sibling-branch structure is the trigger, not raw
  blob count** — falsifies atomic repro 3's implicit corollary that it
  might just take more blobs; here 3 blobs across 2 sibling branches
  broke immediately, while 7 blobs on 1 branch never did.
  
  **A `--branch feature-a` clone request produced `want=3` too**
  (not `want=1`) — confirming clone always requests the full combined
  tip set regardless of which single branch is being checked out,
  consistent with the documented "one combined mpack per repo" design.
  This looks like correct, intentional client behavior, not a bug.
  
  **Exact object presence, verified with precise path construction
  (not fuzzy `find -name` matching — an earlier pass using glob
  matching produced a false positive on `a.txt`, corrected here):**
  
  | Clone | `shared.txt` (main, pushed 1st) | `a.txt` (feature-a, pushed 2nd) | `b.txt` (feature-b, pushed 3rd/last) |
  |---|---|---|---|
  | `feature-b`, 1st attempt (immediately after push) | ✅ present | ❌ missing | ❌ missing |
  | `feature-a` | ✅ present | ✅ present | ❌ missing |
  | `feature-b`, retry (~4 min later) | ✅ present | ❌ missing | ❌ missing |
  
  **Sharpened finding:** it is specifically **the blob belonging to
  whichever branch was pushed most recently** that is excluded from the
  eager set — and it does **not** resolve after ~4 minutes, which rules
  out an ordinary short-lived prebuild-catch-up race (that would
  predict the gap closing once the async job completes; it never did
  across two attempts 4 minutes apart, and the served `bundle_id` was
  byte-identical both times, meaning the exact same stale cached
  mpack was served twice with no refresh in between). This points more
  towards **the prebuild job for the third (`feature-b`) push either
  never being enqueued, silently failing, or getting stuck** — squarely
  in the RC-3/RC-4 lineage (job idempotency / ordering), but for a
  trigger neither MWP-3 nor MWP-4 tested: **a brand-new branch's first
  push, when it's the *third or later* distinct branch tip in the
  repo.** `feature-a` (the *second* branch, first-ever sibling) got its
  own blob eagerly — so the failure mode isn't simply "any new sibling
  branch," it specifically appeared on the *next* push after that.
  
  **Immediate next diagnostic, before any code read:** does pushing a
  *fourth* commit (to any branch) finally trigger a prebuild that picks
  up `b.txt`, confirming a "one push behind" lag rather than a
  permanently stuck/dropped job? This is cheap to test and would
  further narrow open questions 1–3 without needing repo access to the
  job queue table.

• [x] Atomic repro 4b — push a 4th commit, does it unstick b.txt? Pushed a 4th commit to main on the atomic-4 repo. ✅ RESULT: no, it does not self-heal — and it got worse. bundle_id changed (a new combined mpack was built, now covering 4 commits), but verify --json now reports promised_objects: 3 (a.txt, b.txt, and fourth.txt, the brand-new commit's own blob) — only shared.txt, the very first blob ever pushed to this repo, remains eager. This rules out a simple "one push behind" lag (which would predict b.txt resolving once a subsequent push's indexing completes) and instead suggests whatever caused b.txt's exclusion is a standing condition for this repo, not a transient race that time or further activity repairs.

• [ ] Atomic repro 5 — delta fetch (have != []) into an existing checkout vs. clone (have == []), to answer open question 8 with a clean, minimal case. Deferred — superseded in priority by the code trace below, which narrowed the search space enough that a targeted diagnostic test is now more valuable than this generic black-box check.

• [x] Atomic repro 6 (generalized) — code trace of the eager-vs-promised decision and the indexing job, done via muse code grep / content-grep / direct reads rather than a new repro. Full findings below.

Each atomic repro should be run in a fresh /tmp scratch repo (naming convention: mwp-qa-atomic-N), with full transport-log output captured (not truncated with tail, learned the hard way in the original repro), and its result recorded back into this ticket before moving to the next.

Exit condition for this phase: either a confirmed root cause (a specific function/line where eager-vs-promised is decided incorrectly, or where the exit code is swallowed), or a clear statement of which atomic repro produced an unexpected result that itself becomes the next thing to instrument. This ticket stays an investigation record — no fix is attempted until the mechanism is understood.

---

Code trace — where eager-vs-promised is actually decided

Full chain, traced via muse code grep / content-grep / direct reads, no production code modified.

1. ObjectState.PROMISED (muse/core/object_availability.py) is generic and reactive, not a decision point

def object_state(root, obj_id, promisor_remotes) -> ObjectState:
    if _object_path_with_fallback(root, obj_id).exists():
        return ObjectState.PRESENT
    if promisor_remotes:
        return ObjectState.PROMISED
    return ObjectState.MISSING

This is purely: "does the file exist on disk? No, but a remote is configured? Then it's PROMISED (benign label), not MISSING (real failure)." It says nothing about why an object wasn't fetched — it's a downstream label applied by verify/build_mpack to whatever the fetch/clone path already did or didn't write to disk. This resolves open question 2 in part: the "promised" label itself lives client-side and is generic; it is not where eager-vs-promised gets decided.

2. build_mpack's promisor-skip logic (muse/core/mpack.py:781-794) is client-push-only, confirmed not used server-side

promisor_remotes = load_promisor_remotes(repo_root)
for oid in sorted(candidate_blob_ids):
    raw = read_object(repo_root, oid)
    if raw is None:
        state = object_state(repo_root, oid, promisor_remotes)
        if state == ObjectState.PROMISED:
            logger.debug(...); continue   # silently skip
        raise ValueError(...)             # hard error otherwise

Confirmed via content-grep "build_mpack|build_wire_mpack" across musehub: the server only ever calls build_wire_mpack (pure serialization of an already-assembled dict) — never build_mpack (the BFS-walk-and-skip function above). This resolves open question 3: the promisor-skip mechanism is a legitimate client-side muse push safety valve (don't hard-fail a push just because your local shallow clone is missing a historical blob the remote already has) — it is architecturally absent from the server path entirely. The server has its own, separate blob-selection logic.

3. The real server-side gate — wire_fetch_mpack (musehub/services/musehub_wire_fetch.py)

Lines 681–694 are the load-bearing correctness gate:

if new_oids:
    indexed_q = await session.execute(
        select(MusehubMPackIndex.entity_id)
        .where(MusehubMPackIndex.entity_id.in_(list(new_oids)))
        .where(MusehubMPackIndex.entity_type == "object")
    )
    indexed_oids = {row[0] for row in indexed_q}
    missing = new_oids - indexed_oids
    if missing:
        raise FetchNotIndexedError(len(missing))   # → HTTP 503, MWP-5 retries

This correctly 503s when an object has no index row at all. Since our repro consistently returns 200, b.txt's oid must have an index row by the time of the fetch. The bug is downstream of this gate: lines 697–820 try three strategies to resolve indexed-but-not-cached oids to actual bytes (cache_hits from MusehubObject.content_cache, mpack_hits by downloading the mpack its index row points to and extracting the blob, and a legacy_hits fallback via backend.get(oid) direct read). The final assembly line:

blob_pairs = [(oid, _all_blob_bytes[oid]) for oid in new_oids if oid in _all_blob_bytes]

silently drops any oid that fails all three strategies — no error, no warning naming which object or why. This is the exact mechanism of the silent partial mpack.

4. process_mpack_index_job (musehub/services/musehub_wire_push.py:1571) — objects are never stored standalone

This job runs once per push, reads that push's own uploaded mpack, and writes MusehubMPackIndex rows keyed to mpack_id = <that push's own mpack_key> — never a combined/consolidated key. It also writes MusehubObject.storage_uri = f"mpack://{mpack_key}" for every object in that push. There is no code path anywhere in this function (or found elsewhere) that stores an object as a standalone, individually-addressable key. An object's bytes are only ever recoverable by re-reading the exact mpack file its index row points to. This means the legacy_hits fallback (backend.get(oid) — a direct, non-mpack-scoped read) is very likely always a no-op for any object pushed through this pipeline — it would only succeed for objects written by some other, genuinely legacy path.

5. process_mpack_gc_job (musehub/services/musehub_wire_fetch.py:913) — real inconsistency found, but confirmed inactive

This function consolidates every per-push mpack for a repo into one, then deletes the old MusehubMPackIndex rows (pointing them at the new consolidated key instead). Its sibling, purge_stale_mpack_index_entries (musehub_wire_push.py:1133), has a docstring claiming "objects with stale entries have valid s3:// storage_uri and are served directly from S3 without needing the mpack path" — this is false per finding #4 above; the actual writer only ever sets mpack://. This is a real bug-in-waiting, but confirmed via content-grep that neither function is called from worker.py's dispatch table — the worker only actively processes "mpack.index" and "fetch.mpack.prebuild"; every other job type (including whatever would trigger GC/consolidation) is a no-op (pass). GC/consolidation cannot be the active mechanism for this bug — it never ran against gabriel/mwp-qa-atomic-4. Flagged separately as its own latent-bug finding, not pursued further here.

6. The one loose thread — why does feature-b's indexed oid fail all three retrieval strategies?

Given #3–#5, the only active job that could index b.txt is feature-b's own mpack.index job, which — if it ran successfully — should leave b.txt's original push mpack fully intact and retrievable (nothing deletes it). That would make mpack_hits succeed. Since it doesn't, one of the following must be true, and cannot be distinguished by further code reading or black-box HTTP testing alone:

• (a) feature-b's own mpack.index job never ran or never completed (dropped by a coarse per-(repo_id, job_type, pending) dedup collision with an earlier still-pending mpack.index job from main's or feature-a's push — the exact RC-3 mechanism MWP-3 fixed, but potentially for a trigger neither MWP-3 nor MWP-4's test suites covered: a third-or-later branch's very first push, in quick succession after two prior pushes). But this should make b.txt's oid absent from indexed_oids entirely, which the gate in finding #3 would catch and turn into a 503 — which we never observed. This is the part that doesn't yet fit.
• (b) The job ran and wrote an index row, but something rewrote or corrupted that row's mpack_id afterward (no known active code path does this, per findings #4–#5, but an unknown one may exist).
• (c) The job ran, wrote the correct row, but backend.get_mpack() for that specific key transiently or permanently fails for a reason unrelated to indexing (e.g., a storage-backend-level issue with that specific upload) — would require direct storage/backend inspection to confirm.

Resolving this requires either live DB/job-log state for gabriel/mwp-qa-atomic-4 (whether feature-b's mpack.index job actually ran, and what MusehubMPackIndex actually contains for b.txt's oid right now) or a new deterministic test that controls job timing precisely — the same methodology MWP-3/MWP-4's own test suites used to confirm their respective root causes before any fix was written.

---

⚠️ RETRACTION of the previous "ROOT CAUSE CONFIRMED — escalated" section below

The section that follows this note (kept for the record, not deleted — see workspace data-integrity convention of never silently erasing a documented finding) was written after reproducing against localhost, but that localhost run was against a stale musehub_worker container whose Docker image dated to 2026-05-15 — over a month old. Nothing in this workspace's documented conventions mentions restarting the worker container (only the musehub API container has a documented restart requirement), so it had been silently running old code the entire time.

After restarting both musehub and musehub_worker (which also surfaced and required fixing an unrelated, separate schema-migration gap — see "Side finding" below) and re-running the identical scenario against genuinely current code, both conclusions below turned out to be wrong:

• "Consolidation (process_mpack_gc_job) is load-bearing, ticket #114's removal recommendation is wrong" — wrong. With the worker actually running current code, process_fetch_mpack_prebuild_job correctly builds one shared mpack across all tips, exactly as documented (fresh logs: tips=3 ... built=3, one shared mpack_id written to all three tips' cache rows in a single ~20ms build). The multi-tip cache-HIT invariant is genuinely satisfiable by current code. musehub#114's original "safe to remove" recommendation was correct; my correction to it was itself wrong and has been un-corrected there.
• "On current dev/HEAD, cloning any multi-branch repo now perpetually fails" — wrong, same cause. A properly-running worker never needs the force_build=True gate to hard-block a real clone — prebuild completes fast enough that the cache is warm by the time a client's own request lands, in every test we ran after the restart.

Side finding, fixed to unblock this re-verification: the restart also surfaced a genuine, unrelated, pre-existing bug — the MusehubVersionTag ORM model (musehub/db/musehub_repo_models.py:505) had no corresponding Alembic migration ever generated, so any environment restarting from a sufficiently old running-process state would fail assert_schema_matches_orm's startup check. Generated and applied alembic/versions/0073_add_musehub_version_tags.py to fix this locally. Worth its own follow-up ticket to check whether staging's DB has the same gap (separate from this investigation).

The actual, confirmed-at-the-database-level root cause is below.

---

🔴 ROOT CAUSE CONFIRMED — a plain logic bug, not a caching/timing/deployment issue

Re-running atomic repro 4's exact scenario against a freshly restarted localhost (confirmed current code, confirmed schema in sync) reproduced the identical bug on the very first attempt — immediately, no retries, no timing dependency. This ruled out every caching/job-timing/deployment-drift theory in one step: the bug is deterministic and present on current code.

The mechanism — found and confirmed via a direct database query

musehub_wire_fetch.py:555-559:

want_tip_snap_q = await session.execute(
    select(MusehubCommitGraph.snapshot_id)
    .where(MusehubCommitGraph.commit_id.in_(_needed_cids))
    .order_by(MusehubCommitGraph.generation.desc())
    .limit(1)
)
want_tip_snap_id = want_tip_snap_q.scalar_one_or_none()

This assumes there is exactly one "the tip" for the entire fetch request, and computes all_oids (the full set of blobs to include in the response, lines 612-648) from only that one snapshot's manifest.

Queried musehub_commit_graph directly for the three commits in the atomic repro 4 scenario:

commit_id                    generation   snapshot_id
sha256:5fb426a0... (main)    0            f411f3d3...
sha256:78ab4d6f... (feat-a)  1            288ed020...
sha256:d44962ef... (feat-b)  1            bcfd655e...   ← ties feature-a

feature-a and feature-b are siblings — both children of the same base commit — so they share generation 1. ORDER BY generation DESC LIMIT 1 on a tie is arbitrary; Postgres returned feature-a's snapshot. all_oids is built from that snapshot's manifest alone. feature-b's manifest — and its unique blob b.txt — is never looked at, never queried, never attempted. This is not a resolution failure downstream (the three-strategy cache_hits/mpack_hits/legacy_hits machinery we traced earlier never even gets a chance to try b.txt, because it's never added to all_oids in the first place).

The boundary, precisely

Triggers whenever two or more requested tips share the same generation — i.e., any two branches cut from a common ancestor at the same depth, which is the ordinary shape of two feature branches. Does not trigger for a single linear branch (only one tip ever exists, no tie possible) — this is exactly why atomic repros 1–3 (all single-branch) never reproduced it, and why atomic repro 4 (the first sibling-branch test) reproduced it immediately and consistently.

Confirmed deterministic, not timing-dependent: explains in hindsight why waiting longer, pushing a 4th commit, or retrying never made a difference in earlier atomic repros — there was never anything to "catch up" on.

What is still genuinely unknown after this finding

• Whether staging's deployed code has this same line — very likely, since this looks like old, foundational logic unrelated to the recent MWP-1..8 campaign, but not yet directly re-confirmed against staging with this specific lens.

• What staging's actual deployed commit is — still unverified; now a secondary question rather than the explanation for the corruption itself.

• The correct fix shape — does all_oids need to become the union of manifests across every tip in want regardless of generation (the fully general fix), or is the tie-at-max-generation case the only broken one? Needs to be checked before writing a fix — e.g. what happens today when requested tips are at genuinely different generations (a deep branch vs a shallow one) — does the shallow one's manifest also get silently dropped, or does _needed_cids/all_oids handle that case some other way we haven't yet traced?

  ✅ RESOLVED — fully general, not tie-specific. Confirmed via three live E2E repros against a freshly-restarted localhost (see below): the bug affects any want set spanning more than one branch tip, tied generations or not, including the realistic case of a plain muse clone of the default branch when any other branch in the repo is deeper.

---

✅ FIXED — Phase 1, TDD

Test-first (RED)

New file tests/test_multi_tip_manifest_union.py, three tests each reproducing one confirmed scope boundary, all RED against the pre-fix code:

• MTU_01 — two sibling branches at the same generation (a tie): the tie-losing sibling's unique blob is dropped.
• MTU_02 — two sibling branches at different generations (no tie, shallower one loses deterministically): the shallower sibling's unique blob is dropped.
• MTU_03 — the default branch itself loses its own newest content merely because another branch in the repo is deeper. This is the realistic "just run muse clone <url>" case, not a contrived edge case.

All three failed with the exact predicted missing blob before the fix.

The fix

musehub_wire_fetch.py, immediately after the existing single-tip want_tip_snap_id block (left untouched — it still guards a different, valid concern: detecting corrupt generation values used by _walk_commit_delta's range scan): added a union step that resolves each tip actually listed in want from its own authoritative MusehubCommit.snapshot_id (already loaded into commit_rows, already the correct source per the existing MWP1_13 comment — "MusehubCommit always has the correct snapshot_id") and unions every tip's manifest into all_oids, instead of relying on a single CommitGraph-generation-ranked pick.

# musehub#113 fix — union every want-tip's own manifest, not one pick.
_pre_union_oids = len(all_oids)
for _want_cid in want:
    _want_commit_row = commit_rows.get(_want_cid)
    if _want_commit_row is None or not _want_commit_row.snapshot_id:
        continue
    _want_snap_entry = snap_map.get(_want_commit_row.snapshot_id)
    if _want_snap_entry:
        all_oids.update(v for v in (_want_snap_entry.get("manifest") or {}).values() if v)

Verified green — unit, regression, and live E2E

• MTU_01, MTU_02, MTU_03 — all 3 GREEN after the fix.
• Adjacent regression suite (all run one file at a time, per workspace convention): test_mwp1_generation_authority.py (13), test_mwp2_walk_fallback.py (10), test_fetch_mpack_prebuild.py (8), test_wire_fetch_mpack.py + test_mpack_fetch_phase3.py (11), test_mwp8_cache_invalidation.py + test_mwp4_prebuild_ordering.py + test_mwp3_job_idempotency.py (42) — 84 tests, zero regressions.
• Full pytest -m wire suite: 56 passed, matching Phase 0's baseline exactly.
• Live E2E confirmation against a fresh scratch repo (gabriel/mwp-qa-local-v4, no prior cache contamination): pushed main (2 commits, main_unique.txt as the newest), pushed feature-deep (3 commits, deeper than main), then a plain muse clone <url> (no --branch flag — the realistic default-branch case) returned no warning, blobs_written=5 (the full cross-tip union), and main_unique.txt present with correct content. muse verify --json: objects_checked: 2 (exactly main's own 2-file manifest), promised_objects: 3 (correctly feature-deep's 3 blobs, legitimately irrelevant to main's own checkout — not a bug, the same benign partial-clone deferral behavior confirmed earlier for non-default-branch checkouts), all_ok: true.

---

The unifying finding — why this closes clone/fetch/pull together, and what push needed

Before fixing anything further, we asked whether push, fetch, and pull each needed their own separate investigation, or whether the all_oids fix already covered them. Verified empirically rather than assumed:

• muse clone is architecturally the only verb that ever constructs a want set spanning every known branch tip in one request — confirmed by transport-log inspection: a plain muse clone <url> with no --branch flag sends want=N for all N known tips, every time.
• muse fetch <remote> --branch X — confirmed via a live transport-log check — always sends want=1, scoped to exactly the one named branch. --all fetches every configured remote, not every branch on one remote. Fetch was never structurally exposed to the all_oids bug at all — not because it was accidentally safe, but because its want set is single-tip by construction.
• muse pull reuses fetch_mpack internally (per the architecture recap) — same conclusion.

So the all_oids fix protects clone (the one place the bug could actually manifest) and is a correctness no-op for fetch/pull (they never sent a multi-tip want to begin with).

That symmetry prompted a direct question: does the same single-tip-pick anti-pattern exist anywhere else in this function? content-grep for the same ORDER BY generation DESC LIMIT 1 shape found a fourth, previously undiscovered instance — the mirror side, have_oids — and it was real (MTU_04, RED before the fix). Unlike all_oids, undercounting have_oids cannot drop content (new_oids = all_oids - have_oids can only grow), but it means any client tracking more than one branch (fetch/pull with a genuinely non-empty, multi-tip have set — the normal shape of a repo with several local branches) gets objects it already has redundantly re-sent. Fixed the same way: union across every have tip via authoritative MusehubCommit rows, and switched off the unsafe raw manifest_blob read onto the same safe snap_map reconstruction the all_oids fix already uses — closing a second, adjacent gap the original BLOB-DEBUG comment had already flagged as a risk but never acted on.

This is the actual unifying principle behind this entire investigation, not just this one ticket: every root cause found across MWP-1 through MWP-9 (RC-1's generation-0 fallback, RC-2's dead DAG-walk fallback, RC-3/4's job-state races, RC-8's uninvalidated cache, and now this) is the same shape — a derived, optimized, or cached value (a CommitGraph.generation column, a MusehubFetchMPackCache row, a MusehubMPackIndex row) being trusted as if it were the authoritative source (MusehubCommit.parent_ids, MusehubCommit.snapshot_id, the actual object bytes) instead of being reconciled against it. push was independently checked for the same pattern (content-grep for the same query shape across musehub_wire_push.py — zero hits) — its own instance of this class of bug (RC-1) was already fixed in MWP-1, and no new instance was found. That is the complete picture for all four verbs: clone and the two wire_fetch_mpack bugs above (fixed), fetch/pull (never exposed, and now carry the have_oids efficiency fix as a bonus), push (already fixed in MWP-1, re-checked here, clean).

---

Impact

This directly falsifies part of #58's Claim 1 ("aaronrene can muse clone <url> — get a complete, correct working tree every time"). It also means muse verify cannot currently be trusted as a post-clone completeness check, which undermines the general recommendation to use it as a sanity gate. Both of these are more severe than the dedup question SC_02b originally set out to answer.

Out of scope (for this ticket)

• Any actual code fix — this ticket is investigation-only until root cause is confirmed (see Plan above).
• Re-verifying SC_02a/dedup correctness — already independently confirmed correct in this session, not implicated in this bug.