# Silent partial clone: required blob missing from working tree, `clone` exits 0, `verify` reports `all_ok: true` > **Parent:** [muse#63](https://staging.musehub.ai/gabriel/muse/issues/63) (MWP-9 — spot check of #58's goal-of-done). Discovered during Phase 2 (`SC_02b`, three-sibling-branch dedup) against the scratch repo `gabriel/mwp-qa-1a` on staging. > **Status: investigation, not yet root-caused.** No production code has been touched. This ticket is a brain dump of everything observed plus the open questions that block a fix. The plan is to reproduce at the most atomic level possible, narrowing one variable at a time, until we hit either a confirmed root cause or a wall that tells us exactly what to instrument next. --- ## Background — what we were actually testing Phase 2 of MWP-9 was verifying Claim 2 of #58 ("push uploads only the true delta, never re-uploads objects already on the remote via any ref"). The specific sub-test (`SC_02b`) was: push two sibling branches (`feature-a`, `feature-b`) off a shared `main`, and confirm the second branch's push dedups correctly against the first. **That part passed, and passed more thoroughly than planned:** `feature-b`'s push sent exactly 1 object (its own `fb.txt` blob), correctly excluding README.md's blob despite 4 incremental versions of it existing across the shared ancestor chain with both `main` and `feature-a`. RC-7 / MWP-7's sibling-ref dedup fix holds up under a stronger test than the original two-branch case it shipped with. **What broke instead was cloning the branch we'd just proven the push logic handled correctly.** --- ## What we know (chronological brain dump) ### 1. The exact repro sequence All against `https://staging.musehub.ai/gabriel/mwp-qa-1a` (scratch repo, created fresh for this QA campaign): ``` commit 1 (main): "init" — README.md = "hello" commit 2 (main): "v2" — README.md = "hello\nv2" commit 3 (main): "a" — README.md = "hello\nv2\na" commit 4 (main): "b" — README.md = "hello\nv2\na\nb" push staging main → commits_sent=1, objects_sent=1 (per commit, each pushed individually as they were made) branch feature-a from main tip (commit 4) commit 5 (feature-a): "feature a" — adds fa.txt = "fa\n" push staging feature-a → commits_sent=1, objects_sent=1 ✅ correct dedup checkout main (back to commit 4), branch feature-b from main tip commit 6 (feature-b): "feature b" — adds fb.txt = "fb\n" push staging feature-b → commits_sent=1, objects_sent=1 ✅ correct dedup rm -rf /tmp/mwp-qa-2b-clone-retry muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2b-clone-retry --branch feature-b --json ``` ### 2. The failure, verbatim ``` [transport] unpackb commits=6 snaps=6 blobs=2 t=0ms [clone] apply_mpack DONE t=12ms blobs_written=2 blobs_skipped=0 commits_written=6 [mpack] fetch/mpack: 0.78s blobs: 2 commits: 6 [clone] apply_manifest START (working tree restore) ⚠️ clone: working tree partially restored — apply_manifest: 1 object(s) missing from the local store ('fb.txt'). The working tree cannot be fully restored. Fetch the missing objects from the remote before retrying. [clone] apply_manifest DONE t=7ms {"...", "exit_code": 0, "status": "cloned", "commits_received": 6, "blobs_written": 2, "skipped_blobs": 0, "head": "sha256:005bc9e6...", "domain": "code", "dry_run": false} ``` `fb.txt` is **completely absent from disk** (confirmed via `ls`, not just empty). **`exit_code` is 0. `status` is `"cloned"`, not an error state.** 100% reproducible — ran it twice in separate fresh directories, identical result both times. ### 3. Ruled out: manifest correctness, tool misuse, and (probably) content-store corruption - The commit's manifest correctly maps `fb.txt` → `sha256:b566176258f9d8...`. Confirmed via `muse hash-object fb.txt` on the *original* working copy — the manifest's declared object ID is exactly right. This is not a metadata/manifest bug. - `--branch feature-b` is the documented, correct clone flag (checked `muse clone --help`). - Initial alarm: running `muse cat-object sha256: --json` on 6 of the 13 objects in the local store returned **SHA-256 integrity-check failures** ("expected X, got Y") — this first read as store corruption. **This was almost certainly a false alarm from misusing `cat-object` on non-blob objects.** Running the *actual* sanctioned integrity tool, `muse verify --json`, on the same clone reports `all_ok: true`, `failures: []`, and critically **`promised_objects: 5`** — i.e., those 6 "corrupt-looking" objects are very likely placeholder/stub entries for a **lazy/partial object model** (`cat-object` appears to hash the placeholder bytes directly rather than recognizing the stub and reporting it as not-yet-materialized, which is itself worth separate scrutiny — see open question 4 below — but is not the primary bug). ### 4. The actual anomaly: a consistent "2 eager blobs" ceiling regardless of target Every single clone/fetch run during this QA session — regardless of which branch was requested or how many files that branch's manifest actually needed — reported **exactly `blobs=2`** materialized eagerly, with the remainder tracked as `promised_objects`: | Clone target | Blobs actually needed by manifest | Blobs eagerly materialized | Promised | |---|---|---|---| | `main` (4 commits, README only) | 1 (README latest) | 2 | — (not observed to fail, README happened to be eager) | | `feature-b` (adds `fb.txt`) | 2 (README latest + fb.txt) | 2 | 5 — **but `fb.txt` was among the promised, not the eager, set** | This strongly suggests the server's combined-mpack / prebuild mechanism has a fixed or coincidentally-fixed eager-blob inclusion count, and nothing guarantees the blob(s) required by the **specifically requested `want` tip's own manifest** are always in that eager set. When they're not, the client's `apply_manifest` step correctly *detects* the gap and prints a warning — but the overall `clone` command still reports success. ### 5. `muse verify --json` output on the broken clone (full) ```json { "repo_id": "sha256:7720f84c823f423f1de706cd9dddefc2e82e98f3301eced36c35ddd8414abc5d", "refs_checked": 3, "commits_checked": 6, "snapshots_checked": 6, "objects_checked": 1, "signatures_checked": 6, "shallow_commits": 0, "promised_objects": 5, "is_shallow": false, "promisor_remotes": ["origin"], "all_ok": true, "nothing_checked": false, "check_objects": true, "strict": false, "branch": null, "failures": [] } ``` Notable: `promisor_remotes: ["origin"]` — but the remote we configured and pushed/cloned through was named `staging`, not `origin`. Either `"origin"` is a hardcoded/generic label unrelated to the actual remote name (cosmetic), or there's a mismatch worth checking (see open question 7). `all_ok: true` despite `feature-b`'s own checked-out HEAD requiring an object that is not present and is only "promised." This means `verify`'s notion of "ok" does not currently check whether promised objects are needed by the actively-checked-out branch — it appears to only check structural consistency of commits/snapshots/signatures, not working-tree completeness. ### 6. Confirmed NOT the bug - **Sibling-ref dedup (RC-7/MWP-7):** correct, verified independently — see §1, `feature-b`'s push sent exactly 1 object. - **Manifest content:** correct — `fb.txt`'s declared object ID matches its real content hash. - **CLI flag usage:** correct — `--branch feature-b` is the documented flag. --- ## What we don't know — open questions to close before any fix is attempted 1. **What determines the eager-blob cap?** Is "2" a hardcoded constant, a config value, or an artifact specific to this session's exact push history shape (e.g., "however many *new* blobs were added since the last successful prebuild")? Needs a code-level answer, not just observed behavior. 2. ~~**Where does the "promised object" concept live in the codebase**~~ **✅ RESOLVED (see Code trace §1-2).** `ObjectState.PROMISED` (`muse/core/object_availability.py`) is a generic, purely local, reactive label — not a decision point. `build_mpack`'s promisor-skip logic (`muse/core/mpack.py:781-794`) is confirmed client-push-only, never invoked server-side. The actual server-side blob-selection logic is `wire_fetch_mpack`'s own three-strategy resolution (`musehub_wire_fetch.py:697-820`), which silently drops any oid that fails all three — that is where the real decision (or non-decision) happens. 3. ~~**Is the promisor/lazy-object model intentional**~~ **✅ RESOLVED.** Yes, intentional and documented — `object_availability.py`'s own docstring explicitly describes it as mirroring git's promisor-remote concept for partial clones, with the *opted-in-unless-opted-out* default explained as the right choice for an agent-first, trusted-hub platform. It is not an accidental byproduct of the MWP campaign. **The actual bug is not the existence of this model — it's that a promised object's bytes should be fetchable on demand from the promisor remote, and nothing in `apply_manifest` currently attempts that on-demand fetch before giving up** (this sharpens open question 5 below into the primary remaining lead, alongside the server-side indexing puzzle in the Code trace). 4. **Why does `apply_manifest` correctly detect the missing object and print a clear warning, yet the parent `clone` command still returns `exit_code: 0` and `"status": "cloned"`?** Where exactly does the warning get swallowed instead of propagating to a non-zero exit / error status? This is the most actionable, narrowly-scoped part of the whole investigation — a bug in exit-code propagation is a much smaller fix than a redesign of the promisor model, if that's all this turns out to be. 5. **Why does `muse verify --json` report `all_ok: true`** even though the actively checked-out branch requires a promised (not-yet-materialized) object? Does `verify` deliberately treat promised objects as fine by definition (correct for a *lazy* clone that will fetch on demand), and is the actual gap that **on-demand fetch never happens** when the working tree is restored? I.e., is the real bug not in `verify` at all, but in `apply_manifest` failing to trigger a promised-object fetch before giving up? 6. **Is a sibling branch a necessary precondition**, or would *any* clone requesting a blob added since the last successful prebuild reproduce this, even on a single-branch repo? (`main`'s own clone in this session happened not to need any blob outside the eager set — but that could be coincidence, not evidence the single-branch case is safe.) 7. **Is `promisor_remotes: ["origin"]` a hardcoded/generic label**, or does it reveal a real mismatch (our remote was named `staging`, not `origin`) that's worth checking as a contributing factor? 8. **Does an explicit `muse fetch` (delta path, `have != []`) into an existing checkout correctly retrieve a promised object that a `clone` left missing**, or does the delta path hit the identical gap? (Started testing this during the investigation but didn't reach a clean from-scratch delta-fetch scenario — needs a dedicated atomic repro.) 9. **Does `muse checkout ` (switching branches within an existing clone, rather than `clone --branch` at creation time) hit the same promised-object gap**, or does that code path differ from `clone`'s built-in `apply_manifest`? 10. **Is `cat-object`'s integrity-check failure on promised/stub objects itself a separate, smaller bug** (i.e., should `cat-object` recognize a promised-but-unmaterialized object and say so clearly, rather than reporting a misleading "SHA-256 integrity check failed / store may be corrupt" message that reads as far more alarming than "not fetched yet")? This caused real confusion during triage and could confuse anyone else who reaches for `cat-object` as a debugging tool. --- ## Plan — reproduce at the most atomic level possible The three-sibling-branch, six-commit, two-feature-branch scratch repo used to discover this has too many moving parts to isolate the trigger. Before touching any production code, narrow the repro one variable at a time: - [x] **Atomic repro 1 — single branch, single push, single clone.** `muse init` → one commit with one file → push → clone (no branches at all). Does this ever produce a `promised` classification for the only blob that exists? (Expected: no, since this worked for the `main` clone in the original repro — but confirm on a truly minimal repo, not one with 4 prior commits.) **✅ RESULT: no bug.** Fresh repo `gabriel/mwp-qa-atomic-1`, exactly one commit, one blob (`file.txt`). Push: `commits_sent=1, objects_sent=1`, clean. Clone: `blobs: 1`, `blobs_written: 1`, zero warnings, exit 0, content byte-identical. `muse verify --json`: `objects_checked: 1`, **`promised_objects: 0`**, `all_ok: true`. **Two findings from this baseline:** 1. The "promised" mechanism does not activate at all for a single-blob repo — it isn't always-on infrastructure that's simply broken; something specific to a multi-blob history triggers it. This narrows the search space for atomic repro 2/3. 2. `promisor_remotes: ["origin"]` appeared in `verify`'s output even with `promised_objects: 0` and even though our remote is named `staging`, not `origin`. This answers open question 7: `"origin"` is very likely a hardcoded/static label in `verify`'s output, unrelated to the actual configured remote name — not a real naming-mismatch bug. Worth a quick code-read confirmation later, not worth its own atomic repro. - [x] **Atomic repro 2 — two commits, two distinct blobs, single branch, immediate clone.** Does the second (newest) blob ever get marked `promised` on a branch with no siblings at all? This isolates whether *sibling branches* are required to trigger the bug, or whether *any* "just-added blob" can be miscategorized (open question 6). **✅ RESULT: no bug.** Fresh repo `gabriel/mwp-qa-atomic-2`, two commits each adding a distinct file (`file1.txt`, `file2.txt`), no branches at all. Push: `commits_sent=2, objects_sent=2`. Immediate clone: `blobs: 2`, `blobs_written: 2`, zero warnings, exit 0, both files present with correct content. `verify`: `objects_checked: 2`, `promised_objects: 0`, `all_ok: true`. **Finding:** sibling branches are *not* required to reach the "2 eager blobs" ceiling — 2 blobs on a single branch stayed fully eager. Combined with atomic repro 1 (1 blob, also fully eager), this narrows the eager-cap boundary to **at or above 2** — consistent with the original bug, where exactly 2 blobs were eager and the rest (4 more, 6 total across the whole multi-branch repo) were promised. Atomic repro 3 (incrementing blob count on a single branch) should find the exact N where promised objects start appearing, without needing any branch structure at all. - [x] **Atomic repro 3 — reproduce with a controlled N-blob history**, incrementing N one at a time (3, 4, 5, 6, 7 distinct blobs across commits) on a single branch, to determine whether "2 eager blobs" is truly a fixed ceiling (independent of N) or scales with something else (open question 1). **✅ RESULT: no bug at any N from 2 through 7.** Fresh repo `gabriel/mwp-qa-atomic-3`, one commit per iteration each adding a new distinct file, single branch throughout. Every single iteration: clean push, clean clone, `promised_objects: 0`, `all_ok: true`, every file present with correct content. | N (blobs) | `promised_objects` | Files on disk | |---|---|---| | 2 | 0 | 2/2 | | 3 | 0 | 3/3 | | 4 | 0 | 4/4 | | 5 | 0 | 5/5 | | 6 | 0 | 6/6 | | 7 | 0 | 7/7 | One noteworthy tangent, recorded for completeness rather than pursued: the N=7 iteration hit a 2-minute timeout mid-script during the clone step. Retrying the identical push+clone immediately afterward completed cleanly in under a second. Most likely cause is staging's own rate limiting after 6 rapid consecutive pushes in one script (the wire suite has a dedicated `test_mpack_rate_limiting_phase4.py`, so this is plausible existing infrastructure) rather than anything related to this bug — it did not reproduce on retry, so not chased further. **This falsifies the "blob count on a single branch" hypothesis entirely** — a lone branch stays fully eager regardless of how many blobs it accumulates, at least up to 7. Combined with atomic repro 2 (siblings not required to reach 2 blobs cleanly) and the original bug (which *did* involve sibling branches), **sibling/multi-branch structure now looks like the necessary trigger, not raw blob count.** Atomic repro 4 is the direct next test. - [x] **Atomic repro 4 — two sibling branches, minimal (2 commits total: 1 shared + 1 per branch)**, to isolate whether the sibling-branch structure itself (as opposed to blob count) is the triggering factor. **✅ REPRODUCED at the minimal possible scale.** Fresh repo `gabriel/mwp-qa-atomic-4`: 1 shared commit on `main` (`shared.txt`), branch `feature-a` off it (+`a.txt`), pushed; branch `feature-b` off `main` (not off `feature-a`) (+`b.txt`), pushed. Three commits, three distinct blobs total — the same blob count that stayed 100% eager on a single branch in atomic repro 3. Cloning `feature-b` (`--branch feature-b`): ``` [transport] want=3 have=0 [transport] unpackb commits=3 snaps=3 blobs=2 ⚠️ clone: working tree partially restored — apply_manifest: 1 object(s) missing from the local store ('b.txt'). exit_code: 0, status: "cloned" ``` **This confirms sibling-branch structure is the trigger, not raw blob count** — falsifies atomic repro 3's implicit corollary that it might just take more blobs; here 3 blobs across 2 sibling branches broke immediately, while 7 blobs on 1 branch never did. **A `--branch feature-a` clone request produced `want=3` too** (not `want=1`) — confirming clone always requests the full combined tip set regardless of which single branch is being checked out, consistent with the documented "one combined mpack per repo" design. This looks like correct, intentional client behavior, not a bug. **Exact object presence, verified with precise path construction (not fuzzy `find -name` matching — an earlier pass using glob matching produced a false positive on `a.txt`, corrected here):** | Clone | `shared.txt` (main, pushed 1st) | `a.txt` (feature-a, pushed 2nd) | `b.txt` (feature-b, pushed 3rd/last) | |---|---|---|---| | `feature-b`, 1st attempt (immediately after push) | ✅ present | ❌ missing | ❌ missing | | `feature-a` | ✅ present | ✅ present | ❌ missing | | `feature-b`, retry (~4 min later) | ✅ present | ❌ missing | ❌ missing | **Sharpened finding:** it is specifically **the blob belonging to whichever branch was pushed most recently** that is excluded from the eager set — and it does **not** resolve after ~4 minutes, which rules out an ordinary short-lived prebuild-catch-up race (that would predict the gap closing once the async job completes; it never did across two attempts 4 minutes apart, and the served `bundle_id` was byte-identical both times, meaning the exact same stale cached mpack was served twice with no refresh in between). This points more towards **the prebuild job for the third (`feature-b`) push either never being enqueued, silently failing, or getting stuck** — squarely in the RC-3/RC-4 lineage (job idempotency / ordering), but for a trigger neither MWP-3 nor MWP-4 tested: **a brand-new branch's first push, when it's the *third or later* distinct branch tip in the repo.** `feature-a` (the *second* branch, first-ever sibling) got its own blob eagerly — so the failure mode isn't simply "any new sibling branch," it specifically appeared on the *next* push after that. **Immediate next diagnostic, before any code read:** does pushing a *fourth* commit (to any branch) finally trigger a prebuild that picks up `b.txt`, confirming a "one push behind" lag rather than a permanently stuck/dropped job? This is cheap to test and would further narrow open questions 1–3 without needing repo access to the job queue table. - [x] **Atomic repro 4b — push a 4th commit, does it unstick `b.txt`?** Pushed a 4th commit to `main` on the atomic-4 repo. **✅ RESULT: no, it does not self-heal — and it got worse.** `bundle_id` changed (a new combined mpack was built, now covering 4 commits), but `verify --json` now reports `promised_objects: 3` (`a.txt`, `b.txt`, **and** `fourth.txt`, the brand-new commit's own blob) — only `shared.txt`, the very first blob ever pushed to this repo, remains eager. This rules out a simple "one push behind" lag (which would predict `b.txt` resolving once a subsequent push's indexing completes) and instead suggests whatever caused `b.txt`'s exclusion is a **standing condition for this repo**, not a transient race that time or further activity repairs. - [ ] **Atomic repro 5 — delta fetch (`have != []`) into an existing checkout** vs. `clone` (`have == []`), to answer open question 8 with a clean, minimal case. **Deferred** — superseded in priority by the code trace below, which narrowed the search space enough that a targeted diagnostic test is now more valuable than this generic black-box check. - [x] **Atomic repro 6 (generalized) — code trace of the eager-vs-promised decision and the indexing job**, done via `muse code grep` / `content-grep` / direct reads rather than a new repro. Full findings below. Each atomic repro should be run in a fresh `/tmp` scratch repo (naming convention: `mwp-qa-atomic-N`), with full transport-log output captured (not truncated with `tail`, learned the hard way in the original repro), and its result recorded back into this ticket before moving to the next. **Exit condition for this phase:** either a confirmed root cause (a specific function/line where eager-vs-promised is decided incorrectly, or where the exit code is swallowed), or a clear statement of which atomic repro produced an unexpected result that itself becomes the next thing to instrument. This ticket stays an investigation record — no fix is attempted until the mechanism is understood. --- ## Code trace — where eager-vs-promised is actually decided Full chain, traced via `muse code grep` / `content-grep` / direct reads, **no production code modified**. ### 1. `ObjectState.PROMISED` (`muse/core/object_availability.py`) is generic and reactive, not a decision point ```python def object_state(root, obj_id, promisor_remotes) -> ObjectState: if _object_path_with_fallback(root, obj_id).exists(): return ObjectState.PRESENT if promisor_remotes: return ObjectState.PROMISED return ObjectState.MISSING ``` This is purely: "does the file exist on disk? No, but a remote is configured? Then it's `PROMISED` (benign label), not `MISSING` (real failure)." It says nothing about *why* an object wasn't fetched — it's a downstream label applied by `verify`/`build_mpack` to whatever the fetch/clone path already did or didn't write to disk. **This resolves open question 2 in part: the "promised" label itself lives client-side and is generic; it is not where eager-vs-promised gets decided.** ### 2. `build_mpack`'s promisor-skip logic (`muse/core/mpack.py:781-794`) is client-push-only, confirmed not used server-side ```python promisor_remotes = load_promisor_remotes(repo_root) for oid in sorted(candidate_blob_ids): raw = read_object(repo_root, oid) if raw is None: state = object_state(repo_root, oid, promisor_remotes) if state == ObjectState.PROMISED: logger.debug(...); continue # silently skip raise ValueError(...) # hard error otherwise ``` Confirmed via `content-grep "build_mpack|build_wire_mpack"` across musehub: the server **only ever calls `build_wire_mpack`** (pure serialization of an already-assembled dict) — never `build_mpack` (the BFS-walk-and-skip function above). **This resolves open question 3: the promisor-skip mechanism is a legitimate client-side `muse push` safety valve (don't hard-fail a push just because your local shallow clone is missing a historical blob the remote already has) — it is architecturally absent from the server path entirely.** The server has its own, separate blob-selection logic. ### 3. The real server-side gate — `wire_fetch_mpack` (`musehub/services/musehub_wire_fetch.py`) Lines 681–694 are the load-bearing correctness gate: ```python if new_oids: indexed_q = await session.execute( select(MusehubMPackIndex.entity_id) .where(MusehubMPackIndex.entity_id.in_(list(new_oids))) .where(MusehubMPackIndex.entity_type == "object") ) indexed_oids = {row[0] for row in indexed_q} missing = new_oids - indexed_oids if missing: raise FetchNotIndexedError(len(missing)) # → HTTP 503, MWP-5 retries ``` This correctly 503s when an object has **no index row at all**. Since our repro consistently returns **200**, `b.txt`'s oid **must have an index row** by the time of the fetch. The bug is downstream of this gate: lines 697–820 try three strategies to resolve indexed-but-not-cached oids to actual bytes (`cache_hits` from `MusehubObject.content_cache`, `mpack_hits` by downloading the mpack its index row points to and extracting the blob, and a `legacy_hits` fallback via `backend.get(oid)` direct read). The final assembly line: ```python blob_pairs = [(oid, _all_blob_bytes[oid]) for oid in new_oids if oid in _all_blob_bytes] ``` **silently drops any oid that fails all three strategies** — no error, no warning naming which object or why. This is the exact mechanism of the silent partial mpack. ### 4. `process_mpack_index_job` (`musehub/services/musehub_wire_push.py:1571`) — objects are never stored standalone This job runs once per push, reads that push's own uploaded mpack, and writes `MusehubMPackIndex` rows keyed to `mpack_id = ` — **never a combined/consolidated key**. It also writes `MusehubObject.storage_uri = f"mpack://{mpack_key}"` for every object in that push. **There is no code path anywhere in this function (or found elsewhere) that stores an object as a standalone, individually-addressable key.** An object's bytes are only ever recoverable by re-reading the exact mpack file its index row points to. This means the `legacy_hits` fallback (`backend.get(oid)` — a direct, non-mpack-scoped read) is very likely **always a no-op for any object pushed through this pipeline** — it would only succeed for objects written by some other, genuinely legacy path. ### 5. `process_mpack_gc_job` (`musehub/services/musehub_wire_fetch.py:913`) — real inconsistency found, but confirmed inactive This function consolidates every per-push mpack for a repo into one, then deletes the old `MusehubMPackIndex` rows (pointing them at the new consolidated key instead). Its sibling, `purge_stale_mpack_index_entries` (`musehub_wire_push.py:1133`), has a docstring claiming *"objects with stale entries have valid `s3://` storage_uri and are served directly from S3 without needing the mpack path"* — **this is false** per finding #4 above; the actual writer only ever sets `mpack://`. This is a real bug-in-waiting, but **confirmed via `content-grep` that neither function is called from `worker.py`'s dispatch table** — the worker only actively processes `"mpack.index"` and `"fetch.mpack.prebuild"`; every other job type (including whatever would trigger GC/consolidation) is a no-op (`pass`). **GC/consolidation cannot be the active mechanism for this bug** — it never ran against `gabriel/mwp-qa-atomic-4`. Flagged separately as its own latent-bug finding, not pursued further here. ### 6. The one loose thread — why does `feature-b`'s indexed oid fail all three retrieval strategies? Given #3–#5, the only active job that could index `b.txt` is `feature-b`'s own `mpack.index` job, which — if it ran successfully — should leave `b.txt`'s original push mpack fully intact and retrievable (nothing deletes it). That would make `mpack_hits` succeed. Since it doesn't, one of the following must be true, and **cannot be distinguished by further code reading or black-box HTTP testing alone**: - **(a)** `feature-b`'s own `mpack.index` job never ran or never completed (dropped by a coarse per-`(repo_id, job_type, pending)` dedup collision with an earlier still-pending `mpack.index` job from `main`'s or `feature-a`'s push — the exact RC-3 mechanism MWP-3 fixed, but potentially for a trigger neither MWP-3 nor MWP-4's test suites covered: a third-or-later branch's very first push, in quick succession after two prior pushes). **But** this should make `b.txt`'s oid absent from `indexed_oids` entirely, which the gate in finding #3 would catch and turn into a 503 — which we never observed. This is the part that doesn't yet fit. - **(b)** The job ran and wrote an index row, but something rewrote or corrupted that row's `mpack_id` afterward (no known active code path does this, per findings #4–#5, but an unknown one may exist). - **(c)** The job ran, wrote the correct row, but `backend.get_mpack()` for that specific key transiently or permanently fails for a reason unrelated to indexing (e.g., a storage-backend-level issue with that specific upload) — would require direct storage/backend inspection to confirm. **Resolving this requires either live DB/job-log state for `gabriel/mwp-qa-atomic-4`** (whether `feature-b`'s `mpack.index` job actually ran, and what `MusehubMPackIndex` actually contains for `b.txt`'s oid right now) **or a new deterministic test** that controls job timing precisely — the same methodology MWP-3/MWP-4's own test suites used to confirm their respective root causes before any fix was written. --- ## ⚠️ RETRACTION of the previous "ROOT CAUSE CONFIRMED — escalated" section below The section that follows this note (kept for the record, not deleted — see workspace data-integrity convention of never silently erasing a documented finding) was written after reproducing against **localhost**, but that localhost run was against a **stale `musehub_worker` container whose Docker image dated to 2026-05-15** — over a month old. Nothing in this workspace's documented conventions mentions restarting the worker container (only the `musehub` API container has a documented restart requirement), so it had been silently running old code the entire time. **After restarting both `musehub` and `musehub_worker`** (which also surfaced and required fixing an unrelated, separate schema-migration gap — see "Side finding" below) **and re-running the identical scenario against genuinely current code, both conclusions below turned out to be wrong:** - ~~"Consolidation (`process_mpack_gc_job`) is load-bearing, ticket #114's removal recommendation is wrong"~~ — **wrong.** With the worker actually running current code, `process_fetch_mpack_prebuild_job` correctly builds **one shared mpack across all tips**, exactly as documented (fresh logs: `tips=3 ... built=3`, one shared `mpack_id` written to all three tips' cache rows in a single `~20ms` build). The multi-tip cache-HIT invariant is genuinely satisfiable by current code. musehub#114's original "safe to remove" recommendation was correct; my correction to it was itself wrong and has been un-corrected there. - ~~"On current dev/HEAD, cloning any multi-branch repo now perpetually fails"~~ — **wrong**, same cause. A properly-running worker never needs the `force_build=True` gate to hard-block a real clone — prebuild completes fast enough that the cache is warm by the time a client's own request lands, in every test we ran after the restart. **Side finding, fixed to unblock this re-verification:** the restart also surfaced a genuine, unrelated, pre-existing bug — the `MusehubVersionTag` ORM model (`musehub/db/musehub_repo_models.py:505`) had **no corresponding Alembic migration ever generated**, so any environment restarting from a sufficiently old running-process state would fail `assert_schema_matches_orm`'s startup check. Generated and applied `alembic/versions/0073_add_musehub_version_tags.py` to fix this locally. Worth its own follow-up ticket to check whether staging's DB has the same gap (separate from this investigation). The actual, confirmed-at-the-database-level root cause is below. --- ## 🔴 ROOT CAUSE CONFIRMED — a plain logic bug, not a caching/timing/deployment issue Re-running atomic repro 4's exact scenario against a **freshly restarted** localhost (confirmed current code, confirmed schema in sync) reproduced the identical bug **on the very first attempt** — immediately, no retries, no timing dependency. This ruled out every caching/job-timing/deployment-drift theory in one step: the bug is deterministic and present on current code. ### The mechanism — found and confirmed via a direct database query `musehub_wire_fetch.py:555-559`: ```python want_tip_snap_q = await session.execute( select(MusehubCommitGraph.snapshot_id) .where(MusehubCommitGraph.commit_id.in_(_needed_cids)) .order_by(MusehubCommitGraph.generation.desc()) .limit(1) ) want_tip_snap_id = want_tip_snap_q.scalar_one_or_none() ``` This assumes there is exactly **one** "the tip" for the entire fetch request, and computes `all_oids` (the full set of blobs to include in the response, lines 612-648) from **only that one snapshot's manifest**. Queried `musehub_commit_graph` directly for the three commits in the atomic repro 4 scenario: ``` commit_id generation snapshot_id sha256:5fb426a0... (main) 0 f411f3d3... sha256:78ab4d6f... (feat-a) 1 288ed020... sha256:d44962ef... (feat-b) 1 bcfd655e... ← ties feature-a ``` `feature-a` and `feature-b` are siblings — both children of the same base commit — so they share **generation 1**. `ORDER BY generation DESC LIMIT 1` on a tie is arbitrary; Postgres returned `feature-a`'s snapshot. `all_oids` is built from that snapshot's manifest alone. `feature-b`'s manifest — and its unique blob `b.txt` — is **never looked at, never queried, never attempted.** This is not a resolution failure downstream (the three-strategy `cache_hits`/`mpack_hits`/`legacy_hits` machinery we traced earlier never even gets a chance to try `b.txt`, because it's never added to `all_oids` in the first place). ### The boundary, precisely Triggers whenever **two or more requested tips share the same `generation`** — i.e., any two branches cut from a common ancestor at the same depth, which is the ordinary shape of two feature branches. Does not trigger for a single linear branch (only one tip ever exists, no tie possible) — this is exactly why atomic repros 1–3 (all single-branch) never reproduced it, and why atomic repro 4 (the first sibling-branch test) reproduced it immediately and consistently. Confirmed deterministic, not timing-dependent: explains in hindsight why waiting longer, pushing a 4th commit, or retrying never made a difference in earlier atomic repros — there was never anything to "catch up" on. ### What is still genuinely unknown after this finding - Whether staging's deployed code has this same line — very likely, since this looks like old, foundational logic unrelated to the recent MWP-1..8 campaign, but not yet directly re-confirmed against staging with this specific lens. - What staging's actual deployed commit is — still unverified; now a secondary question rather than the explanation for the corruption itself. - The correct fix shape — does `all_oids` need to become the union of manifests across **every** tip in `want` regardless of generation (the fully general fix), or is the tie-at-max-generation case the only broken one? Needs to be checked before writing a fix — e.g. what happens today when requested tips are at genuinely *different* generations (a deep branch vs a shallow one) — does the shallow one's manifest also get silently dropped, or does `_needed_cids`/`all_oids` handle that case some other way we haven't yet traced? **✅ RESOLVED — fully general, not tie-specific.** Confirmed via three live E2E repros against a freshly-restarted localhost (see below): the bug affects any `want` set spanning more than one branch tip, tied generations or not, including the realistic case of a plain `muse clone` of the default branch when any other branch in the repo is deeper. --- ## ✅ FIXED — Phase 1, TDD ### Test-first (RED) New file `tests/test_multi_tip_manifest_union.py`, three tests each reproducing one confirmed scope boundary, all RED against the pre-fix code: - `MTU_01` — two sibling branches at the **same** generation (a tie): the tie-losing sibling's unique blob is dropped. - `MTU_02` — two sibling branches at **different** generations (no tie, shallower one loses deterministically): the shallower sibling's unique blob is dropped. - `MTU_03` — the **default branch itself** loses its own newest content merely because another branch in the repo is deeper. This is the realistic "just run `muse clone `" case, not a contrived edge case. All three failed with the exact predicted missing blob before the fix. ### The fix `musehub_wire_fetch.py`, immediately after the existing single-tip `want_tip_snap_id` block (left untouched — it still guards a different, valid concern: detecting corrupt `generation` values used by `_walk_commit_delta`'s range scan): added a union step that resolves each tip actually listed in `want` from its own authoritative `MusehubCommit.snapshot_id` (already loaded into `commit_rows`, already the correct source per the existing MWP1_13 comment — "MusehubCommit always has the correct snapshot_id") and unions every tip's manifest into `all_oids`, instead of relying on a single CommitGraph-generation-ranked pick. ```python # musehub#113 fix — union every want-tip's own manifest, not one pick. _pre_union_oids = len(all_oids) for _want_cid in want: _want_commit_row = commit_rows.get(_want_cid) if _want_commit_row is None or not _want_commit_row.snapshot_id: continue _want_snap_entry = snap_map.get(_want_commit_row.snapshot_id) if _want_snap_entry: all_oids.update(v for v in (_want_snap_entry.get("manifest") or {}).values() if v) ``` ### Verified green — unit, regression, and live E2E - `MTU_01`, `MTU_02`, `MTU_03` — all 3 GREEN after the fix. - Adjacent regression suite (all run one file at a time, per workspace convention): `test_mwp1_generation_authority.py` (13), `test_mwp2_walk_fallback.py` (10), `test_fetch_mpack_prebuild.py` (8), `test_wire_fetch_mpack.py` + `test_mpack_fetch_phase3.py` (11), `test_mwp8_cache_invalidation.py` + `test_mwp4_prebuild_ordering.py` + `test_mwp3_job_idempotency.py` (42) — **84 tests, zero regressions.** - Full `pytest -m wire` suite: **56 passed**, matching Phase 0's baseline exactly. - **Live E2E confirmation** against a fresh scratch repo (`gabriel/mwp-qa-local-v4`, no prior cache contamination): pushed `main` (2 commits, `main_unique.txt` as the newest), pushed `feature-deep` (3 commits, deeper than `main`), then a plain `muse clone ` (no `--branch` flag — the realistic default-branch case) returned **no warning**, `blobs_written=5` (the full cross-tip union), and `main_unique.txt` present with correct content. `muse verify --json`: `objects_checked: 2` (exactly `main`'s own 2-file manifest), `promised_objects: 3` (correctly `feature-deep`'s 3 blobs, legitimately irrelevant to `main`'s own checkout — not a bug, the same benign partial-clone deferral behavior confirmed earlier for non-default-branch checkouts), `all_ok: true`. --- ## The unifying finding — why this closes clone/fetch/pull together, and what push needed Before fixing anything further, we asked whether `push`, `fetch`, and `pull` each needed their own separate investigation, or whether the `all_oids` fix already covered them. Verified empirically rather than assumed: - **`muse clone`** is architecturally the *only* verb that ever constructs a `want` set spanning every known branch tip in one request — confirmed by transport-log inspection: a plain `muse clone ` with no `--branch` flag sends `want=N` for all `N` known tips, every time. - **`muse fetch --branch X`** — confirmed via a live transport-log check — always sends `want=1`, scoped to exactly the one named branch. `--all` fetches every configured *remote*, not every branch on one remote. **Fetch was never structurally exposed to the `all_oids` bug at all** — not because it was accidentally safe, but because its `want` set is single-tip by construction. - **`muse pull`** reuses `fetch_mpack` internally (per the architecture recap) — same conclusion. So the `all_oids` fix protects `clone` (the one place the bug could actually manifest) and is a correctness no-op for `fetch`/`pull` (they never sent a multi-tip `want` to begin with). That symmetry prompted a direct question: does the same single-tip-pick anti-pattern exist anywhere else in this function? `content-grep` for the same `ORDER BY generation DESC LIMIT 1` shape found a fourth, previously undiscovered instance — the mirror side, `have_oids` — and it was real (`MTU_04`, RED before the fix). Unlike `all_oids`, undercounting `have_oids` cannot drop content (`new_oids = all_oids - have_oids` can only grow), but it means any client tracking more than one branch (`fetch`/`pull` with a genuinely non-empty, multi-tip `have` set — the normal shape of a repo with several local branches) gets objects it already has redundantly re-sent. Fixed the same way: union across every `have` tip via authoritative `MusehubCommit` rows, and switched off the unsafe raw `manifest_blob` read onto the same safe `snap_map` reconstruction the `all_oids` fix already uses — closing a second, adjacent gap the original BLOB-DEBUG comment had already flagged as a risk but never acted on. **This is the actual unifying principle behind this entire investigation, not just this one ticket:** every root cause found across MWP-1 through MWP-9 (RC-1's generation-0 fallback, RC-2's dead DAG-walk fallback, RC-3/4's job-state races, RC-8's uninvalidated cache, and now this) is the same shape — a *derived, optimized, or cached* value (a `CommitGraph.generation` column, a `MusehubFetchMPackCache` row, a `MusehubMPackIndex` row) being trusted as if it were the *authoritative* source (`MusehubCommit.parent_ids`, `MusehubCommit.snapshot_id`, the actual object bytes) instead of being reconciled against it. `push` was independently checked for the same pattern (`content-grep` for the same query shape across `musehub_wire_push.py` — zero hits) — its own instance of this class of bug (RC-1) was already fixed in MWP-1, and no new instance was found. That is the complete picture for all four verbs: **clone and the two `wire_fetch_mpack` bugs above (fixed), fetch/pull (never exposed, and now carry the have_oids efficiency fix as a bonus), push (already fixed in MWP-1, re-checked here, clean).** --- ## Impact This directly falsifies part of [#58](https://staging.musehub.ai/gabriel/muse/issues/58)'s Claim 1 ("aaronrene can `muse clone ` — get a complete, correct working tree every time"). It also means `muse verify` cannot currently be trusted as a post-clone completeness check, which undermines the general recommendation to use it as a sanity gate. Both of these are more severe than the dedup question `SC_02b` originally set out to answer. ## Out of scope (for this ticket) - Any actual code fix — this ticket is investigation-only until root cause is confirmed (see Plan above). - Re-verifying `SC_02a`/dedup correctness — already independently confirmed correct in this session, not implicated in this bug.