# Silent partial clone: required blob missing from working tree, `clone` exits 0, `verify` reports `all_ok: true` > **Parent:** [muse#63](https://staging.musehub.ai/gabriel/muse/issues/63) (MWP-9 — spot check of #58's goal-of-done). Discovered during Phase 2 (`SC_02b`, three-sibling-branch dedup) against the scratch repo `gabriel/mwp-qa-1a` on staging. > **Status: investigation, not yet root-caused.** No production code has been touched. This ticket is a brain dump of everything observed plus the open questions that block a fix. The plan is to reproduce at the most atomic level possible, narrowing one variable at a time, until we hit either a confirmed root cause or a wall that tells us exactly what to instrument next. --- ## Background — what we were actually testing Phase 2 of MWP-9 was verifying Claim 2 of #58 ("push uploads only the true delta, never re-uploads objects already on the remote via any ref"). The specific sub-test (`SC_02b`) was: push two sibling branches (`feature-a`, `feature-b`) off a shared `main`, and confirm the second branch's push dedups correctly against the first. **That part passed, and passed more thoroughly than planned:** `feature-b`'s push sent exactly 1 object (its own `fb.txt` blob), correctly excluding README.md's blob despite 4 incremental versions of it existing across the shared ancestor chain with both `main` and `feature-a`. RC-7 / MWP-7's sibling-ref dedup fix holds up under a stronger test than the original two-branch case it shipped with. **What broke instead was cloning the branch we'd just proven the push logic handled correctly.** --- ## What we know (chronological brain dump) ### 1. The exact repro sequence All against `https://staging.musehub.ai/gabriel/mwp-qa-1a` (scratch repo, created fresh for this QA campaign): ``` commit 1 (main): "init" — README.md = "hello" commit 2 (main): "v2" — README.md = "hello\nv2" commit 3 (main): "a" — README.md = "hello\nv2\na" commit 4 (main): "b" — README.md = "hello\nv2\na\nb" push staging main → commits_sent=1, objects_sent=1 (per commit, each pushed individually as they were made) branch feature-a from main tip (commit 4) commit 5 (feature-a): "feature a" — adds fa.txt = "fa\n" push staging feature-a → commits_sent=1, objects_sent=1 ✅ correct dedup checkout main (back to commit 4), branch feature-b from main tip commit 6 (feature-b): "feature b" — adds fb.txt = "fb\n" push staging feature-b → commits_sent=1, objects_sent=1 ✅ correct dedup rm -rf /tmp/mwp-qa-2b-clone-retry muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2b-clone-retry --branch feature-b --json ``` ### 2. The failure, verbatim ``` [transport] unpackb commits=6 snaps=6 blobs=2 t=0ms [clone] apply_mpack DONE t=12ms blobs_written=2 blobs_skipped=0 commits_written=6 [mpack] fetch/mpack: 0.78s blobs: 2 commits: 6 [clone] apply_manifest START (working tree restore) ⚠️ clone: working tree partially restored — apply_manifest: 1 object(s) missing from the local store ('fb.txt'). The working tree cannot be fully restored. Fetch the missing objects from the remote before retrying. [clone] apply_manifest DONE t=7ms {"...", "exit_code": 0, "status": "cloned", "commits_received": 6, "blobs_written": 2, "skipped_blobs": 0, "head": "sha256:005bc9e6...", "domain": "code", "dry_run": false} ``` `fb.txt` is **completely absent from disk** (confirmed via `ls`, not just empty). **`exit_code` is 0. `status` is `"cloned"`, not an error state.** 100% reproducible — ran it twice in separate fresh directories, identical result both times. ### 3. Ruled out: manifest correctness, tool misuse, and (probably) content-store corruption - The commit's manifest correctly maps `fb.txt` → `sha256:b566176258f9d8...`. Confirmed via `muse hash-object fb.txt` on the *original* working copy — the manifest's declared object ID is exactly right. This is not a metadata/manifest bug. - `--branch feature-b` is the documented, correct clone flag (checked `muse clone --help`). - Initial alarm: running `muse cat-object sha256: --json` on 6 of the 13 objects in the local store returned **SHA-256 integrity-check failures** ("expected X, got Y") — this first read as store corruption. **This was almost certainly a false alarm from misusing `cat-object` on non-blob objects.** Running the *actual* sanctioned integrity tool, `muse verify --json`, on the same clone reports `all_ok: true`, `failures: []`, and critically **`promised_objects: 5`** — i.e., those 6 "corrupt-looking" objects are very likely placeholder/stub entries for a **lazy/partial object model** (`cat-object` appears to hash the placeholder bytes directly rather than recognizing the stub and reporting it as not-yet-materialized, which is itself worth separate scrutiny — see open question 4 below — but is not the primary bug). ### 4. The actual anomaly: a consistent "2 eager blobs" ceiling regardless of target Every single clone/fetch run during this QA session — regardless of which branch was requested or how many files that branch's manifest actually needed — reported **exactly `blobs=2`** materialized eagerly, with the remainder tracked as `promised_objects`: | Clone target | Blobs actually needed by manifest | Blobs eagerly materialized | Promised | |---|---|---|---| | `main` (4 commits, README only) | 1 (README latest) | 2 | — (not observed to fail, README happened to be eager) | | `feature-b` (adds `fb.txt`) | 2 (README latest + fb.txt) | 2 | 5 — **but `fb.txt` was among the promised, not the eager, set** | This strongly suggests the server's combined-mpack / prebuild mechanism has a fixed or coincidentally-fixed eager-blob inclusion count, and nothing guarantees the blob(s) required by the **specifically requested `want` tip's own manifest** are always in that eager set. When they're not, the client's `apply_manifest` step correctly *detects* the gap and prints a warning — but the overall `clone` command still reports success. ### 5. `muse verify --json` output on the broken clone (full) ```json { "repo_id": "sha256:7720f84c823f423f1de706cd9dddefc2e82e98f3301eced36c35ddd8414abc5d", "refs_checked": 3, "commits_checked": 6, "snapshots_checked": 6, "objects_checked": 1, "signatures_checked": 6, "shallow_commits": 0, "promised_objects": 5, "is_shallow": false, "promisor_remotes": ["origin"], "all_ok": true, "nothing_checked": false, "check_objects": true, "strict": false, "branch": null, "failures": [] } ``` Notable: `promisor_remotes: ["origin"]` — but the remote we configured and pushed/cloned through was named `staging`, not `origin`. Either `"origin"` is a hardcoded/generic label unrelated to the actual remote name (cosmetic), or there's a mismatch worth checking (see open question 7). `all_ok: true` despite `feature-b`'s own checked-out HEAD requiring an object that is not present and is only "promised." This means `verify`'s notion of "ok" does not currently check whether promised objects are needed by the actively-checked-out branch — it appears to only check structural consistency of commits/snapshots/signatures, not working-tree completeness. ### 6. Confirmed NOT the bug - **Sibling-ref dedup (RC-7/MWP-7):** correct, verified independently — see §1, `feature-b`'s push sent exactly 1 object. - **Manifest content:** correct — `fb.txt`'s declared object ID matches its real content hash. - **CLI flag usage:** correct — `--branch feature-b` is the documented flag. --- ## What we don't know — open questions to close before any fix is attempted 1. **What determines the eager-blob cap?** Is "2" a hardcoded constant, a config value, or an artifact specific to this session's exact push history shape (e.g., "however many *new* blobs were added since the last successful prebuild")? Needs a code-level answer, not just observed behavior. 2. **Where does the "promised object" concept live in the codebase**, and what module decides eager-vs-promised for a given blob during mpack assembly? (Likely somewhere in `musehub_wire_fetch.py`'s combined-mpack / prebuild path, or a client-side promisor-remote layer in `muse` — needs `muse code grep "promised"` / `muse code grep "promisor"` across both repos to locate.) 3. **Is the promisor/lazy-object model intentional and documented**, or is it an accidental byproduct of MWP-3/4/8's job/cache machinery that was never meant to leak into the object-completeness contract for an actively-requested clone? If intentional, where is the design doc for it? (Nothing in #58 or MWP-1..8's own docs mentions a partial-clone / promisor-object feature — if this is by design, it predates or is orthogonal to the MWP campaign and no one currently on this thread has full context on its intended contract.) 4. **Why does `apply_manifest` correctly detect the missing object and print a clear warning, yet the parent `clone` command still returns `exit_code: 0` and `"status": "cloned"`?** Where exactly does the warning get swallowed instead of propagating to a non-zero exit / error status? This is the most actionable, narrowly-scoped part of the whole investigation — a bug in exit-code propagation is a much smaller fix than a redesign of the promisor model, if that's all this turns out to be. 5. **Why does `muse verify --json` report `all_ok: true`** even though the actively checked-out branch requires a promised (not-yet-materialized) object? Does `verify` deliberately treat promised objects as fine by definition (correct for a *lazy* clone that will fetch on demand), and is the actual gap that **on-demand fetch never happens** when the working tree is restored? I.e., is the real bug not in `verify` at all, but in `apply_manifest` failing to trigger a promised-object fetch before giving up? 6. **Is a sibling branch a necessary precondition**, or would *any* clone requesting a blob added since the last successful prebuild reproduce this, even on a single-branch repo? (`main`'s own clone in this session happened not to need any blob outside the eager set — but that could be coincidence, not evidence the single-branch case is safe.) 7. **Is `promisor_remotes: ["origin"]` a hardcoded/generic label**, or does it reveal a real mismatch (our remote was named `staging`, not `origin`) that's worth checking as a contributing factor? 8. **Does an explicit `muse fetch` (delta path, `have != []`) into an existing checkout correctly retrieve a promised object that a `clone` left missing**, or does the delta path hit the identical gap? (Started testing this during the investigation but didn't reach a clean from-scratch delta-fetch scenario — needs a dedicated atomic repro.) 9. **Does `muse checkout ` (switching branches within an existing clone, rather than `clone --branch` at creation time) hit the same promised-object gap**, or does that code path differ from `clone`'s built-in `apply_manifest`? 10. **Is `cat-object`'s integrity-check failure on promised/stub objects itself a separate, smaller bug** (i.e., should `cat-object` recognize a promised-but-unmaterialized object and say so clearly, rather than reporting a misleading "SHA-256 integrity check failed / store may be corrupt" message that reads as far more alarming than "not fetched yet")? This caused real confusion during triage and could confuse anyone else who reaches for `cat-object` as a debugging tool. --- ## Plan — reproduce at the most atomic level possible The three-sibling-branch, six-commit, two-feature-branch scratch repo used to discover this has too many moving parts to isolate the trigger. Before touching any production code, narrow the repro one variable at a time: - [ ] **Atomic repro 1 — single branch, single push, single clone.** `muse init` → one commit with one file → push → clone (no branches at all). Does this ever produce a `promised` classification for the only blob that exists? (Expected: no, since this worked for the `main` clone in the original repro — but confirm on a truly minimal repo, not one with 4 prior commits.) - [ ] **Atomic repro 2 — two commits, two distinct blobs, single branch, immediate clone.** Does the second (newest) blob ever get marked `promised` on a branch with no siblings at all? This isolates whether *sibling branches* are required to trigger the bug, or whether *any* "just-added blob" can be miscategorized (open question 6). - [ ] **Atomic repro 3 — reproduce with a controlled N-blob history**, incrementing N one at a time (3, 4, 5, 6, 7 distinct blobs across commits) on a single branch, to determine whether "2 eager blobs" is truly a fixed ceiling (independent of N) or scales with something else (open question 1). - [ ] **Atomic repro 4 — two sibling branches, minimal (2 commits total: 1 shared + 1 per branch)**, to isolate whether the sibling-branch structure itself (as opposed to blob count) is the triggering factor. - [ ] **Atomic repro 5 — delta fetch (`have != []`) into an existing checkout** vs. `clone` (`have == []`), to answer open question 8 with a clean, minimal case. - [ ] **Atomic repro 6 — instrument or trace the exit-code path** for `apply_manifest`'s warning to find exactly where/why it doesn't propagate to a non-zero clone exit code (open question 4) — this can likely be answered by reading code (`muse code grep "partially restored"` / `muse code impact` on `apply_manifest`) without needing a new repro. Each atomic repro should be run in a fresh `/tmp` scratch repo (naming convention: `mwp-qa-atomic-N`), with full transport-log output captured (not truncated with `tail`, learned the hard way in the original repro), and its result recorded back into this ticket before moving to the next. **Exit condition for this phase:** either a confirmed root cause (a specific function/line where eager-vs-promised is decided incorrectly, or where the exit code is swallowed), or a clear statement of which atomic repro produced an unexpected result that itself becomes the next thing to instrument. This ticket stays an investigation record — no fix is attempted until the mechanism is understood. --- ## Impact This directly falsifies part of [#58](https://staging.musehub.ai/gabriel/muse/issues/58)'s Claim 1 ("aaronrene can `muse clone ` — get a complete, correct working tree every time"). It also means `muse verify` cannot currently be trusted as a post-clone completeness check, which undermines the general recommendation to use it as a sanity gate. Both of these are more severe than the dedup question `SC_02b` originally set out to answer. ## Out of scope (for this ticket) - Any actual code fix — this ticket is investigation-only until root cause is confirmed (see Plan above). - Re-verifying `SC_02a`/dedup correctness — already independently confirmed correct in this session, not implicated in this bug.