Silent partial clone: required blob missing from working tree, clone exits 0, verify reports all_ok: true

Parent: muse#63 (MWP-9 — spot check of #58's goal-of-done). Discovered during Phase 2 (SC_02b, three-sibling-branch dedup) against the scratch repo gabriel/mwp-qa-1a on staging. Status: investigation, not yet root-caused. No production code has been touched. This ticket is a brain dump of everything observed plus the open questions that block a fix. The plan is to reproduce at the most atomic level possible, narrowing one variable at a time, until we hit either a confirmed root cause or a wall that tells us exactly what to instrument next.

Background — what we were actually testing

Phase 2 of MWP-9 was verifying Claim 2 of #58 ("push uploads only the true delta, never re-uploads objects already on the remote via any ref"). The specific sub-test (SC_02b) was: push two sibling branches (feature-a, feature-b) off a shared main, and confirm the second branch's push dedups correctly against the first.

That part passed, and passed more thoroughly than planned: feature-b's push sent exactly 1 object (its own fb.txt blob), correctly excluding README.md's blob despite 4 incremental versions of it existing across the shared ancestor chain with both main and feature-a. RC-7 / MWP-7's sibling-ref dedup fix holds up under a stronger test than the original two-branch case it shipped with.

What broke instead was cloning the branch we'd just proven the push logic handled correctly.

What we know (chronological brain dump)

1. The exact repro sequence

All against https://staging.musehub.ai/gabriel/mwp-qa-1a (scratch repo, created fresh for this QA campaign):

commit 1 (main): "init"       — README.md = "hello"
commit 2 (main): "v2"         — README.md = "hello\nv2"
commit 3 (main): "a"          — README.md = "hello\nv2\na"
commit 4 (main): "b"          — README.md = "hello\nv2\na\nb"
  push staging main   → commits_sent=1, objects_sent=1 (per commit, each pushed individually as they were made)
branch feature-a from main tip (commit 4)
commit 5 (feature-a): "feature a"  — adds fa.txt = "fa\n"
  push staging feature-a → commits_sent=1, objects_sent=1  ✅ correct dedup
checkout main (back to commit 4), branch feature-b from main tip
commit 6 (feature-b): "feature b"  — adds fb.txt = "fb\n"
  push staging feature-b → commits_sent=1, objects_sent=1  ✅ correct dedup

rm -rf /tmp/mwp-qa-2b-clone-retry
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2b-clone-retry --branch feature-b --json

2. The failure, verbatim

[transport] unpackb  commits=6  snaps=6  blobs=2  t=0ms
[clone] apply_mpack DONE  t=12ms  blobs_written=2 blobs_skipped=0  commits_written=6
[mpack] fetch/mpack: 0.78s  blobs: 2  commits: 6
[clone] apply_manifest START  (working tree restore)
⚠️ clone: working tree partially restored — apply_manifest: 1 object(s) missing
   from the local store ('fb.txt'). The working tree cannot be fully restored.
   Fetch the missing objects from the remote before retrying.
[clone] apply_manifest DONE  t=7ms
{"...", "exit_code": 0, "status": "cloned", "commits_received": 6, "blobs_written": 2,
 "skipped_blobs": 0, "head": "sha256:005bc9e6...", "domain": "code", "dry_run": false}

fb.txt is completely absent from disk (confirmed via ls, not just empty). exit_code is 0. status is "cloned", not an error state. 100% reproducible — ran it twice in separate fresh directories, identical result both times.

3. Ruled out: manifest correctness, tool misuse, and (probably) content-store corruption

• The commit's manifest correctly maps fb.txt → sha256:b566176258f9d8.... Confirmed via muse hash-object fb.txt on the original working copy — the manifest's declared object ID is exactly right. This is not a metadata/manifest bug.
• --branch feature-b is the documented, correct clone flag (checked muse clone --help).
• Initial alarm: running muse cat-object sha256:<id> --json on 6 of the 13 objects in the local store returned SHA-256 integrity-check failures ("expected X, got Y") — this first read as store corruption. This was almost certainly a false alarm from misusing cat-object on non-blob objects. Running the actual sanctioned integrity tool, muse verify --json, on the same clone reports all_ok: true, failures: [], and critically promised_objects: 5 — i.e., those 6 "corrupt-looking" objects are very likely placeholder/stub entries for a lazy/partial object model (cat-object appears to hash the placeholder bytes directly rather than recognizing the stub and reporting it as not-yet-materialized, which is itself worth separate scrutiny — see open question 4 below — but is not the primary bug).

4. The actual anomaly: a consistent "2 eager blobs" ceiling regardless of target

Every single clone/fetch run during this QA session — regardless of which branch was requested or how many files that branch's manifest actually needed — reported exactly blobs=2 materialized eagerly, with the remainder tracked as promised_objects:

This strongly suggests the server's combined-mpack / prebuild mechanism has a fixed or coincidentally-fixed eager-blob inclusion count, and nothing guarantees the blob(s) required by the specifically requested want tip's own manifest are always in that eager set. When they're not, the client's apply_manifest step correctly detects the gap and prints a warning — but the overall clone command still reports success.

5. muse verify --json output on the broken clone (full)

{
  "repo_id": "sha256:7720f84c823f423f1de706cd9dddefc2e82e98f3301eced36c35ddd8414abc5d",
  "refs_checked": 3,
  "commits_checked": 6,
  "snapshots_checked": 6,
  "objects_checked": 1,
  "signatures_checked": 6,
  "shallow_commits": 0,
  "promised_objects": 5,
  "is_shallow": false,
  "promisor_remotes": ["origin"],
  "all_ok": true,
  "nothing_checked": false,
  "check_objects": true,
  "strict": false,
  "branch": null,
  "failures": []
}

Notable: promisor_remotes: ["origin"] — but the remote we configured and pushed/cloned through was named staging, not origin. Either "origin" is a hardcoded/generic label unrelated to the actual remote name (cosmetic), or there's a mismatch worth checking (see open question 7).

all_ok: true despite feature-b's own checked-out HEAD requiring an object that is not present and is only "promised." This means verify's notion of "ok" does not currently check whether promised objects are needed by the actively-checked-out branch — it appears to only check structural consistency of commits/snapshots/signatures, not working-tree completeness.

6. Confirmed NOT the bug

• Sibling-ref dedup (RC-7/MWP-7): correct, verified independently — see §1, feature-b's push sent exactly 1 object.
• Manifest content: correct — fb.txt's declared object ID matches its real content hash.
• CLI flag usage: correct — --branch feature-b is the documented flag.

What we don't know — open questions to close before any fix is attempted

1. What determines the eager-blob cap? Is "2" a hardcoded constant, a config value, or an artifact specific to this session's exact push history shape (e.g., "however many new blobs were added since the last successful prebuild")? Needs a code-level answer, not just observed behavior.

2. Where does the "promised object" concept live in the codebase, and what module decides eager-vs-promised for a given blob during mpack assembly? (Likely somewhere in musehub_wire_fetch.py's combined-mpack / prebuild path, or a client-side promisor-remote layer in muse — needs muse code grep "promised" / muse code grep "promisor" across both repos to locate.)

3. Is the promisor/lazy-object model intentional and documented, or is it an accidental byproduct of MWP-3/4/8's job/cache machinery that was never meant to leak into the object-completeness contract for an actively-requested clone? If intentional, where is the design doc for it? (Nothing in #58 or MWP-1..8's own docs mentions a partial-clone / promisor-object feature — if this is by design, it predates or is orthogonal to the MWP campaign and no one currently on this thread has full context on its intended contract.)

4. Why does apply_manifest correctly detect the missing object and print a clear warning, yet the parent clone command still returns exit_code: 0 and "status": "cloned"? Where exactly does the warning get swallowed instead of propagating to a non-zero exit / error status? This is the most actionable, narrowly-scoped part of the whole investigation — a bug in exit-code propagation is a much smaller fix than a redesign of the promisor model, if that's all this turns out to be.

5. Why does muse verify --json report all_ok: true even though the actively checked-out branch requires a promised (not-yet-materialized) object? Does verify deliberately treat promised objects as fine by definition (correct for a lazy clone that will fetch on demand), and is the actual gap that on-demand fetch never happens when the working tree is restored? I.e., is the real bug not in verify at all, but in apply_manifest failing to trigger a promised-object fetch before giving up?

6. Is a sibling branch a necessary precondition, or would any clone requesting a blob added since the last successful prebuild reproduce this, even on a single-branch repo? (main's own clone in this session happened not to need any blob outside the eager set — but that could be coincidence, not evidence the single-branch case is safe.)

7. Is promisor_remotes: ["origin"] a hardcoded/generic label, or does it reveal a real mismatch (our remote was named staging, not origin) that's worth checking as a contributing factor?

8. Does an explicit muse fetch (delta path, have != []) into an existing checkout correctly retrieve a promised object that a clone left missing, or does the delta path hit the identical gap? (Started testing this during the investigation but didn't reach a clean from-scratch delta-fetch scenario — needs a dedicated atomic repro.)

9. Does muse checkout <branch> (switching branches within an existing clone, rather than clone --branch at creation time) hit the same promised-object gap, or does that code path differ from clone's built-in apply_manifest?

10. Is cat-object's integrity-check failure on promised/stub objects itself a separate, smaller bug (i.e., should cat-object recognize a promised-but-unmaterialized object and say so clearly, rather than reporting a misleading "SHA-256 integrity check failed / store may be corrupt" message that reads as far more alarming than "not fetched yet")? This caused real confusion during triage and could confuse anyone else who reaches for cat-object as a debugging tool.

Plan — reproduce at the most atomic level possible

The three-sibling-branch, six-commit, two-feature-branch scratch repo used to discover this has too many moving parts to isolate the trigger. Before touching any production code, narrow the repro one variable at a time:

• [ ] Atomic repro 1 — single branch, single push, single clone. muse init → one commit with one file → push → clone (no branches at all). Does this ever produce a promised classification for the only blob that exists? (Expected: no, since this worked for the main clone in the original repro — but confirm on a truly minimal repo, not one with 4 prior commits.)
• [ ] Atomic repro 2 — two commits, two distinct blobs, single branch, immediate clone. Does the second (newest) blob ever get marked promised on a branch with no siblings at all? This isolates whether sibling branches are required to trigger the bug, or whether any "just-added blob" can be miscategorized (open question 6).
• [ ] Atomic repro 3 — reproduce with a controlled N-blob history, incrementing N one at a time (3, 4, 5, 6, 7 distinct blobs across commits) on a single branch, to determine whether "2 eager blobs" is truly a fixed ceiling (independent of N) or scales with something else (open question 1).
• [ ] Atomic repro 4 — two sibling branches, minimal (2 commits total: 1 shared + 1 per branch), to isolate whether the sibling-branch structure itself (as opposed to blob count) is the triggering factor.
• [ ] Atomic repro 5 — delta fetch (have != []) into an existing checkout vs. clone (have == []), to answer open question 8 with a clean, minimal case.
• [ ] Atomic repro 6 — instrument or trace the exit-code path for apply_manifest's warning to find exactly where/why it doesn't propagate to a non-zero clone exit code (open question 4) — this can likely be answered by reading code (muse code grep "partially restored" / muse code impact on apply_manifest) without needing a new repro.

Each atomic repro should be run in a fresh /tmp scratch repo (naming convention: mwp-qa-atomic-N), with full transport-log output captured (not truncated with tail, learned the hard way in the original repro), and its result recorded back into this ticket before moving to the next.

Exit condition for this phase: either a confirmed root cause (a specific function/line where eager-vs-promised is decided incorrectly, or where the exit code is swallowed), or a clear statement of which atomic repro produced an unexpected result that itself becomes the next thing to instrument. This ticket stays an investigation record — no fix is attempted until the mechanism is understood.

Impact

This directly falsifies part of #58's Claim 1 ("aaronrene can muse clone <url> — get a complete, correct working tree every time"). It also means muse verify cannot currently be trusted as a post-clone completeness check, which undermines the general recommendation to use it as a sanity gate. Both of these are more severe than the dedup question SC_02b originally set out to answer.

Out of scope (for this ticket)

• Any actual code fix — this ticket is investigation-only until root cause is confirmed (see Plan above).
• Re-verifying SC_02a/dedup correctness — already independently confirmed correct in this session, not implicated in this bug.