gabriel / muse public
Closed #35 Bug
filed by aaronrene human · 134 days ago

bug: rc11 clone exits 0 but writes 0 commits after snapshot hash mismatches

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Summary

Muse CLI 0.2.0rc11 no longer crashes on the apply_mpack missing-snapshot path, but a fresh clone of the public gabriel/muse staging repo still does not functionally complete. The command exits 0 while writing 0 commits, 0 snapshots, and no working tree.

This blocks any trust decision that depends on muse clone / muse pull proving staging object-store integrity after the rc11 content-addressable mpack / byte-range blob-index rollout.

Environment

  • Client: muse 0.2.0rc11
  • Hub: https://staging.musehub.ai
  • Target repo: gabriel/muse (public)
  • Run time: 2026-06-12T12:24Z

Reproduction

rm -rf /tmp/muse-rc11-clonetest
cd /tmp
muse clone https://staging.musehub.ai/gabriel/muse muse-rc11-clonetest
cd /tmp/muse-rc11-clonetest
muse status
muse log
muse pull origin main

Observed result

muse clone exits 0, but the mpack application reports zero committed history and the working tree is not restored:

[clone] apply_mpack START  blobs=99 commits=1325 snaps=1303
⚠️ apply_mpack: snapshot sha256:ef2ecaf5098e7 hash mismatch (reconstructed=sha256:be9d5d3870b85) — skipped
...
⚠️ apply_mpack: commit sha256:78c1b9e061a3 skipped — snapshot sha256:1b8f5633e00c was not in this mpack and is not in the local store (server sent snaps=0). The next pull will re-request this commit with its snapshot.
⚠️ apply_mpack: commit sha256:18e585c043b10139697a2d40412cf49b2efd39ac6366827a41299037fc915f4c skipped — parent not in mpack or local store
[clone] apply_mpack DONE  t=284ms  blobs_written=99 blobs_skipped=0  commits_written=0
[mpack] fetch/mpack: 33.48s  blobs: 99  commits: 0
[clone] apply_manifest START  (working tree restore)
⚠️ clone: commit sha256:53386fd8a81ed8223bf9274bfbc201141ea13cfe6bc6dfb154aace6a787d2276 not found after apply_mpack — working tree not restored
[clone] apply_manifest DONE  t=0ms
✅ Cloned into 'muse-rc11-clonetest' — 0 commit(s), 99 blob(s), domain=code, branch=main (sha256:53386fd8a81ed8223bf9274bfbc201141ea13cfe6bc6dfb154aace6a787d2276)

Post-clone state:

muse status
# On branch main
# Your branch is up to date with 'origin/main'.
# Nothing to commit, working tree clean

muse log
# (no commits)

.muse/refs/heads/main -> sha256:53386fd8a81ed8223bf9274bfbc201141ea13cfe6bc6dfb154aace6a787d2276
.muse/commits count -> 0
.muse/snapshots count -> 0
top-level working tree entries -> only .muse

A follow-up pull does not repair the skipped commit/snapshot state:

muse pull origin main
Already up to date.
✅ Fetched 0 commit(s), 0 new blob(s) from origin/main (sha256:53386fd8a81ed8223bf9274bfbc201141ea13cfe6bc6dfb154aace6a787d2276)

post-pull:
commits=0
snapshots=0
working tree still empty

Expected result

A successful clone of gabriel/muse should materialize the commit DAG, snapshots, and working tree. If the server sends unusable snapshots, the command should not report a successful clone with "up to date" status while the local repo contains no commits.

Security / release-gate impact

This is independent of the private-repo visibility issue (#87), which returned non-200 on all unauthenticated SSR/raw probes during the same session. However, this defect blocks rc11_backfill_confirmed for downstream private-repo staging gates: a corrupt or inconsistent mpack/backfill state cannot be used as proof that staging object storage is safe before proprietary history is pushed.

Acceptance criteria

  • Fresh muse clone https://staging.musehub.ai/gabriel/muse writes the expected commits/snapshots and restores the working tree.
  • Follow-up muse pull origin main can recover any snapshot that was intentionally skipped by the guard.
  • A clone that writes 0 commits for a non-empty remote does not exit as a successful completed clone.
  • Regression test covers snapshot hash mismatch / skipped snapshot recovery for apply_mpack.
◎ Activity1
●
aaronrene opened this issue 134 days ago
○
gabriel 2 days ago

Resolved — this was NOT outdated, it's a real, live bug on current code

Commit: sha256:a5f8a020b53ddab68f231cf27489fe865525f85ee924133034262d39b332076e (branch fix/35-clone-pull-snapshot-hash-mismatch, fast-forward merged dev→main, pushed to local/staging/production).

The ticket was suggested as a likely-outdated close candidate. Before closing, I audited the current apply_mpack/clone/pull code and confirmed — with a live reproduction, not just code reading — that the exact bug still reproduces on 0.2.1rc12. It was never actually fixed; only the earlier rc11 crash (mentioned in the ticket's own first sentence) had been addressed.

  1. muse/core/mpack.py — apply_mpack's snapshot-writing loop: when _apply_snapshot_deltas's delta reconstruction fails (hash mismatch, invalid object IDs, unresolvable parent chain), the snapshot ID was simply absent from resolved_manifests, and the main loop did a bare continue — never recording it in skipped_snapshot_ids, which was only ever populated by the separate "references a failed blob object" check. ApplyResult.skipped_snapshots stayed empty even though a snapshot (and every commit depending on it) had just been silently dropped. Reproduced directly via a standalone script before touching any code: commits_written=0, skipped_snapshots=[] — zero trace anywhere.

  2. muse/cli/commands/clone.py — found the smoking gun: the branch-ref-writing loop had a comment literally reading "Only advance refs for branches whose tip commit landed cleanly" — but the code never implemented that check. write_branch_ref and set_remote_head ran unconditionally for every branch the server advertised, regardless of whether apply_mpack actually wrote that branch's tip commit. This is the bug: the local branch ref ends up pointing at a commit that doesn't exist in the store (0 commits, muse log shows nothing) while muse status reports a clean, up-to-date tree. Also fixed the exit-code computation, which previously only counted blob-level failures — a 0-commit clone with 0 corrupted blobs still reported "✅ Cloned" success.

  3. muse/cli/commands/pull.py — the same unconditional-advance bug, but for the remote-tracking ref. This is what made the corruption permanent: once the tracking ref wrongly equals the remote's head, the next pull's own short-circuit (already_known == remote_commit_id → skip the fetch entirely) means the skipped commit/snapshot can never be re-requested — exactly the ticket's "follow-up pull does not repair the skipped state" symptom.

Both clone.py and pull.py's new guards defensively treat a malformed/legacy-format commit ID as "not written" rather than letting validate_object_id's ValueError propagate and abort the whole command.

Verification

  • 9 new/updated tests: PF-5/PF-5b in tests/test_apply_mpack_partial_failure.py (hash-mismatch skip is recorded; a retry with the correct delta for the same snapshot_id succeeds — proving the skip is recoverable), TestSnapshotHashMismatchGuard (3 tests) in tests/test_cmd_clone_hardening.py, TestTrackingRefNotAdvancedOnSkippedSnapshot (2 tests) in tests/test_cmd_pull_hardening.py, plus 2 pre-existing tests updated where their old assertions literally described the bug's own undercounting/unconditional-write behavior.
  • Confirmed via genuine RED: reverted all three fixes in turn — the clone.py revert reproduced the ticket's exact bug (branch ref written pointing at a commit ID that was never stored) — then restored and confirmed GREEN.
  • 579-test regression sweep across every mpack/clone/pull-relevant suite — zero regressions.
  • Live-verified the normal (non-corrupted) clone/push/pull path end-to-end against a real local MuseHub server: pushed 2 commits, cloned cleanly, working tree restored correctly, muse log/muse status both accurate.

Acceptance criteria

  • ✅ Fresh clone writes the expected commits/snapshots and restores the working tree — verified live for the normal path; the corrupted-snapshot path is now correctly reported as partial, not silently as success.
  • ✅ Follow-up pull can recover a skipped snapshot — proven at the apply_mpack layer (PF-5b) and structurally guaranteed by fix #3 (the tracking ref no longer advances past the gap, so the next pull's Phase 1 check correctly sees it's NOT up to date and re-fetches).
  • ✅ A clone that writes 0 commits for a non-empty remote no longer exits as a successful completed clone — exit code is PARTIAL, JSON status is "partial", new skipped_snapshots/commits_missing fields make the shortfall explicit.
  • ✅ Regression test covers snapshot hash mismatch / skipped-snapshot recovery for apply_mpack — PF-5/PF-5b.

Thank you for flagging this as a close candidate — the investigation it prompted surfaced a genuinely severe, still-live data-integrity bug that would otherwise have stayed invisible.

✓
closed this issue 2 days ago