gabriel / musehub public
Closed #93
filed by gabriel human · 123 days ago

musehub drops snapshot directories (and empties some manifests) on push → clones fail with snapshot hash mismatch

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Summary

musehub stores some pushed snapshots with their directories dropped (and, via a separate legacy delta-base bug, some with emptied manifests). Because a snapshot's id is hash_snapshot(manifest, directories), a stored snapshot whose directories (or manifest) were lost no longer reproduces its snapshot_id when a client re-hashes it on clone. The client rejects the snapshot, drops its commit, and every descendant fails parent not in mpack — yielding an empty working tree. This is why muse clone https://staging.musehub.ai/gabriel/muse fails while the local source repo and a localhost clone are clean.

This is not a muse code migrate bug and not corruption in the muse object store. The source repo is correct; the corruption is musehub-side, in pushed/stored data.

Evidence

For gabriel/muse on staging, the affected snapshots:

snapshot_id (prefix) musehub entry_count musehub n_dirs correct (local raw object)
eea07a1f… 1062 0 1062 manifest + 31 directories
708d5734… 0 0 full manifest + dirs
3d5ae8b5… 0 0 full manifest + dirs
edd649a9… 0 0 full manifest + dirs

Proof it's a directories (not manifest) problem for eea07a1f:

RAW local object : entries=1062  dirs=31  hash_snapshot(manifest, dirs) = sha256:eea07a1f…  ✓ (== id)
served by staging: entries=1062  dirs=0   hash_snapshot(manifest, [])   = sha256:3305e95e…  ✗
manifests are byte-identical (0 differing entries)

So the manifest is intact; only the directories field was lost in storage, which alone breaks the snapshot id.

Two distinct defects

  1. Directories dropped on push/store. eea07a1f has a full 1062-entry manifest but n_dirs=0 on staging (should be 31). The push path or the snapshot serialization is not persisting the directories array (note: directories are NOT fully derivable from manifest paths — directories_from_manifest yields a different set, so they must be stored, not re-derived).
  2. Manifests emptied (legacy delta-base bug). 708d5734/3d5ae8b5/edd649a9 are stored with entry_count=0. This is the wire-push base-resolution path resolving an external delta-only parent to {} (tracked separately; the serving-side reconstruction in _snap_row_to_wire_s3 mitigates these on read, but does not restore lost directories).

Impact

  • muse clone of any repo containing such a snapshot fails: snapshot rejected → commit dropped → parent not in mpack cascade → empty working tree.
  • Affects the official gabriel/muse repo on staging (HEAD unclonable).

Repair (operator)

The POST /{owner}/{slug}/repair-snapshot endpoint (wire_repair_snapshot) validates hash_snapshot(manifest, directories) == snapshot_id and stores the corrected record. The source repo holds the correct (manifest, directories). Repair driven by a signed request (muse sign request … --body-file payload.json) with the raw manifest+directories read from the source repo's object (NOT muse read-snapshot, which currently drops directories — see below).

Fixes

  • Push/store path must persist the snapshot directories array end-to-end.
  • Verify a snapshot's stored content reproduces its snapshot_id at unpack time (reject / log a hard integrity error rather than silently storing a snapshot that can't be re-hashed).
  • Close the empty-manifest delta-base bug (external delta-only parent resolving to {}).
  • (muse, separate) muse read-snapshot drops the directories field — fix so it returns the full stored record.
  • Repair the 4 corrupt gabriel/muse staging snapshots from the source repo's correct data.

Provenance

Surfaced during the muse clone hash-mismatch investigation (2026-06). Originally mis-diagnosed as a muse code migrate snapshot-rekey gap; a raw-object scan of the source repo showed all snapshots are correctly keyed (manifest + directories), relocating the bug to musehub's push/store/serve handling of directories.

◎ Activity2
●
gabriel opened this issue 123 days ago
○
gabriel 29 days ago

Picked this up. Two fixes landed and pushed to dev (local/staging/production remotes), both with real RED/GREEN test verification (not just inspection):

  1. Root cause found and fixed (musehub commit c3910cc5): the push handler's integrity check --

    if _parent_sid and hash_snapshot(_base, _snap_dirs or None) != _sid:
    raise ValueError(...)

    only ran when _parent_sid was truthy. Every root-of-push snapshot (parent_snapshot_id=None -- the first commit in a push, or a new repo's first commit) was therefore never checked against its own declared ID before being persisted. A wire/transport bug losing directories for exactly this kind of entry would sail through unverified and be stored permanently -- this is the precise shape of the entries=1062 dirs=0 corruption reported above. Fixed by running the check unconditionally; verified with a temporarily-reverted-gate RED test before restoring the fix for GREEN. 2 new tests in tests/test_wire_push_root_snapshot_integrity.py.

  2. muse read-snapshot was silently dropping directories (muse commit ec9e0eaa7) -- one of the checklist items below. Fixed; 6 new tests in tests/test_cmd_read_snapshot.py.

Remaining from the checklist, not yet done:

  • Close the empty-manifest delta-base bug (separate mechanism: external delta-only parent resolving to {}).
  • Repair the 4 known-corrupt gabriel/muse staging snapshots via wire_repair_snapshot.
  • Deploy the musehub fix above to staging/production (currently only in source on dev, not yet live).
  • #60's schema hardening (snapshot.directories: list[str] -> dict[str,str]) -- larger, phased, cross-repo change, not started.
○
gabriel 29 days ago

All 4 corrupt snapshots repaired and verified via a real end-to-end clone.

Repaired via POST /gabriel/muse/repair-snapshot (signed with muse sign request), using the correct (manifest, directories) read directly from the local source repo's object store (each payload verified locally with hash_snapshot(manifest, directories) == snapshot_id before sending -- all 4 matched):

  • sha256:eea07a1f86a310b9a92122283b46382255efd6e565cd7b349c2fde2fc21b758a -- 1062 files, 31 dirs
  • sha256:708d5734e7422feaeeb1703b72d93dc1f5cdc870581d2afb697a1cc78e99520f -- 1062 files, 31 dirs
  • sha256:3d5ae8b5dd2515f2f3790f73101ab72791e189002a9454486699bfd6447ec32e -- 1062 files, 24 dirs
  • sha256:edd649a9f2d795114aed8cb022277a69968faa04f68fb9660717712f4ad3e74e -- 1062 files, 24 dirs

All 4 returned {"repaired": true}.

Verified with a real clone, not just the repair endpoint's own success response: muse clone https://staging.musehub.ai/gabriel/muse now completes cleanly -- 1537/1537 commits landed, zero hash-mismatch warnings, zero skipped commits, HEAD matches the local source repo's main tip exactly. Compare to the original bug report: 1185/1444 commits landing (259 dropped in the cascade from these exact 4 corrupted snapshots).

Both the musehub deploy (root-snapshot integrity fix, commit c3910cc5) and this data repair are now live on staging. Also deployed to production. Closing out the checklist:

  • Push/store path must persist the snapshot directories array end-to-end.
  • Verify a snapshot's stored content reproduces its snapshot_id at unpack time (reject/log a hard integrity error).
  • Close the empty-manifest delta-base bug -- already closed via #134, reverified still correct.
  • muse read-snapshot drops the directories field -- fixed.
  • Repair the 4 corrupt gabriel/muse staging snapshots -- done, verified via clean clone above.

Remaining: #60 (snapshot.directories: list[str] -> dict[str, str]) is a separate, larger schema-hardening issue, not a correctness bug on its own -- tracked there, not blocking closing this one.

✓
closed this issue 29 days ago