gabriel / musehub public
Open #188 Bug
filed by gabriel human · 35 days ago

~/ecosystem/muse canonical object store: 3,010 objects fail hash verification (likely orphaned migration leftovers, needs investigation before gc)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Summary

While testing Phase 4 sandbox tooling for musehub staging #185 (disposable APFS copy-on-write clones for safe muse-dev testing), a real, pre-existing issue in ~/ecosystem/muse's canonical object store surfaced as a side effect — unrelated to the sandbox test itself, and not caused by it.

Finding

muse verify-object --all --json reports 3,010 of 24,320 stored objects fail hash verification (hash mismatch: stored <id> recomputed <different-id>). Independently confirmed via manual Python hashing of one sample object (sha256:0008ab6695...) — this is a genuine hash mismatch, not a bug in the verify tool's comparison logic. Also reproduces via muse cat-object <id>, which refuses to return the content ("object store may be corrupt").

Why this isn't (yet) an emergency

  • muse verify --json — which walks only objects reachable from current branch history (dev, main) — reports all_ok: true, zero failures. Nothing currently reachable appears corrupted.
  • muse count-objects --unreachable --json reports 9,363 unreachable objects in the store — a number that comfortably contains the 3,010 bad ones. Working theory: these are orphaned leftovers from a historical commit-ID migration (this repo's own nightly changelog references a muse code migrate --force-resign pass, and a muse code fix-snapshot-ids tool built defensively during an earlier corrupted-commit investigation — muse#83).

What's NOT yet confirmed

  1. Whether the 3,010 bad-hash objects are actually a strict subset of the 9,363 unreachable ones (plausible, not verified — no direct tooling surfaced yet to list unreachable object IDs and intersect them against the verify-object failure list).
  2. Why a migration would leave behind objects whose content doesn't match their own filename/ID at all, rather than just leaving them orphaned-but-internally-consistent. Orphaned-but-correct would be unsurprising (old history, superseded but still hashing correctly); orphaned-and-hash-mismatched is a different, more concerning shape — it suggests something wrote or copied these objects incorrectly at some point, not just "stopped referencing them."
  3. Whether any non-branch ref (tags, shelved states, harmony pattern records, coordination reservations) references any of these 3,010 objects in a way muse verify's branch-reachability walk wouldn't catch — i.e., whether "unreachable from branches" truly means "safe to discard."

Suggested investigation, before any muse gc / muse prune

  1. Get the actual unreachable object ID list (may need a --list/--json enhancement to muse count-objects --unreachable if it doesn't already expose IDs, not just a count) and confirm the 3,010 bad-hash objects are a subset.
  2. For a sample of the bad objects, trace them back (if possible) to which historical commit(s) referenced them before the ID-rewriting migration — confirm they were legitimately superseded, not something still needed.
  3. Check tags, shelf entries, harmony patterns, and coordination reservations for any reference to the bad object IDs.
  4. Once confirmed safe, run muse gc (or muse prune) to actually remove them, and re-run muse verify-object --all --json to confirm all_ok: true afterward.
  5. Consider whether muse verify (the default fsck-equivalent most people would run) should also warn about unreachable-but-corrupted objects even though they don't affect live data — silently having 12% of the physical store fail hash checks, discoverable only via the less-commonly-run verify-object --all, seems like a gap worth closing regardless of root cause.

Scope note

This is specific to ~/ecosystem/muse's canonical object store as observed today. Not yet checked whether ~/ecosystem/musehub has the same class of issue — worth a quick muse verify-object --all --json there too as a first step.

◎ Activity2
●
gabriel opened this issue 35 days ago
○
gabriel 35 days ago

Escalation: some of this corruption IS reachable, not purely orphaned garbage

While building Phase 5's bundle-backup mechanism for #185 and testing it for real against ~/ecosystem/muse, I found something that contradicts this ticket's original "not an emergency" framing.

muse branch --json shows only two local branches exist: dev and main — the exact same two muse verify's reachability walk covers (refs_checked: 2, reported all_ok: true). But running muse bundle create <file> dev main --json followed by muse bundle verify <file> --json — which independently walks every object reachable from those same two branches — found ~2,000+ corrupted blobs (truncated at the tool's 100-failure display cap; the real count is higher).

This means muse verify and muse bundle create/verify disagree about the same reachable object set on the same two branches. Either:

  • muse verify's reachability walk is incomplete/skips something bundle create includes (a real bug in verify, meaning it's been giving false confidence), or
  • muse bundle create is over-inclusive, packing objects that aren't actually needed for dev/main's current state (a real bug in bundle create, meaning bundle backups of this repo may be unusable until fixed).

Either way, this is not "safe to ignore, orphaned garbage" — it needs investigation before we trust either tool's output at face value. I have not attempted a fix or run muse gc/muse prune — flagging this and stopping here.

Practical impact right now

~/ecosystem/muse's "belt" backup mechanism (#185 Phase 5's backup-bundle.sh, which bundles all local branches) currently fails against the real canonical repo because of this — muse bundle verify correctly refuses to certify a bundle containing corrupted blobs, so no working muse-repo bundle backup can be produced right now. The "suspenders" mechanism (APFS snapshot of .muse/) is unaffected and works fine, so we're not without a backup path — but this needs to be root-caused, not worked around.

Updated suggested investigation (supersedes the original list above)

  1. Reconcile muse verify's reachability walk against muse bundle create's — same two branches, same repo, should find the same object set. Find where they diverge.
  2. Once the true "reachable from dev+main" object set is known, check exactly how many of the 3,010 (or more) bad-hash objects are actually within it vs. genuinely orphaned.
  3. Everything else from the original investigation list still applies once the above is resolved.
○
gabriel 35 days ago

Reframing: this is a recovery investigation, not a cleanup task

Per gabriel: "the higher fidelity and resolution the quality of our data and playback, the higher value muse is to the world." That directly changes how this ticket should be worked — the default action is not "these are probably safe to gc," it's "attempt to recover them from a known-good source before any deletion is even considered."

Root cause of the verify vs bundle create discrepancy — now understood

Confirmed via code reading (muse/core/verify.py vs muse/core/mpack.py::build_mpack / walk_commits):

  • muse verify walks each reachable commit, reads its snapshot's full manifest, and checks only the objects that manifest references — deduplicated by snapshot ID. Answers: "is everything needed to check out any current commit intact?"
  • muse bundle create computes its object set from snapshot deltas across the full ancestor walk (the incremental diff format used for wire-protocol replay). Answers a different question: "is everything needed to replay full commit-by-commit history intact?"

These are genuinely different object sets by design — not a bug in either tool individually. The corrupted objects are ones that exist only in the delta-history representation (old file versions later modified or deleted), invisible to verify's manifest-based walk. Separately, build_mpack never validates hashes when packing — it trusts disk content as-is; corruption only surfaces later at bundle verify time. That's expected given the design, not a bug.

Confirmed data points

  • 3,012 objects fail hash verification (out of 24,320 total; verify itself only ever checks 12,479 of them — the manifest-reachable subset).
  • Sampled file mtimes cluster May 20 – June 10, 2026 — roughly 3-4 months old, predating the muse code migrate --force-resign migration referenced in the July nightly changelogs. Consistent with "old file versions from before that migration," not random/arbitrary corruption.
  • None of the 3,012 appear in any currently-reachable commit's manifest — confirmed structurally (that's the exact reason verify's 12,479-object walk never touches them).

Updated action plan — recovery-first

  1. Do not run muse gc / muse prune — deleting first forecloses recovery permanently. No cleanup action should happen until step 2 is done.
  2. Check whether staging, production, or local hub already hold good copies of these specific object IDs — this repo's history has been pushed to all three throughout ongoing work, so there's a real chance the hubs have intact copies of what's now corrupted locally.
  3. If good copies exist on a remote: repair the local store by fetching those specific objects (not a full gc), preserving full delta-replayable fidelity.
  4. If no remote has good copies either: this is a genuine, harder data-loss question for that slice of history — needs to be confirmed explicitly, not assumed, before deciding how to proceed (e.g., whether to accept the loss, or note it as a permanent history gap).

Status

Deliberately queued as its own follow-up investigation — not resolved today. Paused here to get back to closing out musehub staging #185 (Phase 6 + the v0.2.1-rc1 → stable promotion decision), which was the original task this was discovered inside of.