Investigate: clone can silently deliver a truncated working tree on repos not owned locally (found via aaronrene/knowtation)
Background
Split off from #39 (now closed — the muse/musehub fetch-cache self-heal bug is fixed and verified on local/staging/production).
While re-investigating #39, cloning aaronrene/knowtation on staging (the original report's own "control" repo, previously used to prove the bug was scoped elsewhere) produced a silent, worse failure:
$ muse clone https://staging.musehub.ai/aaronrene/knowtation repo
⚠️ apply_mpack: commit sha256:db88a6e3… skipped — parent not in mpack or local store
[... 30+ more identical warnings ...]
[clone] apply_mpack DONE blobs_written=1515 blobs_skipped=0 commits_written=79
⚠️ clone: commit sha256:4215cecb… not found after apply_mpack — working tree not restored
✅ Cloned into 'repo' — 79 commit(s), 1515 blob(s), domain=code, branch=main (sha256:4215cecb…)
The clone reports success. The working tree is completely empty (only .muse/ exists, zero files), and muse log shows 0 commits locally, despite muse status claiming a clean checkout at a real commit hash.
What's already ruled out
Verified directly against the staging RDS database (read-only queries, no writes made): this is not data corruption. musehub_commits and musehub_commit_graph both have correct, self-consistent parent_ids for the HEAD commit (single parent, generation ~1127). The repo has 1247 total commits. The server's fetch/mpack response for this clone only contained 79 commits — nowhere near the ~1127-deep ancestor chain that should have been walked — and didn't even include the requested HEAD commit itself.
Given what was subsequently found and fixed in #39 (a stale musehub_fetch_mpack_cache entry can be served without validating it still reflects current branch state), the most likely explanation is the same class of bug: a stale cache HIT serving old, incomplete content rather than a MISS. This is unconfirmed — the #39 fix only addressed the MISS path (enqueueing a rebuild when no cache entry exists); it did not add any validation that a cache HIT's content is actually complete/current. Worth checking whether the cache-HIT path in wire_fetch_mpack needs a similar hardening pass (e.g. verifying the cached mpack's commit count/generation against live musehub_branches/musehub_commit_graph state before trusting it).
Constraint
Neither gabriel nor Claude has a canonical local copy of aaronrene/knowtation to safely experiment against — it's aaronrene's repo. Per gabriel: aaronrene should either debug this directly against his own copy, or the repo should be re-cloned from its GitHub mirror and re-pushed to MuseHub fresh so it can be investigated without risk to the only remaining good copy.
Suggested next steps
- aaronrene: confirm whether
aaronrene/knowtationstill reproduces this today (the #39 fix may have incidentally helped if a MISS was involved, but the symptom described above looks like a HIT, not a MISS — no retry-loop was observed). - If it still reproduces: get a private copy of the repo's real history (or a synthetic repo shaped the same way) for safe experimentation.
- Audit
wire_fetch_mpack's cache-HIT path (musehub/services/musehub_wire_fetch.py) for whether it validates cached content against live branch/commit-graph state before serving it. - Once root-caused, follow the same test-first pattern used for #39: unit test proving the gap, fix, full regression pass, then real end-to-end verification against the actual affected repo.
Re-tested this today, and I think the stale-cache hypothesis can be ruled out — the mpack is not truncated.
The "server's fetch/mpack response only contained 79 commits" figure is commits_written, which is the count the client wrote, not what the server sent. Full transport log from a fresh staging clone:
[transport] GET done status=200 size=95.23MB
[transport] unpackb commits=1220 snaps=956 blobs=1515
[transport] coerce commits=1220 snaps=956 blobs=1515 heads=0
[clone] apply_mpack DONE blobs_written=1515 blobs_skipped=0 commits_written=79
The server sent 1,220 commits. Everything is lost client-side, and the cause is the first commit processed — knowtation's root:
❌ Commit sha256:e3574dcaff08ff0d… failed content-hash verification
Expected sha256:e3574dcaff08ff0d…
recomputed sha256:9b5565dfd8cdf033…
⚠️ apply_mpack: malformed commit — skipped: Refusing to write commit 'sha256:e3574dca…'
Once the root is refused, every descendant fails for a missing parent. Census over the full 1,158-line log:
| Message | Count |
|---|---|
failed content-hash verification |
1 (the root) |
parent not in mpack or local store |
1,140 (cascade) |
not found after apply_mpack |
1 (tip 4215cecb…) |
integrity / ObjectHashMismatch / mismatch |
0 |
One real error, 1,140 consequences. The 79 written commits are the subset not descending from the poisoned root.
Root cause is committed_at timezone normalisation — musehub_repo_models.py:187 stores it as timestamptz, musehub_wire_shared.py:379 rebuilds the identity string from the normalised value, and committed_at is hashed literally in the commit id. Normalising 2026-03-13T08:37:25-07:00 to UTC reproduces 9b5565df… to all 64 digits.
Full evidence and the corrected blast-radius counts are on #195. Suggest closing this as a duplicate of #195, or keeping it as the symptom with #195 as the cause — a cache-validation fix alone won't make this repo clonable.
(Also worth noting for the DB check in the description: the DB is fine, exactly as you found. The commits and the graph are self-consistent; the defect is in how committed_at is re-serialised on the way out, not in what is stored.)
Root-caused by aaronrene — same bug as #195, not a stale-cache HIT.
Revised diagnosis
The original diagnosis ("the server's response only contained 79 commits" implying a stale cache HIT) was based on misreading the client log: 79 is commits_written — what the client successfully wrote — not the size of the server's payload.
A fresh transport-level capture tells the real story:
[transport] GET done status=200 size=95.23MB
[transport] unpackb commits=1220 snaps=956 blobs=1515
[clone] apply_mpack DONE blobs_written=1515 commits_written=79
The server sent all 1,220 commits. The mpack is complete. The cache is not implicated at all.
The real failure is a single content-hash mismatch at knowtation's root commit:
❌ Commit sha256:e3574dcaff08ff0d… failed content-hash verification
Expected sha256:e3574dcaff08ff0d…
recomputed sha256:9b5565dfd8cdf033…
— which cascades into 1,140 'parent not in mpack or local store' warnings (every descendant of the poisoned root), leaving only the 79 commits that don't descend from it. Census over the full 1,158-line log: 1 real error (the root), 1,140 consequences, 0 integrity/ObjectHashMismatch errors — i.e. exactly the shape you'd expect from one bad root cascading, not a cache or integrity-check bug.
9b5565dfd8cdf033… is reproduced to all 64 digits just by normalizing the root commit's 2026-03-13T08:37:25-07:00 timestamp to UTC before hashing — this is exactly #195's bug (Postgres's timestamptz column silently normalizing a non-UTC committed_at before it's re-serialized and re-hashed on fetch), observed live on the one real repo it was theorized to affect.
Resolution
#195 is fixed and deployed to staging and production (additive committed_at_raw column preserves the client's exact original string; _to_wire_commit prefers it over reconstructing from the normalized timestamptz). That fix is what makes knowtation clonable again — no cache-layer change was ever needed for this specific symptom.
Closing as a duplicate of #195 — the underlying cache-HIT-validation question this ticket raised is worth keeping in mind in the abstract, but the actual observed failure here wasn't a cache bug.
For context on how this ticket's original content was briefly lost and had to be recreated verbatim: see #184 (phantom-success write investigation).