2026-05-08-issue-39-clone-corruption.md
file-level
1
files
1
commits
0
hotspots
0
🧊 dead
0
💥 blast risk
| 1 | --- |
| 2 | date: 2026-05-08 |
| 3 | status: filed |
| 4 | issue_number: 39 |
| 5 | issue_id: sha256:c570f79c16f0e472731c474782ac664f686c43ff4f7b2e7088551d0f656780d0 |
| 6 | repo: gabriel/musehub |
| 7 | url: https://staging.musehub.ai/gabriel/musehub/issues/39 |
| 8 | title: muse clone gabriel/muse fails on both staging (object integrity) and production (msgpack) |
| 9 | labels: [bug, clone] |
| 10 | event_type: discover |
| 11 | tags: [contributions, musehub, bug-report, clone, infrastructure, blocks-contribution] |
| 12 | related: [issue-36, issue-38] |
| 13 | --- |
| 14 | |
| 15 | # Issue #39 — `muse clone gabriel/muse` fails on both staging and production |
| 16 | |
| 17 | Filed: 2026-05-08 |
| 18 | Repo: gabriel/musehub |
| 19 | URL: https://staging.musehub.ai/gabriel/musehub/issues/39 |
| 20 | |
| 21 | Discovered while attempting to submit issue #38 fix as a proper merge proposal (the contributor flow). Both clone endpoints fail in different ways. Blocks any external contribution work against `gabriel/muse`. |
| 22 | |
| 23 | ## Issue body (as filed) |
| 24 | |
| 25 | # `muse clone gabriel/muse` fails on both staging and production |
| 26 | |
| 27 | **Affected component:** `muse clone` against the `gabriel/muse` repo |
| 28 | **Affected versions:** client `0.2.0rc7` against `staging.musehub.ai` and `musehub.ai` (2026-05-09) |
| 29 | **Severity:** High — blocks the entire contributor merge-proposal workflow against `gabriel/muse`. External contributors cannot fork, branch, push, or open proposals. |
| 30 | **Reporter:** @aaronrene |
| 31 | **Related:** Issue #38 (the bridge exec-bit fix that the reporter would have submitted as a merge proposal but couldn't, because of this issue). |
| 32 | |
| 33 | --- |
| 34 | |
| 35 | ## Plain-language summary |
| 36 | |
| 37 | I tried to do exactly what an outside contributor is supposed to do: clone the `muse` source repository, make a small bug fix on a feature branch, push, and open a merge proposal. Both clone endpoints are broken in different ways. **Staging** returns ~25 files whose downloaded content doesn't match the hash the server says they should have, then dies on a missing object (HTTP 404). **Production** rejects every request because the wire format the server is sending isn't something the latest client can parse. Either of these alone would block clone; together they block clone entirely. |
| 38 | |
| 39 | This means there is currently no way for a third party to make code contributions to `gabriel/muse` through the standard MuseHub flow, even when the contributor has a working client, valid auth, and a perfectly good patch ready to go. |
| 40 | |
| 41 | ## Technical summary |
| 42 | |
| 43 | Two distinct, simultaneously-occurring failure modes against the same logical repo: |
| 44 | |
| 45 | **Failure A — staging integrity violations + 404** |
| 46 | ``` |
| 47 | $ muse clone https://staging.musehub.ai/gabriel/muse muse-source |
| 48 | clone: skipping corrupted object sha256:1e4d2c138a5b…: Content integrity failure: |
| 49 | expected object sha256:1e4d2c138a5b… got sha256:cf9ad6812fc5… |
| 50 | clone: skipping corrupted object sha256:90ecc4781a20…: Content integrity failure: |
| 51 | expected object sha256:90ecc4781a20… got sha256:0764d5e337ea… |
| 52 | [… 23 more "Content integrity failure" lines, each pair of expected/got hashes is consistent (same expected → same got) ...] |
| 53 | ❌ Fetch failed: fetch/presign: object GET HTTP 404 |
| 54 | exit_code: 3, elapsed_ms: 117059 |
| 55 | ``` |
| 56 | |
| 57 | The 25 reported pairs are all of the form `expected sha256:X got sha256:Y` where X and Y are deterministic — re-running the clone gets the same hash mismatch for each object, suggesting these are not transient read errors but objects whose stored bytes do not match their advertised content hashes. After 25 such warnings the clone hits an `HTTP 404` on a presigned object URL and aborts, producing no usable repository. |
| 58 | |
| 59 | **Failure B — production msgpack mismatch** |
| 60 | ``` |
| 61 | $ muse clone https://musehub.ai/gabriel/muse muse-source |
| 62 | Cloning from https://musehub.ai/gabriel/muse … |
| 63 | ❌ Cannot reach remote: Server returned invalid msgpack: unpack(b) received extra data. |
| 64 | exit_code: 3, elapsed_ms: 1469 |
| 65 | ``` |
| 66 | |
| 67 | Production rejects `0.2.0rc7` immediately. `unpack(b) received extra data` means msgpack reached the end of one valid object but found additional bytes in the buffer that don't belong to it — the server is either framing responses differently than the client expects (e.g. concatenated NDmsgpack or a different content-length contract), or running an older/newer schema entirely. |
| 68 | |
| 69 | ## Reproducer |
| 70 | |
| 71 | ```bash |
| 72 | # Pre-conditions — fresh client install |
| 73 | curl -fsSL https://staging.musehub.ai/install.sh | sh |
| 74 | muse --version # confirms 0.2.0rc7 |
| 75 | |
| 76 | # Reproducer A — staging |
| 77 | mkdir -p /tmp/clone-test-A && cd /tmp/clone-test-A |
| 78 | muse clone https://staging.musehub.ai/gabriel/muse repo |
| 79 | # Observe: many "Content integrity failure" lines, then HTTP 404, exit 3. |
| 80 | |
| 81 | # Reproducer B — production |
| 82 | rm -rf /tmp/clone-test-B && mkdir /tmp/clone-test-B && cd /tmp/clone-test-B |
| 83 | muse clone https://musehub.ai/gabriel/muse repo |
| 84 | # Observe: "Server returned invalid msgpack: unpack(b) received extra data", exit 3. |
| 85 | |
| 86 | # Sanity — does aaronrene/knowtation clone successfully? (yes, both endpoints) |
| 87 | mkdir -p /tmp/clone-test-C && cd /tmp/clone-test-C |
| 88 | muse clone https://staging.musehub.ai/aaronrene/knowtation repo |
| 89 | # Observe: clean clone, no integrity failures. |
| 90 | ``` |
| 91 | |
| 92 | This pattern — `aaronrene/knowtation` clones cleanly while `gabriel/muse` does not — strongly suggests the issue is **specific to the `gabriel/muse` repo's stored objects on staging**, not a generic clone bug. Production's msgpack failure is presumably an unrelated server-version mismatch. |
| 93 | |
| 94 | ## Expected behaviour |
| 95 | |
| 96 | - Both endpoints accept the latest client and return a complete, hash-verified clone. |
| 97 | - A contributor with a fresh `0.2.0rc7` client can clone, create a feature branch, commit changes, push, and open a merge proposal against `gabriel/muse`. |
| 98 | |
| 99 | ## Actual behaviour |
| 100 | |
| 101 | - Staging produces a partial clone with 25 unverifiable objects then aborts. |
| 102 | - Production rejects every request before any objects transfer. |
| 103 | - No contributor merge-proposal flow is currently possible against `gabriel/muse`. |
| 104 | |
| 105 | ## Possible root causes (best guesses, not verified) |
| 106 | |
| 107 | For staging: |
| 108 | 1. **Stale data in the object store.** The 25 objects' stored bytes were rewritten at some point (perhaps during the `compute_commit_id` deployment that fixed issue #36), but their addresses in the store are still the legacy hashes. Re-uploading the corrected blob bytes under their new hashes — and updating commit/snapshot pointers to match — would resolve. |
| 109 | 2. **Pack-rebuild interrupted mid-flight.** A re-pack process started but didn't complete, leaving some objects with manifest entries that point to in-flight content. A `muse gc` or admin re-pack on the server side may fix. |
| 110 | 3. **Migration from v1 → v2 commit-ID formula didn't propagate to all blob references.** If snapshots inside old commits still address blobs by their pre-migration hashes, those would now hash-mismatch. |
| 111 | |
| 112 | For production: |
| 113 | 1. **`musehub.ai` (production) is running a different protocol version than the client.** Client `0.2.0rc7` was installed from `staging.musehub.ai/install.sh` and is presumably matched to staging's protocol. Production may be on an older or newer ABI. Could be fixed by aligning protocols, or by the client downgrading/upgrading transparently per server version. |
| 114 | |
| 115 | I don't have access to either server's logs and these are pure inference — open to being told which of these (or something else entirely) is correct. |
| 116 | |
| 117 | ## Workaround in use today |
| 118 | |
| 119 | Reporter applied the patch from issue #38 directly to the local `0.2.0rc7` install (`StrReplace` against `~/.local/share/muse/venv/lib/python3.14/site-packages/muse/cli/commands/bridge.py`), validated the fix end-to-end, then rolled back. The patch was submitted as issue body / unified diff in #38 rather than as a merge proposal. **This works for tiny patches but doesn't scale** — anything more than ~50 lines, or anything that needs a real test suite committed alongside the fix, requires a working clone. |
| 120 | |
| 121 | ## What I'd like to do once this is fixed |
| 122 | |
| 123 | 1. Re-clone `gabriel/muse` cleanly. |
| 124 | 2. Apply the patch from issue #38 to a feature branch (`feat/bridge-exec-bit-fix`). |
| 125 | 3. Push to `gabriel/muse`. |
| 126 | 4. Open a proper merge proposal targeting `main`. |
| 127 | 5. Get the merge proposal credited with proper attribution (commit `Co-authored-by: aaronrene` or similar). |
| 128 | 6. Use the same flow for any future contributions. |
| 129 | |
| 130 | ## Severity / urgency |
| 131 | |
| 132 | **High but not urgent for the reporter individually** — issue #38 contains a workable fix even without a clone, so the immediate symptom is unblocked. **High and urgent for the project** — every prospective contributor hits this wall. The longer it persists the smaller the contributor pool gets. |
| 133 | |
| 134 | ## Suggested labels |
| 135 | |
| 136 | `bug` · `clone` · `infrastructure` · `blocks-contribution` |
| 137 | |
| 138 | ## Related |
| 139 | |
| 140 | - Issue #36 (resolved): `compute_commit_id` formula divergence — fixed via server reset of `staging/main`. The 25 corrupted-object reports here may be related: if the server reset migrated commits but didn't migrate blob addresses, that would explain the integrity failures we observe. |
| 141 | - Issue #38 (open): `muse bridge git-export` strips exec bit. Complete patch attached, would have been a merge proposal if this clone issue weren't blocking. |