Critical: muse hub proposal merge can silently corrupt dev if called twice (stale state read + stale-base re-merge)

Note: this ticket number originally briefly overwrote/collided with an unrelated pre-existing ticket (the aaronrene/knowtation truncated-clone investigation, assigned to aaronrene). That ticket's content did not survive the collision and has been recreated from scratch as #183 (reassigned to aaronrene). Root cause of the numbering collision itself is still open — investigating.

Confirmed: no data was actually lost

Verified after the fact — dev currently sits at the correct commit (sha256:8a1389ae...) on local, staging, and production (all three checked directly via muse rev-parse/muse ls-remote). The manifest at that commit contains every file that would have been wiped by the bad re-merge (alembic/versions/0075_fix_proposal_snapshot_commit_naming.py, both new/recovered test files, the overseer-provenance service+route, and the corrected musehub/config.py) — confirmed present via muse read dev --json --manifest. The 47-file regression only ever landed on staging's branch ref for a few minutes before being force-reset back; it never reached local or production.

Root-cause hypothesis (read merge_proposal in musehub/services/musehub_proposals.py)

merge_proposal's default overlay strategy recomputes to_manifest/from_manifest from to_b.head_commit_id / from_b.head_commit_id read fresh at call time — it does not use a fixed merge-base captured at proposal-creation time. That part is actually fine for idempotency if the branch heads reflect reality.

The likely defect: on the first merge call, the branch-content push (whatever writes the actual pushable branch ref/blob state) succeeded, but the DB write updating musehub_branches.head_commit_id (for to_branch) and/or the musehub_proposals row (state/merge_commit_id/merged_at) did not commit — same symptom in both places, which suggests a single shared cause: either an exception after the DB writes were queued but before session.commit(), or a commit that raised and was swallowed by a broad except higher up. Because to_b.head_commit_id was then still pointing at the pre-merge commit, the second merge call recomputed the overlay from a stale to_manifest, silently discarding merge #1's already-live content.

This needs verification, not just theorizing — the actual DB transaction boundary in the merge route/service layer needs to be traced end-to-end.

Belt-and-suspenders plan (do all of these, not just one)

1. Idempotency guard: merge_proposal must re-check proposal.state in the same DB transaction immediately before writing the merge commit (not just at function entry) and hard-fail (409) if it's already merged/closed — closes the "ran twice" hole regardless of root cause.
2. Transaction atomicity: wrap the branch-head update + proposal-state update + commit-row insert in one transaction with an explicit session.commit() and no swallowed exceptions in between. Add a regression test that forces a failure between steps and asserts nothing partially persists.
3. Merge-base staleness guard: before executing the overlay, compare the to_branch's current head_commit_id against what the proposal's merge preview/diff was last computed against (or at minimum log/flag when to_branch has advanced significantly since the proposal was opened) — this doesn't need to block the merge, but should make a stale-base merge loud instead of silent.
4. Route-level protection against double-submit: the API route calling merge_proposal should treat a second merge request on an already-merged/closed proposal as a clear 409, with a test hitting the route twice back-to-back.
5. Regression tests (write before touching the fix):
   • Merge a proposal, assert state/merge_commit_id/merged_at persist and are readable in a fresh session/query — this is the exact gap that slipped through today.
   • Call merge twice on the same proposal — second call must 409, never recompute/re-merge.
   • Simulate to_branch advancing between proposal open and merge (e.g. another commit landed on dev) — merge must not silently drop that intervening commit's content.
6. Post-merge sanity check: after merge, diff the new to_branch head's manifest against the pre-merge head's manifest — if it does anything other than a superset (i.e. any file that existed pre-merge disappears without being part of the incoming branch's deletions), fail loudly rather than committing.

Follow-up ticket needed (separate from this one)

The issue-numbering collision this bug's own creation triggered (a second, unrelated ticket — the aaronrene/knowtation investigation — was overwritten under the same number and had to be recovered as #183) is a distinct bug in muse hub issue create's number-allocation path and should be tracked separately once root-caused.

Filed the numbering-collision root-cause investigation separately: #184.

musehub#182 — root cause: MuseHub destroys non-UTC committed_at, breaking clone of any bridged repo

Status: DRAFT for review before posting to musehub#182 Date: 2026-09-10 Author: aaronrene Severity: blocks clone of every muse bridge git-import repo; muse clone reports success on total failure Scope: root cause verified; fix not yet built

Summary

muse clone of aaronrene/knowtation produces an empty working tree while exiting 0 and printing ✅ Cloned. The transfer is not at fault and the object stores are not corrupt.

MuseHub persists commit timestamps as a timestamptz column and reconstructs the identity-bearing committed_at string from it on the serve path. Muse computes a commit ID over the literal ISO-8601 string. Any commit whose original string was not UTC is therefore re-served under a different identity than it was created with, the receiving client correctly rejects it as content-hash-corrupt, and every descendant cascade-fails with parent not in mpack or local store.

Knowtation is affected because all 1,778 of its git-import commits carry git's original -07:00 author offset. musehub and muse are unaffected because native muse commit writes UTC — which is exactly why all four muse/musehub × staging/production clone combinations pass while knowtation fails.

Reproduction

muse clone https://staging.musehub.ai/aaronrene/knowtation kn

Observed (staging, 2026-09-10):

[transport] GET done  status=200  size=95.23MB
[transport] sha256  match=True
[transport] unpackb  commits=1220  snaps=956  blobs=1515
[clone] apply_mpack START  blobs=1515 commits=1220 snaps=956
❌ Commit sha256:e3574dca… failed content-hash verification — core fields are corrupt
   (snapshot_id, message, committed_at, or parent IDs).
   Expected sha256:e3574dca…, recomputed sha256:9b5565df…
⚠️ apply_mpack: malformed commit — skipped
… 1,140 × "skipped — parent not in mpack or local store" …
[clone] apply_mpack DONE  commits_written=79
⚠️ clone: commit sha256:4215cec… not found after apply_mpack — working tree not restored
✅ Cloned into 'kn' — 79 commit(s), 1515 blob(s), branch=main

Resulting clone: 0 files in the working tree, muse log → (no commits), refs/heads/main written to a commit that does not exist in the store. Exit code 0.

The mpack itself is intact: sha256 match=True, and the server sent the full 1,220-commit history. Exactly one commit is rejected on hash grounds; the other 1,139 skips are the cascade from it, because the rejected commit is a root commit (parent_commit_id: null).

Root cause (verified, not inferred)

The rejected commit:

Muse's commit identity is sha256("commit <size>\0" + canonical) where canonical is the null-separated payload parents ‖ snapshot_id ‖ message ‖ committed_at_iso ‖ author ‖ signer_public_key (muse/core/snapshot.py::compute_commit_id / commit_identity_bytes). committed_at_iso is hashed as a string.

Recomputing that formula over this commit's real fields, varying only the timestamp rendering:

Both matched to the full 64 hex digits. The commit was created with the -07:00 rendering and is being re-served with the UTC rendering.

Where the string is lost

Storage keeps a datetime, not the original string:

musehub/db/musehub_repo_models.py:187

timestamp: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)

The serve path then reconstructs the identity-bearing field from that column:

musehub/services/musehub_wire_shared.py:379

committed_at=row.timestamp.isoformat() if row.timestamp else "",

timestamptz normalizes to UTC on read, so .isoformat() can only ever emit +00:00. The original offset is not recoverable from the database — which matters for the fix (see below).

This is a content-addressing violation: an identity input is being reconstructed from a lossy representation rather than preserved verbatim.

Blast radius

Every non-UTC commit is a git-import commit. Native muse commit writes +00:00 with microseconds and round-trips losslessly.

This is not knowtation-specific. Any repo onboarded through muse bridge git-import inherits git's original author timezone offsets and will be unclonable from MuseHub. That makes it a blocker on the third-party-domain path generally, not just this one repo.

Existing precedent in the codebase

musehub/services/musehub_wire_push.py already carries a repair-commit endpoint whose docstring describes this exact failure mode for a different identity field — signer_public_key stamped by the rc10 object-store migration without recomputing the id:

Because signer_public_key is part of the commit identity hash, the serve path then recomputes a different id, the client's hash check fails on clone, the commit is dropped, and every descendant fails "parent not in mpack", emptying the working tree.

That is precisely the mechanism here, via committed_at instead. Note that the existing endpoint cannot repair the timezone case, because it pins the value to the serve representation before verifying:

ts = _parse_iso(wire.committed_at)
wire.committed_at = ts.isoformat()

so a commit whose true identity used -07:00 can never validate through it. The docstring is explicit that such commits are rejected rather than repaired. The precedent is right; the timestamp field just was not covered.

Two hypotheses to rule out

Both suggestions from the 2026-09-09 discussion can be closed:

1. "Your local .muse object store is in an old state." It is not. muse verify on the local knowtation repo reports ✅ Repository is healthy across 140 refs, 1,307 commits, 1,042 snapshots and 4,797 re-hashed objects. The rejected commit's stored bytes hash to its own claimed ID exactly under the canonical formula — it is only invalid after the server's round-trip.

2. "Bridge an up-to-date git repo → muse and re-push." This will not help. The offset is destroyed at ingest/serve, so a re-push through the current path regenerates the same broken serve representation. The fix must land before any re-push, or the re-push is wasted.

Proposed fix (MuseHub side)

1. Persist the commit's committed_at verbatim (e.g. committed_at_iso: Mapped[str]) alongside the existing timestamp column, which stays for indexing/sorting.
2. _to_wire_commit emits row.committed_at_iso when present, falling back to row.timestamp.isoformat() for rows predating the column.
3. Extend repair-commit to accept and verify against the true string rather than pinning to the serve representation.
4. Add a wire-level invariant test: for a corpus including non-UTC and non-microsecond timestamps, recompute(serve(ingest(commit))) == commit_id.

Sequencing constraint (important)

The original strings are already gone from MuseHub's database and survive only in the pusher's local store. So the backfill cannot be a server-side migration — it must be a re-push (or a repair-commit sweep) sourced from a local store, performed after the ingest/serve fix lands. Ordering matters or knowtation still will not clone.

Related client-side findings (Muse, filed separately)

These are in the Muse client and touch code currently under active work (#39 / #131), so they are listed here only for traceability:

1. muse clone reports success on total failure — exit 0 and ✅ Cloned with an empty working tree and zero commits. Any CI or replication system records this as a good clone. It should fail closed when the requested branch tip is absent after apply_mpack.
2. muse verify cannot detect this class — it validates that commits are well-formed and re-hashes snapshot-referenced blobs, but never recomputes commit IDs, so 1,778 untransferable commits pass as healthy.
3. muse verify-object --all mass false positives — reports 3,239 of 9,080 objects as hash mismatches on a healthy repo, including the current main tip. That is exactly 2,197 commits + 1,042 snapshots: it applies the blob formula sha256("blob <size>\0" + data) to typed commit/snapshot records. Its own docstring recommends it to backup, replication and CI pipelines.
4. Commit metadata dropped on serve — _to_wire_commit hardcodes metadata={}, so the git_sha provenance on every bridged commit is lost on clone. Not identity-affecting, but it is silent data loss on the bridge path.

Evidence index

Correction to my earlier comment, plus a pointer to #63 which I should have found first.

Corrected counts

The blast-radius figures I posted earlier ("1,778 of 2,197" for knowtation, "1,568/1,568" for musehub) were wrong — I had only walked main. Recounted by walking every branch head and reading each commit through the client's own muse.core.commits.read_commit:

890 of those 890 non-UTC commits carry git_sha metadata, out of 891 git-import commits total — so the correlation with muse bridge import is essentially perfect, and zero natively-authored commits are affected. The conclusion is unchanged; only the magnitudes were wrong. Apologies for the noise.

This is an instance of #63

#63 ("All objects (commits, snapshots, blobs) must be written to the object store — S3 canonical, DB is index") already describes this failure mode generically:

Blobs are correctly written to S3 with a DB index row … Commits and snapshots are not — they are DB-only … muse pull after a server-side merge fails because the merge commit was never written to S3 and wire_fetch_mpack has to reconstruct it from DB fields rather than serving canonical bytes.

That is exactly the mechanism here: the reconstruction at musehub/services/musehub_wire_shared.py:379 rebuilds the identity string with row.timestamp.isoformat(), and musehub_repo_models.py:187 stores timestamp as DateTime(timezone=True), so a non-UTC offset is normalised away before it is re-served.

So there are two coherent fixes, and the choice is yours:

1. Narrow — preserve the verbatim string in an additive committed_at_raw column, plus a fail-closed check so the server never serves a commit id it cannot reproduce. Small, additive, and still correct after #63 lands.
2. Structural — do #63, store canonical commit bytes, and let it subsume this issue entirely.

I have (1) specced with a seven-tier test matrix and ready to build. If you would rather go straight to #63, say so and I will withdraw it rather than rework it.

One thing I cannot explain yet, and will not guess

#39 (which I filed 124 days ago) records muse clone https://staging.musehub.ai/aaronrene/knowtation succeeding on both endpoints with no integrity failures. knowtation's non-UTC root commit is dated 2026-03-13, so it already existed then; and timestamp has been DateTime(timezone=True) since alembic/versions/0001_consolidated_schema.py:262, so the column type is not what changed.

Something between then and now started surfacing this rather than silently tolerating it. I have not established what, so I am flagging it rather than asserting a cause. It matters beyond this ticket: if clones previously accepted these commits without verifying ids, that bounds how many already-distributed clones hold commits whose ids do not match their content.

Re-verified, since I got the counts wrong

Recomputing the root commit id with the installed client's own compute_commit_id (rc15), from the stored record's own fields:

stored committed_at : 2026-03-13T08:37:25-07:00
recomputed          : sha256:e3574dcaff08ff0d4a7d80b81c9d7c720727f3aa6e4e062e39224019e2f761c9
stored commit_id     : sha256:e3574dcaff08ff0d4a7d80b81c9d7c720727f3aa6e4e062e39224019e2f761c9   -> match

UTC-normalised      : 2026-03-13T15:37:25+00:00
recomputed          : sha256:9b5565dfd8cdf0339eec15f5a24cb271c294b645ceaea84067cdee2e8035858b   -> mismatch

A store "in an old state" could not be self-consistent under the current formula, so that hypothesis is refuted by construction rather than by a tool's opinion.

Status: bug #2 (no-staleness-guard) root-caused, fixed, deployed. Bug #1 (state-persistence) investigated, not reproduced.

Bug #2 — no merge-base staleness guard: FIXED

Root cause: the default "overlay" proposal-merge strategy runs through muse.core.merge_engine.STRATEGY_MAP["overlay"], configured with diff_unit="snapshot". run_merge() unconditionally treats diff_unit="snapshot" as effective_base={} — i.e. it computes the merge as if from_branch/to_branch share zero history, regardless of whether a real common ancestor is actually resolvable. Combined with resolution="prefer_theirs", that means any path present on from_branch wins outright — even files from_branch never touched since a stale fork point. This is exactly the mechanism behind the 47-file incident described above: dev had ~90 commits of unrelated work since from_branch's July fork point, and all of it looked like a "revert" to the merge algebra because the base was empty, not the real ancestor.

Fix (commit 385e16d0, merged to dev, deployed to staging + production):

• musehub_proposals.py: resolve a real ancestor manifest for "overlay" too (previously only weave/replay/selective/phased), in both the real-merge path and the preview/simulation path.
• proposal_merge_strategies.py::execute_merge_strategy: when strategy is "overlay" and a real ancestor was resolved, swap the engine to STRATEGY_MAP["theirs"] before calling run_merge(). Same resolution="prefer_theirs" contract (from_branch still wins genuine conflicts), but diff_unit="three_way" — a real 3-way diff against the actual ancestor. Paths from_branch never touched are left alone; only real conflicts still resolve to from_branch. When no ancestor is resolvable (bounded BFS gives up), behavior is unchanged.

Verified via a new regression test (TestMergeDoesNotRevertUnrelatedToBranchChanges) that reproduces the incident's shape exactly: an ancestor commit, a from_branch forked there that only adds a new file, and a to_branch that independently advances two other unrelated files. Confirmed red (unrelated to_branch file reverted to stale ancestor content) on unmodified code, green after the fix. Full tests/test_merge_proposals.py + tests/test_musehub_proposals.py run: 137/137 passed, no regressions. Broader sweep across tests/ filtered to merge/overlay/proposal: 772 passed; the only failures/errors were in files this change never touched, confirmed pre-existing.

Bug #1 — state-persistence: investigated, could NOT reproduce

Wrote two targeted regression tests: a real HTTP merge followed by a fresh HTTP read expecting state == "merged", and a second merge attempt on an already-merged proposal expecting 409. Both passed against unmodified code — the persistence gap described in the original report did not reproduce via this path. Also ruled out RDS read-replica lag as a candidate explanation: aws rds describe-db-instances confirms no read replicas are configured on either staging DB instance (musehub-staging-db, musehub-staging-db-v2).

This does not mean bug #1 didn't happen — only that it isn't reproducing under normal HTTP-level test conditions, and the original incident's exact trigger remains unexplained. If a proposal is ever again seen returning merged: true from merge_proposal without state flipping to merged on an immediate follow-up read, that's this same bug recurring — capture the request timing/connection details and reopen.

Closing

Closing this ticket for bug #2, the more consequential and now-fixed half of the original report (real data loss, now root-caused and guarded against). Bug #1 remains a distinct, unexplained mystery — if it recurs, please open a fresh ticket referencing this one rather than reopening, since the reproduction conditions are what's actually missing, not a known fix.