[Reconstruction — see #184] committed_at timezone normalization breaks commit identity on non-UTC commits

The original evidence was not lost — it is intact on #182

Nothing needs re-pasting. Two things went slightly wrong in the reconstruction's premise:

1. I never authored #182. Gabriel filed it ("Ok first, I'm unable to clone the knowtation repo … that has been assigned to you", 2026-09-09) and assigned it to me. My contribution was always comments, not an issue body — so a search for an aaronrene-authored ticket in the #178–192 range correctly finds nothing, and no phantom-success write destroyed anything of mine.
2. Both of my comments survived the number collision and are still readable on #182 right now:
   • aaronrene, 2026-09-10T13:29:07Z — 10,158 chars, the full root-cause report
   • aaronrene, 2026-09-11T15:11:27Z — 3,796 chars, the corrected counts and the #63 linkage

They are on #182 because I posted them after the collision reassigned that number, so they now sit under the proposal-merge ticket. Worth copying across to this ticket, but none of it is gone.

More important: #195 and #183 are the same bug, and the #183 diagnosis needs revising

#183 concludes the mpack is incomplete and suspects a stale musehub_fetch_mpack_cache HIT, based on "the server's fetch/mpack response for this clone only contained 79 commits". That figure is commits_written, which is what the client wrote, not what the server sent.

Fresh clone of aaronrene/knowtation from staging, full transport log, today:

[transport] GET done   status=200  size=95.23MB
[transport] unpackb    commits=1220  snaps=956  blobs=1515
[transport] coerce     commits=1220  snaps=956  blobs=1515  heads=0
...
[clone] apply_mpack DONE  blobs_written=1515  blobs_skipped=0  commits_written=79

The server sent 1,220 commits. The mpack is essentially complete — this is not a stale or truncated cache entry. The loss happens entirely client-side, and the cause is the very first commit processed:

❌ Commit sha256:e3574dcaff08ff0d… failed content-hash verification — core fields are
   corrupt (snapshot_id, message, committed_at, or parent IDs).
   Expected   sha256:e3574dcaff08ff0d4a7d80b81c9d7c720727f3aa6e4e062e39224019e2f761c9
   recomputed sha256:9b5565dfd8cdf0339eec15f5a24cb271c294b645ceaea84067cdee2e8035858b
⚠️ apply_mpack: malformed commit — skipped: Refusing to write commit 'sha256:e3574dca…'

That is knowtation's root commit. Once it is refused, every descendant fails for a missing parent. Error census over the whole 1,158-line log:

So: 1 real error, 1,140 consequences. The 79 written commits are the subset whose ancestry does not pass through the poisoned root. The tip is never written, the working tree is empty, and muse clone still exits 0 printing ✅ Cloned.

Why the root fails — reproducible in isolation

committed_at is hashed literally inside the commit identity. MuseHub stores it as timestamptz (musehub_repo_models.py:187) and rebuilds the identity string from the normalised value (musehub_wire_shared.py:379, committed_at=row.timestamp.isoformat()). Recomputing with the installed client's own compute_commit_id (0.2.0rc15) from the record's own fields:

stored committed_at : 2026-03-13T08:37:25-07:00
recomputed          : sha256:e3574dcaff08ff0d…   <-- matches stored commit_id
UTC-normalised      : 2026-03-13T15:37:25+00:00
recomputed          : sha256:9b5565dfd8cdf033…   <-- exactly what clone reports

The 9b5565df… in the clone error is reproduced to all 64 hex digits by normalising the offset. That is the mechanism, confirmed end to end.

This also refutes the "your .muse object store is in an old state" hypothesis by construction: the local record is self-consistent under the current formula. A stale store could not be.

Corrected blast radius

Counted by walking every branch head (not just main) and reading each commit via muse.core.commits.read_commit:

890 of those 890 carry git_sha metadata (891 git-import commits total), so every affected commit is a muse bridge import and zero natively-authored commits are affected. That is why all four muse/musehub × staging/prod clone combinations pass.

Consequences for the fix decision

• A cache-validation fix for #183 alone will not make knowtation clonable — the cache is not the problem.
• Either fix here does: narrow (committed_at_raw + fail-closed check) or structural (#63, canonical bytes).
• No server-side backfill is possible: the original offset strings were never persisted, and -07:00 / +00:00 are the same instant, so a migration cannot recover them. Recovery is a re-push from my local store after the fix lands, which is why the local knowtation store must not be re-cloned over, pruned, or GC'd.

Suggest closing #183 as a duplicate of this ticket, or relabelling it as the symptom with this as the root cause.

Shipped — narrow fix, per gabriel's decision ✅

Added an additive committed_at_raw column (migration 0077) that stores the exact original committed_at ISO string. The serve path (_to_wire_commit) now prefers it verbatim; when absent (legacy rows), it falls back to timestamp.isoformat() exactly as before — unchanged behavior, not newly broken.

Populated at every commit write site: wire push ingest (the real affected path — captures the true client-original string before it's parsed away), proposal merge + rebase replay, and (for consistency) repo-init and sync's synthetic commits.

Scope note, found during implementation: I initially built an automatic "fail closed if a served commit's bytes don't reproduce its id" check and wired it into the general serve path. Reverted it after confirming experimentally it breaks two real things:

1. wire_repair_commit's entire purpose is repairing a commit whose identity fields don't currently reproduce its id — the automatic check fired mid-repair.
2. Two legitimate write sites (repo-init, sync's synthetic content commit) never used hash_commit-derived ids in the first place — filed as #200, since it's a real, separate, pre-existing defect this surfaced.

The verification helper (verify_commit_identity_or_raise) still exists, tested, and available for explicit/diagnostic use (same pattern wire_repair_commit already uses) — it's just not an automatic gate on every serve.

Legacy non-UTC rows (pre-migration, committed_at_raw IS NULL) still serve a mismatched committed_at exactly as before this fix. Not fixed retroactively — that's genuinely out of scope for "narrow" — but explicitly proven not worsened (test_e2_legacy_non_utc_row_missing_raw_is_unchanged_not_worsened).

345/345 across every touched and adjacent test file, no regressions. Deployed and migration applied cleanly on both staging and production.

Along the way: found and cleaned up 17 duplicate (repo_id, number) issue rows in the local dev Postgres container blocking migrations past #184's constraint (pre-existing local cruft, not staging/production), and installed a missing zstandard package in this repo's .venv that was masking real test signal.

Closing.