gabriel / musehub public
Closed #197 analysis
filed by gabriel human · 35 days ago

mpack unpack has no streaming (3GB memory stopgap) — risk for the planned knowtation recovery re-push

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Summary

Deploy-side notes call out that mpack unpack "loads the whole payload + all decoded objects into memory at once (no streaming)," with a 3 GB limit documented as a stopgap rather than a real fix.

Why this matters now

knowtation is a large repo, and the recovery push planned once #195 (committed_at identity mismatch) is fixed may re-push a substantial payload. If that payload's decoded size approaches or exceeds the 3 GB stopgap, the re-push could fail on the memory limit even after the underlying identity bug is fixed — turning one blocker into a second one right behind it.

Ask

  • Confirm the actual in-memory footprint the knowtation re-push is likely to hit (decoded objects, not just wire-payload size) and whether it's within the current 3 GB stopgap.
  • If it's close or over: either raise the stopgap for this one operation, or prioritize adding real streaming to mpack unpack ahead of the knowtation recovery push, not after.
  • #195 — the committed_at fix that must land before this recovery push can happen at all.
  • #61 — async push/unpack model work, may be the natural home for a streaming redesign if one is undertaken.

Source

Reported by aaronrene, 2026-09-10 (relayed via message to gabriel, processed 2026-09-11): "Related and worth knowing before the knowtation recovery push: deploy.sh:41-46 notes mpack unpack 'loads the whole payload + all decoded objects into memory at once (no streaming)' and calls the 3g limit a stopgap. knowtation is large, so that re-push may hit it even after #182 is fixed."

◎ Activity1
●
gabriel opened this issue 35 days ago
○
gabriel 35 days ago

Resolved via direct measurement — no code change needed for the knowtation recovery push specifically.

Findings

Queried staging directly: knowtation's total object footprint (all objects reachable from the repo, via musehub_object_refs JOIN musehub_objects) is:

n_objects = 4,737
raw_bytes = 165,466,229  (~158 MB)

That's the full repo's object payload — the worst case for a from-scratch re-push, since #195 is now fixed and deployed and the recovery push can proceed. Even assuming decoded-in-memory overhead runs several times the raw byte count (Python object overhead, msgpack decode buffers, etc.), 158 MB has enormous headroom under the current 3 GB stopgap — roughly 19x before even accounting for overhead multipliers.

Answering the two asks

  1. Actual in-memory footprint vs. the 3 GB stopgap: not close. knowtation is not a large repo by the metric that matters here (total object bytes), whatever its commit count or history length.
  2. Raise the stopgap or prioritize streaming ahead of the recovery push: neither is needed for this specific push. The stopgap is fine as-is for knowtation.

Not closing the door on the underlying issue

The lack of real streaming in mpack unpack is still a real structural gap for future large repos — just not a blocker for knowtation specifically. That work remains tracked under #61 (async push/unpack model), which is the right home for a streaming redesign if/when a repo actually approaches the stopgap.

Closing this ticket since its specific ask (de-risk the knowtation recovery push) is answered: it's not at risk.

✓
closed this issue 35 days ago