Wire push /push/unpack-mpack: synchronous unpack OOMs/times out on large repos

Summary

Pushing muse's full history (1,492 commits, 11,313 blobs, ~389MB mpack) to production repeatedly failed until worked around with a manual staged-push (splitting the push into ~250-400 commit chunks across many separate muse push calls). The root cause is architectural, not configuration, and will recur for any sufficiently large repo pushed in one shot.

Root causes found (three distinct, all confirmed live on musehub.ai)

1. App container OOM — /push/unpack-mpack loads the entire mpack into memory, decodes all objects, and processes them in one synchronous request with no streaming. At ~389MB this exceeded the app container's then-2GiB memory cgroup limit, confirmed via dmesg OOM-killer log entries tied directly to the container's cgroup. Stopgap applied: raised the container memory limit to 3GiB (deploy/deploy.sh, APP_MEMORY_LIMIT).

2. Postgres OOM on bulk insert — the unpack does a single bulk INSERT INTO musehub_commits (...) VALUES (...) for all new commits in one statement/transaction (1,492 rows including sizeable JSONB structured_delta payloads per row). This crashed postgres, which had only a 256MiB memory limit — confirmed via a second dmesg OOM-killer entry naming the postgres process directly. Stopgap applied: bumped postgres's live container memory limit to 1GiB via docker update (not yet captured in any tracked provisioning script — postgres's container has no deploy script of its own, unlike the app/worker containers; this is a real gap worth fixing too, since its config isn't reproducible from source).

3. CPU-bound / timeout ceiling independent of memory — even after fixing both memory limits, a ~230-400 commit chunk pegged the app container at ~100% CPU for minutes, blocking even /healthz on the same worker process (single-threaded/blocking work occupying the whole event loop), and eventually exceeded both Cloudflare's edge timeout (~100s, returns 524 to the client) and some internal supervisor timeout that kills and restarts the worker process (~180-200s in, independent of the OOM fixes). This is why even memory-safe chunks still needed to stay well under ~250-350 commits / ~3000 blobs per single push to reliably complete client-side, even though the fetch+integrity-check steps alone only take ~10-15s.

Also observed: no object-level dedup during unpack

Every staged push reported blobs_skipped: 0, even for chunks whose content almost certainly overlapped with objects already stored from a prior stage covering an adjacent commit range. Total blobs_written summed across all stages of a single branch push exceeded the actual unique blob count for the full repo. This suggests unpack always re-writes objects rather than checking whether they already exist in the object store — wasteful, and likely contributes to the CPU/time cost in point 3.

Real fix (not a stopgap)

Make /push/unpack-mpack asynchronous:

• Client POSTs the mpack reference, server responds immediately with a job ID
• Server processes the unpack as a background job — streaming/chunked, not loading everything into memory at once, and skipping objects that already exist in the store
• Client polls a status endpoint until the job completes or fails, instead of holding one long synchronous HTTP request open

This decouples payload/repo size entirely from any single request's timeout (Cloudflare's, nginx's, or an internal worker's), which is the actual structural problem — the current design has a hard ceiling on pushable repo size no matter how much memory or CPU headroom is added.

Workaround used today (not a permanent solution)

Split the push into ~6-7 sequential muse push calls, each targeting a temporary branch pointing at a commit roughly 250-400 commits further along the history than the last, so each individual unpack request stayed under the ~200s combined memory+CPU+timeout ceiling. Cleaned up the temporary branches afterward. This is viable for a one-time initial publish but not a real solution — it doesn't scale to routine pushes of a repo this size, and depends on manually tuning chunk size per repo.