gabriel / musehub public
Open #61 Performance
filed by gabriel human · 14 days ago

Async push unpack model — Phase 3 and 3b

0 Anchors
Blast radius
Churn 30d
0 Proposals

Context

Deferred from issue #56. Push/fetch/clone/pull are working end-to-end. This is a performance and reliability improvement for large repos and high concurrency, not a correctness fix.

Phase 3 — Async unpack model

Replace the synchronous POST /push/unpack-mpack (28s for large repos) with a fast 202 Accepted response that enqueues a background job.

New table: musehub_push_jobs

job_id, repo_id, mpack_key, branch, head_commit_id, force,
status (pending|running|done|failed),
commits_written, objects_written, error_message,
created_at, updated_at

HTTP changes

  • POST /push/unpack-mpack → 202 {job_id, status: "pending", poll_url}
  • New GET /push/jobs/{job_id}{status, commits_written, branch_head, error}

Background worker

Existing worker.py infrastructure. Job claims with atomic CAS (status: pending → running), runs the full inline pipeline, marks done/failed.

Client changes (muse/push.py)

Poll loop with 500ms interval, 5 min deadline. Already sketched in issue #56 pseudocode.

Phase 3b — OID-only scan

First pass extracts oid list only (no content bytes). Second pass loads content only for new oids. Drops RAM from ~59 MB to <1 MB during dedup phase for incremental pushes.

Existing test baseline

  • test_wire_mpack_unpack_step3_e2e.py — 7 tests (tiers 1-3), currently test 200 response
  • test_wire_mpack_unpack_step3_tiers4567.py — 12 tests (tiers 4-7)
  • test_mpack_push_async.py — 3 tests, currently skipped — replace with correct async assertions

All existing tests must be updated from 200 → 202 when Phase 3 ships.

Implementation order

  1. Migration (musehub_push_jobs table)
  2. 202 response + job row creation (fast path only)
  3. Polling endpoint
  4. Background worker (port inline logic)
  5. Client poll loop (muse/push.py)
  6. Phase 3b: OID-only scan
  7. Update all existing tests
Activity
gabriel opened this issue 14 days ago
No activity yet. Use the CLI to comment.