Async push unpack model — Phase 3 and 3b
Context
Deferred from issue #56. Push/fetch/clone/pull are working end-to-end. This is a performance and reliability improvement for large repos and high concurrency, not a correctness fix.
Phase 3 — Async unpack model
Replace the synchronous POST /push/unpack-mpack (28s for large repos) with a fast 202 Accepted response that enqueues a background job.
New table: musehub_push_jobs
job_id, repo_id, mpack_key, branch, head_commit_id, force,
status (pending|running|done|failed),
commits_written, objects_written, error_message,
created_at, updated_at
HTTP changes
POST /push/unpack-mpack→ 202{job_id, status: "pending", poll_url}- New
GET /push/jobs/{job_id}→{status, commits_written, branch_head, error}
Background worker
Existing worker.py infrastructure. Job claims with atomic CAS (status: pending → running), runs the full inline pipeline, marks done/failed.
Client changes (muse/push.py)
Poll loop with 500ms interval, 5 min deadline. Already sketched in issue #56 pseudocode.
Phase 3b — OID-only scan
First pass extracts oid list only (no content bytes). Second pass loads content only for new oids. Drops RAM from ~59 MB to <1 MB during dedup phase for incremental pushes.
Existing test baseline
test_wire_mpack_unpack_step3_e2e.py— 7 tests (tiers 1-3), currently test 200 responsetest_wire_mpack_unpack_step3_tiers4567.py— 12 tests (tiers 4-7)test_mpack_push_async.py— 3 tests, currently skipped — replace with correct async assertions
All existing tests must be updated from 200 → 202 when Phase 3 ships.
Implementation order
- Migration (musehub_push_jobs table)
- 202 response + job row creation (fast path only)
- Polling endpoint
- Background worker (port inline logic)
- Client poll loop (muse/push.py)
- Phase 3b: OID-only scan
- Update all existing tests