gabriel / musehub public
feat patch dev #9 / 100
AI Agent gabriel · 6 days ago · Jun 1, 2026 · Diff

feat: resurrect process_mpack_index_job with byte-range indexing (TDD)

Restores the mpack.index background job handler that was deleted. Now: - process_mpack_index_job in wire_push fetches mpack from MinIO, parses MUSE binary or msgpack format, and populates MusehubMPackIndex rows with byte_offset + byte_length using compute_object_byte_offsets - Wire module re-exports process_mpack_index_job - Worker dispatch handles mpack.index job_type (no more 'unknown job_type') - 5 TDD tests covering import, dispatch, missing key, byte offsets, byte range Without byte ranges: 30-35s page loads (390MB download). With them: <1s.

sha256:8a7ff43f27504c1f6abba59ffbab3dc89bd1d8bcfa4c57f6e280286d1a58195a sha
sha256:fa68fb6b242d872f0f8f8a300e4e7bad498a4dd17fdc76ddd87245b460e6aa47 parent
+22 ~15 symbols
sha256:435bebd5328efffa0177879a78a670b7a568e59c580d25b20690b882ff3da8fb snapshot
+22
symbols added
~15
symbols modified
0
dead code introduced
Semantic Changes 37 symbols
~ tests/test_process_mpack_index_job.py .py 19 symbols added
+ AsyncSession import import AsyncSession L21–21
+ MusehubBackgroundJob import import MusehubBackgroundJob L25–25
+ MusehubBranch import import MusehubBranch L26–26
+ MusehubMPackIndex import import MusehubMPackIndex L26–26
+ MusehubRepo import import MusehubRepo L26–26
+ annotations import import annotations L16–16
+ blob_id import import blob_id L23–23
+ compute_identity_id import import compute_identity_id L24–24
+ compute_job_id import import compute_job_id L24–24
+ compute_repo_id import import compute_repo_id L24–24
+ datetime import import datetime L18–18
+ fake_id import import fake_id L23–23
+ pytest import import pytest L19–19
+ select import import select L20–20
+ test_PIJ1_process_mpack_index_job_importable function function test_PIJ1_process_mpack_index_job_importable L33–35
+ test_PIJ2_worker_handles_mpack_index_job_type function function test_PIJ2_worker_handles_mpack_index_job_type L42–49
+ test_PIJ3_missing_mpack_key_raises function async_function test_PIJ3_missing_mpack_key_raises L57–77
+ test_PIJ4_byte_offset_populated_after_job function async_function test_PIJ4_byte_offset_populated_after_job L85–145
+ test_PIJ5_byte_range_locates_correct_bytes function async_function test_PIJ5_byte_range_locates_correct_bytes L149–205
~ musehub/services/musehub_wire.py .py 1 symbol added, 14 symbols modified
+ process_mpack_index_job import import process_mpack_index_job L36–36
~ _COMMIT_BATCH
~ _EMPTY_OID
~ _ObjFetchMap
~ _fetch_commit
~ _fetch_object_meta
~ _topological_sort
~ check_push_anomaly
~ record_mpack_bytes_uploaded
~ wire_fetch_objects
~ wire_push_mpack_presign
~ wire_push_unpack_mpack
~ wire_refs
~ wire_repair_object
~ wire_repair_snapshot
~ musehub/services/musehub_wire_push.py .py 2 symbols added
+ MusehubBackgroundJob import import MusehubBackgroundJob L16–16
+ process_mpack_index_job function async_function process_mpack_index_job L958–1065
~ musehub/worker.py .py 1 symbol modified
~ _process_one
Older fix: enqueue mpack.index after every push (TDD) sha256:fa68fb6b242d872f0f8f8a300e4e7bad498a4dd17fdc76ddd87245b460e6aa47 All commits Newer feat: byte-range blob reads, file attribution DAG walk, branch d… sha256:92528ae07d0e1239d87fd5fd1f439e8fbb49c9778a9a400bc4a736073fb28316

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:8a7ff43f27504c1f6abba59ffbab3dc89bd1d8bcfa4c57f6e280286d1a58195a --body "your comment"