Asyncpg Caps Query Parameters

Asyncpg Caps Query Parameters — Batch mpack_index Upsert

Problem

Pushing a large repo (1289 commits, 12111 objects) failed with HTTP 500 at POST /push/unpack-mpack. The error was a SQLAlchemy IntegrityError at wire_push_unpack_mpack line 567 — the mpack_index upsert.

Root cause: asyncpg caps query parameters at 32767. The mpack_index upsert generates 6 columns per row. For a repo with 12111 objects that works out to 72666 parameters in a single statement, more than double the limit.

asyncpg.exceptions.TooManyParametersError: too many parameters in the query

The push correctly wrote all objects and snapshots before hitting this step. It was only the final mpack_index write that blew up.

Fix

musehub/services/musehub_wire_push.py — batch the mpack_index upsert at 5461 rows per statement (5461 * 6 = 32766, just under the 32767 cap):

# asyncpg caps query parameters at 32767; 6 columns per row -> 5461 rows/batch
_MPACK_IDX_BATCH = 5461
for _batch_start in range(0, len(_mpack_idx_rows), _MPACK_IDX_BATCH):
    _batch = _mpack_idx_rows[_batch_start:_batch_start + _MPACK_IDX_BATCH]
    _idx_stmt = _pg_insert(MusehubMPackIndex).values(_batch)
    await session.execute(
        _idx_stmt.on_conflict_do_update(
            index_elements=["entity_id", "mpack_id"],
            set_={...},
            where=MusehubMPackIndex.byte_offset.is_(None),
        )
    )

Also in this branch

• TDD: mpack content-addressability (tests/test_push_clone_roundtrip.py, tests/test_minio_backend.py) — four-tier test suite proving the full push/clone/verify pipeline works correctly and that every mpack stored in MinIO satisfies sha256(stored_bytes) == key
• fix: snapshot manifest under manifest key not delta_upsert (musehub_wire_shared.py) — _snap_row_to_wire was sending the full manifest as delta_upsert; the client applied it as a delta, causing deleted files to linger and hash_snapshot() to return the wrong ID
• fix: reconstruct manifest for snapshots with null manifest_blob (musehub_wire_fetch.py) — server-side merge commits may have manifest_blob=NULL in the DB; fallback to _reconstruct_manifest via delta chain walk before serializing
• diag: per-stage FETCH-DIAG logging (musehub_wire_fetch.py) — instruments every stage of the blob extraction pipeline (content_cache, mpack_index lookup, _extract_from_mpack, per-object S3 fallback, final tally) so a single muse clone run reveals exactly where objects are lost

Verification

Push of the full gabriel/muse repo (1289 commits, 12111 objects, 377 MB mpack) now completes successfully. Clone produces all 1067 tracked files with correct content and muse verify returns all_ok: true.