Fetch and clone: bundle-based download mirrors bundle-based push

Parent ticket

#46 — MuseWire Protocol — GitHub-parity performance across all wire commands

Prerequisite

#45 (push bundle path) must be merged first. Fetch downloads what push uploaded. If push stores a bundle, fetch reads from that same storage model.

The simple version

Push sends one box to the server. Fetch sends one box back to the client. Same box format (MPackBundle). Same one-request principle. Opposite direction.

What fetch does today (the slow path)

For a large fetch (≥ 500 objects OR ≥ 50 MB):

1. Client POSTs want/have to /fetch/presign
2. Server walks DAG, computes delta
3. Server generates N individual presigned GET URLs (one per object)
4. Client downloads N objects in parallel (up to 32 concurrent)
5. Client assembles result locally

N individual presigned URL generations server-side. N individual GET requests client-side. Same envelope-per-object problem as push, just in reverse.

What fetch should do

Five steps

1. NEGOTIATE    client tells server what it wants and what it has
2. PACK         server assembles the delta into one MPackBundle
3. TRANSFER     client downloads the bundle — one request, direct from MinIO/R2
4. UNPACK       client extracts objects, commits, snapshots into local store
5. ADVANCE REF  client updates local branch pointer

Step 1 — Negotiate

• Client sends POST /fetch/bundle with {want, have, ttl_seconds}
• Server walks DAG from want, stops at have
• Clone: have is empty — delta is the entire repo

Step 2 — Pack (server-side)

• A. Walk from want, stop at have
• B. Collect all commits in delta
• C. Collect all snapshot manifests for those commits
• D. Collect all file objects referenced by those snapshots
• E. Assemble as MPackBundle: {commits, snapshots, objects, branch_heads}
• F. bundle_id = sha256(bundle_bytes) — content-addressed
• G. Write bundle to MinIO: backend.put(bundle_id, bundle_bytes)
• H. Generate one presigned GET URL: backend.presign_get(bundle_id, ttl)

Step 3 — Transfer

• Stream path (small): server returns bundle bytes inline in response body
• Presign path (large): server returns {bundle_id, presigned_url, branch_heads}
  • Client does one GET presigned_url → bundle_bytes
  • Cloudflare never sees the object data

Step 4 — Unpack (client-side)

• A. Verify sha256(bundle_bytes) == bundle_id — one check covers all contents
• B. msgpack.unpackb(bundle_bytes) → extract commits, snapshots, objects
• C. Write each file object to local store (write_object)
• D. Write each snapshot to local store (write_snapshot)
• E. Write each commit to local store (write_commit)
• No per-object hash loop. Step A is the integrity check for everything.

Step 5 — Advance ref

• Update local branch pointer to want tip

New endpoint

POST /{owner}/{slug}/fetch/bundle

Request body (msgpack):

{
  "want":        ["sha256:…"],   // commits the client wants
  "have":        ["sha256:…"],   // commits the client already has
  "ttl_seconds": 3600
}

Response (presign path):

{
  "presign":       true,
  "bundle_id":     "sha256:…",
  "presigned_url": "https://…",  // one GET URL for the entire bundle
  "branch_heads":  {"main": "sha256:…", "dev": "sha256:…"},
  "commit_count":  1024,
  "object_count":  5139
}

Response (stream path):

{
  "presign": false,
  "bundle":  <msgpack bytes>  // bundle inline
}

Server-side implementation — wire_fetch_bundle

async def wire_fetch_bundle(
    session: AsyncSession,
    repo_id: str,
    want: list[str],
    have: list[str],
    ttl_seconds: int = 3600,
) -> dict:
    # 1. Walk DAG — compute delta commits
    delta_commits = await _compute_fetch_delta(session, repo_id, want, have)

    # 2. Collect snapshot IDs and object IDs for delta
    snap_ids = {c.snapshot_id for c in delta_commits if c.snapshot_id}
    snapshots = await _load_snapshots(session, repo_id, snap_ids)
    all_oids = {oid for s in snapshots for oid in s["manifest"].values()}

    # 3. Load object bytes from MinIO in parallel
    objects = await _load_objects_parallel(all_oids)

    # 4. Assemble bundle
    bundle = {
        "commits":      [c.to_dict() for c in delta_commits],
        "snapshots":    snapshots,
        "objects":      [{"object_id": oid, "content": data} for oid, data in objects.items()],
        "branch_heads": await _get_branch_heads(session, repo_id),
    }
    bundle_bytes = msgpack.packb(bundle, use_bin_type=True)
    bundle_id = "sha256:" + hashlib.sha256(bundle_bytes).hexdigest()

    # 5. Write bundle to MinIO, issue presigned GET URL
    await backend.put(bundle_id, bundle_bytes)
    presigned_url = await backend.presign_get(bundle_id, ttl=ttl_seconds)

    return {
        "presign":       True,
        "bundle_id":     bundle_id,
        "presigned_url": presigned_url,
        "branch_heads":  bundle["branch_heads"],
        "commit_count":  len(delta_commits),
        "object_count":  len(objects),
    }

Client-side implementation — _run_fetch_bundle_path

In muse/muse/cli/commands/fetch.py (or equivalent fetch transport):

async def _run_fetch_bundle_path(response_data) -> FetchResult:
    bundle_id    = response_data["bundle_id"]
    presigned_url = response_data["presigned_url"]

    # One GET — entire bundle
    async with httpx.AsyncClient() as client:
        resp = await client.get(presigned_url)
    bundle_bytes = resp.content

    # Verify — one hash covers all contents
    actual_id = "sha256:" + hashlib.sha256(bundle_bytes).hexdigest()
    assert actual_id == bundle_id, f"bundle integrity failure: {bundle_id} != {actual_id}"

    # Unpack
    bundle = msgpack.unpackb(bundle_bytes, raw=False)
    for obj in bundle.get("objects", []):
        write_object(root, obj["object_id"], obj["content"])
    for snap in bundle.get("snapshots", []):
        write_snapshot(root, snap)
    for commit in bundle.get("commits", []):
        write_commit(root, commit)

    return FetchResult(
        commits=bundle["commits"],
        branch_heads=bundle["branch_heads"],
    )

Content-addressing checklist

Performance target

Bundle cleanup

Bundles are transient transfer artifacts. After the client confirms receipt (or after TTL), delete the bundle blob from MinIO. Individual objects remain under their own sha256 keys.