Wire Protocol

Overview

Client                                              Server
──────                                              ──────
GET  /{owner}/{slug}/refs                       →   branch heads + repo metadata

── PUSH ─────────────────────────────────────────────────────────────────────
POST /{owner}/{slug}/push/mpack-presign         →   {mpack_key, size_bytes}
                                                ←   {upload_url, mpack_key}
PUT  <upload_url>                               →   MPack binary (direct to R2/MinIO)
POST /{owner}/{slug}/push/unpack-mpack          →   {mpack_key, branch, head, …}
                                                ←   {commits_written, snapshots_written, blobs_written}

── FETCH: single-MPack protocol (primary) ───────────────────────────────────
POST /{owner}/{slug}/fetch                      →   {want, have}
                                                ←   {mpack_id, mpack_url, commit_count, object_count}
GET  <mpack_url>                               →   MPack binary (application/x-muse-pack)
                                                    verify sha256(body) == mpack_id, then apply

── FETCH: per-object presign (large repos) ──────────────────────────────────
POST /{owner}/{slug}/fetch/presign              →   {want, have, ttl_seconds}
                                                ←   {presign, object_urls, commits, snapshots, …}
GET  <object_urls[oid]>                        →   raw object bytes (direct from R2/MinIO)

Push is always three steps: the client obtains a presigned PUT URL from the server (push/mpack-presign), uploads the MPack binary directly to R2 or MinIO, then calls push/unpack-mpack to instruct the server to fetch, SHA-256 verify, parse, and index the binary. No object bytes flow through the server on push.

Fetch has two paths. The primary path (fetch) returns a single presigned URL for a content-addressed MPack covering the full commit delta — ideal for most clones and pulls. The presign path (fetch/presign) returns per-object presigned GET URLs for very large repos where parallel downloads outperform a single large file. Both paths produce identical logical results.

Push — MPack upload

A push is always three steps. The client builds a single MPack binary containing all new objects, commits, and snapshots, then transfers it directly to object storage via a presigned PUT URL. The server never sees the object bytes in transit — it fetches the binary from storage itself during unpack.

Step 1 — POST /push/mpack-presign

The client declares the MPack it is about to upload and receives a time-limited presigned PUT URL. MSign authentication is required.

POST /{owner}/{slug}/push/mpack-presign
Content-Type: application/x-msgpack
Authorization: MSign …

// request
{
  "mpack_key":  "sha256:<64-hex>",  // sha256: of the MPack binary the client will upload
  "size_bytes": 1048576           // byte length of the MPack binary
}

// response
{
  "upload_url": "https://<bucket>.r2.cloudflarestorage.com/mpacks/sha256:ac7…?X-Amz-Signature=…",
  "mpack_key":  "sha256:ac7…"
}

The server validates that mpack_key starts with "sha256:" and is exactly 71 characters (7-char prefix + 64 hex). It also enforces a per-user daily upload byte quota. The presigned URL expires after 3600 seconds.

Step 2 — PUT MPack to upload_url

Upload the MPack binary directly to the presigned URL. No MSign header is needed — the URL is self-authenticating and scoped to one object key.

PUT https://<bucket>.r2.cloudflarestorage.com/mpacks/sha256:ac7…?X-Amz-Signature=…
Content-Length: 1048576

<MPack binary — application/x-muse-pack>

The MPack binary must be a valid Muse MPack: b"MUSE" magic, section table, section data, and 32-byte SHA-256 footer. The SHA-256 of the binary (excluding the 32-byte footer) must equal the mpack_key declared in step 1 — the server verifies this in step 3.

Step 3 — POST /push/unpack-mpack

Instruct the server to fetch the uploaded MPack from storage, verify its SHA-256, parse all sections, and index commits, snapshots, and objects into the database.

POST /{owner}/{slug}/push/unpack-mpack
Content-Type: application/x-msgpack
Authorization: MSign …

// request
{
  "mpack_key":      "sha256:ac7…",  // must match the key used in step 1
  "branch":         "dev",          // branch to advance
  "head":           "sha256:abc…",  // commit ID the branch will point to after push
  "commits_count":  3,             // advisory — used for progress logging
  "blobs_count":    47,            // advisory — used for progress logging
  "force":          false          // true = force-push (non-fast-forward)
}

// response — success
{
  "commits_written":   3,
  "snapshots_written": 3,
  "blobs_written":     47
}

Error responses

MPack binary format

A Muse MPack (application/x-muse-pack) is the unit of transfer for both push and fetch. It carries commits, snapshots, objects, and tags in a single self-verifying binary. It is not msgpack — msgpack.unpackb() will fail on it.

offset  field             size   notes
──────  ────────────────  ─────  ────────────────────────────────────────────
0x00    magic             4 B    b"MUSE"
0x04    version           1 B    currently 1
0x05    section_count     1 B    N sections follow
0x06    section table     N×17B  per section: 1B type + 8B offset LE + 8B length LE
        section data      …      sections concatenated, no padding between them
-32     footer            32 B   SHA-256 of every byte above (not self-referential)

Section types

Integrity model

The last 32 bytes of every MPack are SHA-256(all_preceding_bytes). This value equals the mpack_id / mpack_key used throughout the protocol (with the sha256: prefix stripped). Clients must verify this before calling apply_mpack(). The server performs the same check during push/unpack-mpack.

// Python verification — required before apply_mpack()
import hashlib
body   = mpack_bytes[:-32]
footer = mpack_bytes[-32:]
assert hashlib.sha256(body).digest() == footer
assert mpack_id == "sha256:" + footer.hex()

Byte-level layout — 5-section MPack header

offset  hex              decoded
──────  ───────────────  ──────────────────────────────────────────────
0x00    4d 55 53 45      magic → b"MUSE"
0x04    01               version → 1
0x05    05               section_count → 5
        ── section table (5 × 17 = 85 bytes) ──────────────────────────
0x06    01  <offset LE>  <len LE>   section type 1 (OBJECTS)
0x17    02  <offset LE>  <len LE>   section type 2 (COMMITS)
0x28    03  <offset LE>  <len LE>   section type 3 (SNAPSHOTS)
0x39    04  <offset LE>  <len LE>   section type 4 (TAGS)
0x4a    05  <offset LE>  <len LE>   section type 5 (META)
        ── section data (in declaration order) ────────────────────────
0x5b    <OBJECTS bytes>
        <COMMITS bytes>
        <SNAPSHOTS bytes>
        <TAGS bytes>
        <META bytes>
-32     <32-byte SHA-256 footer>

Snapshot delta encoding

Snapshots inside an MPack are delta-encoded, not full manifests. The SNAPSHOTS section carries one entry per snapshot in commit-graph order (oldest first). Each entry stores only the files that changed relative to the previous snapshot, cutting transfer size by 10–100× on typical commit chains.

SnapshotDelta schema

{
  "snapshot_id":        "sha256:…",  // ID of the *full* manifest — the proof
  "parent_snapshot_id": "sha256:…",  // null for the root snapshot
  "delta_upsert":  { "src/main.py": "sha256:…" },  // added or modified files
  "delta_remove": ["old/path.py"]               // deleted files
}

Reconstruction

Apply deltas oldest-first. The hash at the end is the integrity proof — no external validation is needed.

resolved = {}  # snapshot_id → full manifest dict

for entry in snapshots:  # oldest first
    parent = resolved.get(entry["parent_snapshot_id"], {})
    manifest = dict(parent)
    manifest.update(entry["delta_upsert"])
    for path in entry["delta_remove"]:
        del manifest[path]
    assert compute_snapshot_id(manifest) == entry["snapshot_id"]  # hash IS the proof
    resolved[entry["snapshot_id"]] = manifest

The first entry always has parent_snapshot_id: null and delta_upsert equal to the complete manifest. Every subsequent entry carries only changed paths. A receiver can verify every snapshot independently — no trust is placed in the sender's delta computation.

Fetch — MPack download

POST /{owner}/{slug}/fetch — the primary fetch endpoint. The server walks the commit DAG to find the delta between want and have, assembles a single MPack binary, stores it ephemerally in the object store, and returns a presigned GET URL. The client downloads one file, verifies the SHA-256 footer, and applies it.

Step 1 — POST /fetch

POST /{owner}/{slug}/fetch
Content-Type: application/x-msgpack
Authorization: MSign …  (required for private repos; optional for public)

// request
{
  "want": ["sha256:abc…"],  // commit IDs to fetch (max 1,000); must start with "sha256:"
  "have": ["sha256:xyz…"]   // commits already local — server omits their objects (max 1,000)
}

// response — delta available
{
  "mpack_id":     "sha256:ac7…",                                           // sha256: of the MPack binary
  "mpack_url":    "https://minio…/mpacks/sha256:ac7…?X-Amz-Signature=…",  // presigned GET URL
  "commit_count": 3,
  "object_count": 47
}

// response — already up-to-date (empty want, or client has everything)
{
  "mpack_id":     null,
  "mpack_url":    null,
  "commit_count": 0,
  "object_count": 0
}

Step 2 — GET and verify the MPack

Download mpack_url directly — no MSign header needed, the URL is self-authenticating. Verify the SHA-256 footer before writing anything to disk.

GET <mpack_url>
// 200 OK  Content-Type: application/x-muse-pack

// verify before applying
body   = response.content
assert hashlib.sha256(body[:-32]).digest() == body[-32:]
assert mpack_id == "sha256:" + body[-32:].hex()

pack = parse_wire_mpack(body)
apply_mpack(repo_root, pack)

503 — objects not yet indexed

After a push, a background job promotes objects to permanent storage and updates the MPack index. If a fetch arrives before indexing completes, the server returns 503 with a Retry-After header.

HTTP/1.1 503 Service Unavailable
Retry-After: 60

Objects not yet indexed: 12 missing. Retry shortly.

Other error responses

Rate limit: 120 requests / minute.

Fetch — per-object presign

POST /{owner}/{slug}/fetch/presign — returns per-object presigned GET URLs for large fetches. The server walks the commit DAG, generates a presigned GET URL for each needed object via asyncio.gather with a semaphore of 50 (zero object bytes are read), and returns the full set along with commit and snapshot metadata. The client downloads all objects directly from R2 or MinIO, bypassing the server entirely.

POST /{owner}/{slug}/fetch/presign
Content-Type: application/x-msgpack
Authorization: MSign …  (required for private repos; optional for public)

// request
{
  "want":        ["sha256:abc…"],  // commit IDs to fetch (max 1,000)
  "have":        ["sha256:xyz…"],  // commits already local (max 1,000)
  "depth":       null,              // optional — shallow fetch depth
  "ttl_seconds": 3600              // presigned URL lifetime (default 3600)
}

// response
{
  "presign":        true,              // false if backend does not support presigned URLs
  "object_urls":    {                  // oid → presigned GET URL; empty when presign=false
    "sha256:abc…": "https://r2…?X-Amz-Signature=…"
  },
  "commits":        [ /* WireCommit[] — full delta */ ],
  "snapshots":      [ /* WireSnapshot[] — manifest included */ ],
  "branch_heads":   { "main": "sha256:…" },
  "repo_id":        "sha256:…",
  "domain":         "code",
  "default_branch": "main",
  "expires_at":     "2026-05-18T13:00:00Z",  // null when presign=false
  "commit_count":   3,
  "object_count":   47
}

When presign=false (backend limitation), object_urls is empty but all metadata fields are still populated — the client can fall back to fetch/objects for the actual bytes.

Fetch — individual objects

POST /{owner}/{slug}/fetch/objects — returns raw bytes for a list of object IDs as concatenated msgpack frames, one frame per found object. Used as a fallback when presigned URLs are not available, or to retrieve individual objects by ID.

POST /{owner}/{slug}/fetch/objects
Content-Type: application/x-msgpack

// request
{
  "object_ids": ["sha256:abc…", "sha256:def…"]
}

// response — concatenated msgpack frames (one per found object)
{ "object_id": "sha256:abc…", "content": <bytes> }
{ "object_id": "sha256:def…", "content": <bytes> }
// objects not found in storage are silently omitted

The response body is a stream of self-delimiting msgpack values — read frames until EOF. Objects not found in storage are omitted without error. Auth is optional on public repos; required on private repos.

Refs

GET /{owner}/{slug}/refs returns the current branch heads and repo metadata. The client calls this first to discover what the remote has before deciding which commits to push or fetch. Private repos return 404 to unauthenticated callers — same response as a non-existent repo to avoid existence leaks.

// WireRefsResponse
{
  "repo_id":        "sha256:…",
  "domain":         "code",
  "default_branch": "main",
  "branch_heads":   {
    "main": "sha256:abc…",
    "dev":  "sha256:def…"
  }
}

Rate limit: 120 requests / minute (no auth required for public repos).

Wire types

These record types appear in the COMMITS and SNAPSHOTS sections of every MPack (on both push and fetch) and in fetch/presign response JSON.

WireCommit fields

WireSnapshot fields

WireSnapshotDelta fields (inside MPack SNAPSHOTS section)

Integrity & signatures

Object integrity

Every object is content-addressed. The OBJECTS section of an MPack stores objects keyed by their SHA-256 ID. On unpack the server decompresses each object and recomputes SHA-256(raw bytes) — if the result does not match the declared ID the entire push is rejected. No separate checksum field exists — content-addressing is the integrity check.

MPack integrity

The 32-byte SHA-256 footer at the end of every MPack binary covers all preceding bytes. Both the server (during push/unpack-mpack) and the client (after downloading a fetch MPack) verify this footer independently. A single-bit corruption anywhere in the binary is detected before any data is written.

Commit signature verification

When a WireCommit arrives with a non-empty signature and signer_public_key, the server cryptographically verifies the Ed25519 signature before accepting the push. The verification input is the SHA-256 of the canonical provenance payload:

// provenance payload format (v2)
"muse-provenance-v2\n" +
commit_id "\x00" author "\x00" agent_id "\x00" model_id "\x00"
toolchain_id "\x00" prompt_hash "\x00" committed_at

payload = SHA-256(above)
valid   = Ed25519Verify(signer_public_key, payload, signature)

Any verification failure rejects the entire push with 422 — a single bad commit invalidates the batch atomically. A commit that has signature set but an empty signer_public_key is also rejected: without the public key verification is impossible.

Unsigned commits (signature: "") are governed by the repo-level require_signed_commits setting. When off (the default), unsigned commits are accepted — their absence of provenance signals a human acted directly. When on, every commit must carry a valid signature or the push is rejected.

MSign request authentication

Write endpoints require an Authorization: MSign … header on the HTTP request itself. MSign uses the same Ed25519 keypair used for commit signing — one key authenticates both the request and the commit's authorship claims. The request signature binds method, path, timestamp, and body hash, preventing replay and MITM attacks.

Storage backends

The server uses a single backend — BlobBackend — for all object and MPack storage. It is content-addressed and S3-compatible, targeting Cloudflare R2 in production and MinIO in local dev. Presigned URLs (PUT and GET) are supported on all targets.

Backend selection (get_backend())

Key format

Local dev vs staging vs prod

Invariants

• The server always stores raw (decompressed) bytes. Wire-level compression is stripped before storage.
• The object ID is always SHA-256(raw bytes). Content-addressing is the integrity check: a stored object whose bytes do not hash to its key is corrupt.
• The blob key never includes a repo prefix — objects/sha256:abc…, not gabriel/muse/objects/sha256:abc…. The object namespace is global across all repos on the same bucket.

Limits

All wire endpoints