Bundle push security: size gates, content validation, and abuse defense

Which repo

MuseHub, not Muse. Security enforcement must be server-side — client-side checks are UX, not security. Any caller hitting the API directly bypasses the CLI entirely. All four phases live in musehub except minor client hints in Phase 4.

---

Attack surface summary

The bundle push path has four exploitable properties today:

1. No size cap — presign accepts any size_bytes. A 50 GB bundle uploads successfully.
2. Content is opaque until background job runs — illegal content lands in MinIO before any scan.
3. Client-supplied counts are trusted — commits_count=1 with a 10M-object bundle passes the sync path and blows up the indexer.
4. No decompression guard — a 47 MB zstd bundle that expands to 4 GB crashes the background worker.

---

Phase 1 — Size gates (one sitting, zero risk)

These are the cheapest guards with the highest leverage. All changes in musehub/.

1a. Presign endpoint: reject oversized bundles before issuing a URL

musehub/musehub/api/routes/wire.py — push_bundle_presign:

_MAX_BUNDLE_BYTES = 512 * 1024 * 1024  # 512 MB — tune per tier

if size_bytes > _MAX_BUNDLE_BYTES:
    raise HTTPException(
        status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
        detail=f"bundle size {size_bytes:,} bytes exceeds limit {_MAX_BUNDLE_BYTES:,}",
    )

No DB access needed. Fires before the presigned URL is issued, so the client never gets a URL to abuse.

1b. Unpack-bundle: verify wire size after MinIO GET

musehub/musehub/services/musehub_wire.py — wire_push_unpack_bundle, after the MinIO GET:

if len(wire_bytes) > _MAX_BUNDLE_BYTES:
    raise ValueError(
        f"bundle {bundle_key[:20]} exceeds size limit: {len(wire_bytes):,} bytes"
    )

Defends against a client that bypasses the presign check and PUTs directly to MinIO.

1c. Route layer: sanity-bound client-supplied counts

musehub/musehub/api/routes/wire.py — push_unpack_bundle:

_MAX_COMMITS_PER_PUSH  = 50_000
_MAX_OBJECTS_PER_PUSH  = 500_000

if commits_count > _MAX_COMMITS_PER_PUSH or objects_count > _MAX_OBJECTS_PER_PUSH:
    raise HTTPException(status_code=422, detail="commits_count or objects_count exceeds limit")

These are logging inputs, not trusted counts — but bounding them prevents absurd log lines and future code that might use them for allocation.

1d. Move limits to config

All caps (_MAX_BUNDLE_BYTES, _MAX_COMMITS_PER_PUSH, _MAX_OBJECTS_PER_PUSH) should read from musehub/musehub/config.py so they can be tuned per environment (dev vs staging vs prod) without a code change.

Acceptance criteria:

• POST /push/bundle-presign with size_bytes=600_000_000 returns 413
• POST /push/unpack-bundle where MinIO blob exceeds cap raises 422
• All caps live in config, not hardcoded

---

Phase 2 — Bundle content validation (background job)

These run inside the bundle.index background worker, not the sync path. Failures quarantine the bundle rather than crash the worker.

2a. Decompression size guard (zip bomb)

When the background job unpacks objects, track cumulative decompressed bytes:

_MAX_DECOMPRESSED_BYTES = 4 * 1024 * 1024 * 1024  # 4 GB

total_decompressed = 0
for obj in raw_objects:
    content = obj["content"]
    encoding = obj.get("encoding", "")
    if encoding == "zstd":
        raw = zstd.decompress(content)
    else:
        raw = content
    total_decompressed += len(raw)
    if total_decompressed > _MAX_DECOMPRESSED_BYTES:
        raise BundleValidationError("decompressed size exceeds limit — possible zip bomb")

2b. Actual object count verification

After unpacking, verify the actual counts match what the client declared:

if abs(len(raw_objects) - declared_objects_count) > 10:  # small tolerance
    raise BundleValidationError(
        f"object count mismatch: declared {declared_objects_count}, actual {len(raw_objects)}"
    )

2c. Per-object sha256 verification

Every object in the bundle: sha256(content) == object_id. Content-addressing is the proof — verify it here before writing to MinIO.

for obj in raw_objects:
    oid = obj["object_id"]
    _, expected_hex = split_id(oid)
    actual_hex = hashlib.sha256(obj["content"]).hexdigest()
    if actual_hex != expected_hex:
        raise BundleValidationError(f"object {oid[:20]} content does not match declared id")

2d. Quarantine state

Add a quarantine boolean column to musehub_bundles (or equivalent tracking table). On any validation failure:

1. Mark bundle as quarantined
2. Move blob to a non-public MinIO bucket (muse-quarantine)
3. Log the failure with full detail
4. Do NOT advance the branch pointer or write objects to the main bucket
5. Notify gabriel (email or webhook) — configurable

Acceptance criteria:

• A crafted bundle with mismatched object sha256 is quarantined, not indexed
• A zip bomb bundle is caught before full decompression
• Branch pointer is never advanced for a quarantined bundle

---

Phase 3 — Content scanning

This is the legally load-bearing phase. Required before any public launch that allows arbitrary user content.

3a. Known-hash blocklist

Maintain a musehub_blocked_hashes table (object_id text PRIMARY KEY). Before writing any object to MinIO, check this list. Seed it from:

• NCMEC hash lists (CSAM)
• Project VIC
• Internal DMCA takedown history

blocked = await session.execute(
    select(db.MusehubBlockedHash.object_id)
    .where(db.MusehubBlockedHash.object_id.in_(all_object_ids))
)
if blocked.scalars().all():
    raise BundleValidationError("bundle contains blocked content")

O(1) per object against an indexed table — negligible cost.

3b. CSAM scanning integration

For image/binary objects above a size threshold, enqueue a secondary scan job that calls an external CSAM API (Microsoft PhotoDNA or equivalent). Objects stay in quarantine bucket until scan clears.

Architecture:

bundle.index job
  → writes objects to muse-quarantine (not muse-objects)
  → enqueues content.scan job per binary object

content.scan job
  → calls CSAM API
  → on PASS: moves object from muse-quarantine → muse-objects
  → on FAIL: keeps in quarantine, fires alert, suspends repo

3c. DMCA takedown infrastructure

Add POST /admin/takedown endpoint that:

1. Adds object_id(s) to musehub_blocked_hashes
2. Moves existing MinIO objects to quarantine
3. Marks affected repos with a dmca_hold flag
4. Returns a takedown receipt

Acceptance criteria:

• An object matching a blocked hash is rejected at index time
• DMCA takedown endpoint removes object from public access within one request
• CSAM scan failure suspends the repo and fires an alert

---

Phase 4 — Rate limiting and abuse detection

4a. Per-user bundle rate limits

In musehub/musehub/api/routes/wire.py, tighten WIRE_PUSH_LIMIT for the bundle endpoints specifically. Track per-user daily bundle upload bytes in Redis (or pg) and reject when exceeded.

4b. Anomaly detection

Background job: compare current push volume (bytes, objects, commits) against the user's 30-day rolling average. If 10x above average, flag for review rather than auto-reject.

4c. Client-side hints (the one Muse CLI piece)

muse/muse/cli/commands/push.py — before presigning, check bundle size against the server's declared limit (returned in a /caps endpoint or in the presign error). Print a clear message:

❌ Bundle too large: 600 MB exceeds server limit of 512 MB.
   Split your push into smaller increments or contact [email protected].

This is UX, not enforcement. The server still enforces in Phase 1.

---

Implementation order

Phase 1 is one sitting and should ship immediately — it closes the most obvious DoS vector with near-zero risk.

Phase 2 ships with the bundle.index background worker (already planned).

Phase 3 is gated on legal review of which CSAM API to use and which jurisdictions require it. Do not launch public signups without at least 3a (known-hash blocklist).

Phase 4 is operational hardening — do it after the platform has real traffic data to set sensible thresholds.

---

Files touched

Phase	File
1a, 1c	musehub/musehub/api/routes/wire.py
1b	musehub/musehub/services/musehub_wire.py
1d	musehub/musehub/config.py
2a–2d	musehub/musehub/workers/bundle_index.py (new)
2d	musehub/musehub/db/musehub_models.py (quarantine column)
3a	musehub/musehub/db/musehub_models.py (blocked hashes table)
3b	musehub/musehub/workers/content_scan.py (new)
3c	musehub/musehub/api/routes/admin.py
4c	muse/muse/cli/commands/push.py