gabriel / muse public
Closed #40 Bug
filed by aaronrene human · 33 days ago

push: 300s PUT timeout too short for large mpack uploads to R2

1 Anchors
Blast radius
Churn 30d
0 Proposals

Summary

muse push fails on large proprietary history pushes because the presigned R2 PUT step uses a 300s default timeout (_TIMEOUT_SECONDS in muse/core/transport.py). Scooling Slice 10 (2026-06-19) hit this twice before succeeding.

Repro

From aaronrene/scooling on staging:

  • 148 commits, 2078 objects, mpack ~35.9 MB (sha256:ba9bd1b4…)
  • Push reaches step 6 (PUT to presigned R2 URL), then fails:
❌ Push failed: The write operation timed out
  • Wall time before failure: ~300–398s (matches _TIMEOUT_SECONDS = 300)
  • Staging tip unchanged; no partial branch advance until mpack fully uploaded + unpack succeeds

Workaround (local only)

Temporarily raised _TIMEOUT_SECONDS to 1800 in local venv transport.py. Third attempt: PUT HTTP 200 at ~401s, then --force needed for unpack (prior timed-out PUT had already landed the mpack). Not a durable fix.

Expected

PUT timeout should scale with payload size (or be configurable, e.g. MUSE_HTTP_TIMEOUT_SECONDS), so ~35 MB uploads on normal home uplinks complete without patching the CLI.

Context

  • Muse CLI 0.2.0rc13, hub staging.musehub.ai
  • Knowtation push (72 commits / ~1 MB) succeeded in ~11s — same code path, smaller mpack
  • Slice 10 eventually PASS: 148 commits indexed, clone clean (148 commits, 1334 blobs, ~7.7s)

Suggested fix

  • Increase default PUT timeout for presigned uploads (e.g. 900–1800s), or
  • Size-aware timeout: max(300, size_bytes / min_bytes_per_sec), or
  • Env override documented in muse push --help
Activity1
aaronrene opened this issue 33 days ago
gabriel 18 days ago

Already fixed — push_mpack_put (muse/core/transport.py) computes a size-aware timeout via _mpack_put_timeout(): scaled at a 50 KB/s floor, clamped to [_PUT_TIMEOUT_FLOOR, _PUT_TIMEOUT_MAX], with a MUSE_PUSH_TIMEOUT_SECONDS env override. The flat 300s constant only remains as an unrelated fallback. Regression coverage exists in tests/test_transport_push_mpack_put_timeout.py (test_pt4-test_pt7) — all 7 tests pass. Closing as already resolved.

closed this issue 18 days ago