Closed #89

filed by aaronrene human · 103 days ago

bug: muse clone stuck on fetch/mpack "server busy" (cold have=0 never completes)

0 Anchors

— Blast radius

— Churn 30d

0 Proposals

Issue draft — `muse clone` stuck on `fetch/mpack` “server busy” (pack never ready)

Target repo: gabriel/muse (and/or MuseHub staging ops)
Reporter: aaronrene
Date: 2026-07-13
Client: muse 0.2.0.dev4 (installed via https://staging.musehub.ai/install.sh)
Hub: https://staging.musehub.ai

Summary

Cold muse clone against staging never completes. Auth, hub ping, and muse ls-remote succeed. The clone loop repeatedly POSTs /{owner}/{slug}/fetch/mpack with have=0 and prints “remote preparing fetch data (server busy)” until --retry-timeout expires. No objects are written locally.

This blocks collaborators who need a fresh Muse checkout of large or any repo on this machine (reproduced on both aaronrene/knowtation and gabriel/muse).

What works

Check	Result
`muse auth whoami`	Handle `aaronrene`, hub `staging.musehub.ai`
`muse hub ping --hub https://staging.musehub.ai`	HTTP 200
`muse ls-remote https://staging.musehub.ai/aaronrene/knowtation`	Returns `main` + other branch tips
`muse clone --dry-run …`	Succeeds (no network pack assembly)

What fails

muse clone --depth 1 --branch main --retry-timeout 300 \
  https://staging.musehub.ai/aaronrene/knowtation ~/knowtation-muse

Observed (representative):

[transport] POST https://staging.musehub.ai/aaronrene/knowtation/fetch/mpack  want=2 have=0
⏳ remote preparing fetch data (server busy, attempt N); retrying in 30s …
…
❌ Remote still preparing clone data after 300s — try again shortly.

Same pattern for:

muse clone --depth 1 --branch main --retry-timeout 90 \
  https://staging.musehub.ai/gabriel/muse /tmp/muse-tiny-clone

(want=3 have=0, same busy loop.)

Also reproduced earlier with --retry-timeout 1200 (20 minutes) on aaronrene/knowtation — never recovered.

Installed client transport (…/site-packages/muse/core/transport.py):

Uses fetch/mpack only (fetch/presign string absent).
Staging MuseHub tree for gabriel/muse recently shows a commit titled approximately:
“fix: try fetch/presign before fetch/mpack to avoid Cloudflare…”
That fix does not appear in the 0.2.0.dev4 sdist from install.sh.

Hypothesis A: hub pack assembly / indexing for cold have=0 fetches is stuck or never finishes (503 + Retry-After forever).
Hypothesis B: client should fall back to fetch/presign (per-object GETs) but shipped staging CLI never tries that path, so clones die on mpack prep / Cloudflare limits.

Expected

muse clone of a public or authorized repo completes within a normal budget (or returns a hard, actionable error with request id), and writes a usable .muse/ with objects.

Ask

Confirm whether staging fetch/mpack pack prep is healthy for cold clones (have=[]).
Publish a staging CLI that includes presign-before-mpack (or document how to force fetch/presign).
If this is known 503-while-indexing: surface a terminal error / request id after N retries instead of an indefinite “busy” loop.

Non-goals / not the cause

Not an auth/handle problem (ls-remote + ping work).
Not missing mnemonic / wrong machine password.
Not specific to one private repo (fails on gabriel/muse too).

Workarounds for operators (until fixed)

Copy an existing .muse/ (or full Muse checkout) from a machine that already has objects.
Do not hammer clone (rate limits / Cloudflare 429 on unauthenticated probes).

◎ Activity1

●

aaronrene opened this issue 103 days ago

○

gabriel 2 days ago

Already fixed — confirmed resolved, no code changes needed.

This is RC-5, not a new bug

The exact symptom string you reported — "remote preparing fetch data (server busy, attempt N); retrying in 30s" — is a verbatim match for the message HttpTransport.fetch_mpack's retry loop prints. This is RC-5 from the master Wire Protocol tracker (muse#58), fixed under MWP-5 (closed as muse#59, not #89 — the two tickets were never cross-linked, which is why this stayed open). Root cause and fix, in full: docs/issues/mwp-5-clone-retry-503.md.

Root cause (your Hypothesis A was right)

The server was never actually stuck — fetch/mpack correctly returns a well-formed 503 + Retry-After header while the async prebuild/index pipeline is briefly in flight (two shapes: MPackNotReadyError → Retry-After: 30, FetchNotIndexedError → Retry-After: 60). The client had two bugs: (1) muse/core/transport.py's urllib error handler read err.code but never err.headers, so the Retry-After value was discarded before any caller could see it; (2) every caller (clone, fetch, pull) treated the first 503 as terminal — clone even rmtree'd the half-created target on it. Two commits (1df101e06..., 2026-06-29) added retry_after to TransportError, parsed the header at both HTTPError seams, and wrapped fetch_mpack in a bounded retry loop (120s default budget, honors the server's actual Retry-After, never spins forever) — see the doc's Phase 0-5 for the full TDD trail (20 tests, including a real-urllib E2E that drives an actual throwaway HTTP server to prove the header survives the real HTTPError path, not just a mocked one).

On your Hypothesis B (presign fallback) — tried and deliberately abandoned

The "fix: try fetch/presign before fetch/mpack" commit you spotted in history (2eaa5d95f) was real, but it was reverted the same day (028bebc48, both 2026-06-12) — about two weeks before MWP-5 landed. The team's actual fix was the server-side prebuild/indexing hardening (musehub#106-109, MWP-1 through MWP-4) plus this client-side bounded-retry fix, not a presign-path fallback. So Ask #2 doesn't apply as written — the published staging CLI was never meant to grow a presign fallback; it was meant to stop giving up on the first 503, which it now does.

Why you still hit this on 2026-07-13

Your installed client was 0.2.0.dev4 from install.sh, which predates the 2026-06-29 fix. Staging currently publishes 0.2.1rc12 (install.sh confirmed just now), which includes the retry loop.

Verification just now

tests/test_mwp5_clone_retry.py — all 20 tests pass on current source.
Live muse clone https://staging.musehub.ai/gabriel/muse against staging (1565 commits, 177MB mpack): completed cleanly in ~21s via a prebuild cache hit, zero 503s, zero busy-loop messages.

If you hit this again, muse --version should report 0.2.1rc12 or newer — if it reports 0.2.0.dev4 or similar, re-run install.sh to pick up the fix.

✓

closed this issue 2 days ago

Assignee

○ unassigned

Release

no commits linked to this issue

create

muse hub issue create \
  --title "..." \
  --body "..." \
  --label bug \
  --anchor path/to/file.py::Symbol \
  --commit-anchor <sha> \
  --repo gabriel/muse

read

muse hub issue get 89 --json
muse hub issue list --state open --json

update

muse hub issue edit 89 \
  --anchor path/to/file.py::Symbol \
  --repo gabriel/muse

comment

muse hub issue comment 89 \
  --body "Fixed in <sha>" \
  --repo gabriel/muse

reopen

muse hub issue reopen 89 \
  --repo gabriel/muse

create

create_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  title: "...",
  body: "...",
  labels: ["bug"],
  symbol_anchors: [
    "path/to/file.py::Symbol"
  ],
  commit_anchors: ["<sha>"]
})

read

get_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 89
})

list_issues({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  state: "open"
})

update

edit_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 89,
  symbol_anchors: ["path/to/file.py::Symbol"]
})

comment

create_issue_comment({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 89,
  body: "..."
})

reopen

reopen_issue({
  repo_id: "sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec",
  issue_number: 89
})

create

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{
    "title": "...",
    "body": "...",
    "labels": ["bug"],
    "symbol_anchors": ["path/to/file.py::Symbol"]
  }'

read

# get one issue
curl http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/89

# list open issues
curl "http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues?state=open"

update

curl -X PATCH \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/89 \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"title": "...", "body": "..."}'

comment

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/89/comments \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"body": "Fixed in <sha>"}'

reopen

curl -X POST \
  http://localhost:10003/api/repos/sha256:200e8689fe34a831289bc1eca17633b2069d595379b7c2f57a158e35d8291bec/issues/89/reopen \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\""

bug: muse clone stuck on fetch/mpack "server busy" (cold have=0 never completes)

Issue draft — muse clone stuck on fetch/mpack “server busy” (pack never ready)

Summary

What works

What fails

Client transport note (likely related)

Expected

Ask

Non-goals / not the cause

Workarounds for operators (until fixed)

Issue draft — `muse clone` stuck on `fetch/mpack` “server busy” (pack never ready)