# Task: Spot Check MWP — deep verification of #58's "definition of MVP done"

## Background

[Issue #58](https://staging.musehub.ai/gabriel/muse/issues/58) — "Muse Wire
Protocol MVP hardening" — lists 8 sub-tickets (MWP-1..MWP-8), all now closed
with green test suites:

| Sub-ticket | Repo | Fixes | Status |
|---|---|---|---|
| MWP-1 | musehub#106 | RC-1 (gen-0 fallback corrupts fetch walk) | ✅ closed |
| MWP-2 | musehub#107 | RC-2 (`_walk_commit_delta` no fallback) | ✅ closed |
| MWP-3 | musehub#108 | RC-3 (job enqueue drops back-to-back pushes) | ✅ closed |
| MWP-4 | musehub#109 | RC-4 (prebuild/index ordering race) | ✅ closed |
| MWP-5 | muse#59 | RC-5 (client doesn't retry on 503) | ✅ closed |
| MWP-6 | musehub+muse | RC-6 (wire suite disabled) | ✅ closed |
| MWP-7 | muse#62 | RC-7 (#55 sibling-ref negotiation, #32 dry-run) | ✅ closed |
| MWP-8 | musehub#111 | RC-8 (repair/force-push don't bust fetch cache) | ✅ closed |

Every sub-ticket shipped with unit/integration tests that pass in isolation
(commit-by-commit, phase-by-phase). **What has not been independently proven
is the compound claim in #58's "Goal — definition of MVP done":** that a real
developer, running the real CLI against real staging, in the order they'd
actually use it, gets correct behavior every time — including at the seams
between fixes (e.g. does MWP-8's cache invalidation interact correctly with
MWP-3/4's job-ordering fix? does MWP-5's retry loop actually fire against a
*real* 503 from staging, not a mocked one?).

This is the gap this spot-check closes. **We are not re-reviewing whether
each MWP's unit tests are well-written — they are.** We are asking: *does the
system, as a whole, running as a black box against staging, satisfy the five
numbered claims in #58's goal section?* Only after this is true do we bring
in aaronrene.

---

## Goal — definition of MVP done (verbatim from #58)

> aaronrene (or any dev) can, against staging, reliably:
>
> 1. `muse clone <url>` — get a complete, correct working tree every time,
>    including immediately after a push (with bounded client retry on 503).
> 2. `muse push` — upload only the true delta; never re-upload objects
>    already on the remote via any ref; survive back-to-back pushes.
> 3. `muse fetch` / `muse pull` — receive exactly the new commits/blobs and
>    converge.
> 4. Do all of the above repeatedly with no stale clones, no phantom large
>    uploads, no manual cache-busting.
> 5. The wire regression suite is **re-enabled and green** as the proof.

Each claim below is broken into: **what fixed it**, **what's already proven**
(from the sub-ticket's own test suite), **the residual gap** (what unit tests
can't tell us), and **the concrete `/tmp` CLI recipe** that closes the gap
against live staging.

---

### Claim 1 — `muse clone` is always complete and correct, including immediately after a push

**Fixed by:** MWP-1 (gen-0 authority), MWP-2 (DAG-walk fallback), MWP-5
(bounded 503 retry), MWP-8 (cache never stale after repair/force-push).

**Already proven (unit/integration level):**
- MWP1_14 — push→clone→push→clone E2E gate (musehub, in-process).
- MWP2_12 — fallback alone ships a complete clone even with repair patched out.
- MWP5 real-urllib E2E against a live localhost server (retry loop fires on
  real 503).
- EX_02 (MWP-8) — clone-after-repair E2E, `@pytest.mark.wire`.

**Residual gap:**
- All of the above run against **localhost** or **in-process fixtures**, not
  **staging** (real network latency, real R2, real multi-second prebuild
  time). The 503→retry window on localhost may not exercise the same timing
  as staging's actual prebuild latency.
- No test proves clone correctness *chained* across a push→clone→push→clone
  cycle that also crosses a *force-push* or *repair* in the middle — i.e. the
  MWP-1/2 fix and the MWP-8 fix have never been exercised in the same
  sequence.
- No test proves clone works correctly for a **brand-new repo that has never
  been pushed to before** against staging specifically (first-ever clone,
  cold cache, real MISS→prebuild→retry→HIT path end to end).

**`/tmp` recipe:**
```bash
# 1a — cold clone of a never-touched repo (first MISS, real prebuild latency)
cd /tmp && rm -rf mwp-qa-1a && mkdir mwp-qa-1a && cd mwp-qa-1a
muse init --json
echo "hello" > README.md && muse code add README.md
muse commit -m "init" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-1a
muse push staging main --json   # first push to a brand-new repo

cd /tmp && rm -rf mwp-qa-1a-clone
time muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1a-clone --json
# expect: complete tree, README.md present, exit 0 — measure how long the
# 503-retry loop actually waited (proves MWP-5's bounded poll against real
# staging timing, not a mock)

# 1b — push then immediate clone (the exact MWP-1/RC-1 regression)
cd /tmp/mwp-qa-1a
echo "v2" >> README.md && muse code add README.md
muse commit -m "v2" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json
cd /tmp && rm -rf mwp-qa-1b-clone
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-1b-clone --json
diff /tmp/mwp-qa-1a/README.md /tmp/mwp-qa-1b-clone/README.md   # must be identical, must contain "v2"
```

---

### Claim 2 — `muse push` uploads only the true delta, never re-uploads objects reachable via any ref, survives back-to-back pushes

**Fixed by:** MWP-3 (job enqueue reflects latest tip), MWP-4 (prebuild/index
ordering), MWP-7 (sibling-ref have-negotiation, dry-run parity).

**Already proven:**
- MWP3_10/11 — two-push full drain, N=10 stress partial drain (in-process job
  queue simulation).
- MWP4_11/12 — E2E claim loop + N=10 stress, zero `FetchNotIndexedError`.
- MWP-7 Phase 3 (VS) — real staging push, 17-commit dev + 1-commit main,
  `commits_sent=1, objects_sent=1`.

**Residual gap:**
- MWP-3/4's stress tests simulate concurrent job claiming in-process — they
  do **not** prove that a human issuing two real `muse push` commands back to
  back (i.e. two separate CLI processes, two separate HTTP round-trips) against
  staging converges correctly. Real back-to-back means real race conditions in
  the async worker, not simulated ones.
- No test proves the **cross-branch dedup** claim ("never re-upload objects
  already on the remote via any ref") for a *third* sibling branch beyond the
  two-branch case MWP-7 tested. If `feature-a`, `feature-b`, and `dev` all
  share a base and `feature-b` is pushed last, does it dedup against **both**
  siblings or just the most-recently-pushed one?
- `--dry-run` parity was fixed for a single scenario (MWP-7 Phase 4/5) — needs
  re-verification against a *different* topology (e.g. a rebased branch) to
  rule out a narrow fix.

**`/tmp` recipe:**
```bash
# 2a — literal back-to-back push (two real CLI invocations, no artificial delay)
cd /tmp/mwp-qa-1a
echo "a" >> README.md && muse code add README.md
muse commit -m "a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json &
echo "b" >> README.md && muse code add README.md    # NOTE: run after push A's mpack-presign step ideally overlaps; simplest form below
wait
echo "b" >> README.md && muse code add README.md
muse commit -m "b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json    # push B immediately after A returns
cd /tmp && rm -rf mwp-qa-2a-clone && muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-2a-clone --json
tail -3 mwp-qa-2a-clone/README.md    # must show both "a" and "b"

# 2b — three-sibling-branch dedup
cd /tmp/mwp-qa-1a
muse checkout -b feature-a && echo "fa" > fa.txt && muse code add fa.txt
muse commit -m "feature a" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-a --json
muse checkout main && muse checkout -b feature-b && echo "fb" > fb.txt && muse code add fb.txt
muse commit -m "feature b" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging feature-b --json --json | jq '.objects_sent'   # should exclude blobs already on staging via feature-a's shared base

# 2c — dry-run parity on a rebased topology
muse rebase main --dry-run --json    # compare counts to a real rebase + push
```

---

### Claim 3 — `muse fetch` / `muse pull` receive exactly the new commits/blobs and converge

**Fixed by:** MWP-1/2 (correct generation walk), MWP-7 (sibling-ref
negotiation applies to fetch's `have` set too).

**Already proven:**
- Fetch-side coverage is implied by the same MWP-1/2 test suite (fetch and
  clone share `_walk_commit_delta`).

**Residual gap:**
- No sub-ticket wrote a **dedicated fetch/pull convergence test against
  staging** — all direct evidence is clone-shaped (`have=[]`). `fetch`/`pull`
  with a non-empty `have` set exercise a materially different code path
  (delta walk, not cache HIT) that has *not* been the subject of an explicit
  E2E in this campaign.
- `pull` = fetch + merge/ff — the merge/ff correctness after a fetch has
  never been explicitly re-tested post-MWP-8 (does a `pull` after a
  force-pushed remote branch behave correctly, or does it silently diverge?).

**`/tmp` recipe:**
```bash
# 3a — fetch convergence: two clones, one pushes, other fetches
cd /tmp && rm -rf mwp-qa-3a-x mwp-qa-3a-y
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-3a-x --json
muse clone https://staging.musehub.ai/gabriel/mwp-qa-1a mwp-qa-3a-y --json
cd /tmp/mwp-qa-3a-x
echo "from-x" >> README.md && muse code add README.md
muse commit -m "from x" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
muse push staging main --json
cd /tmp/mwp-qa-3a-y
muse fetch staging main --json    # non-empty `have` — exercises delta walk, not cache HIT
muse log --json | jq '.commits[0].message'   # should NOT yet show "from x" (fetch != merge)
muse pull staging main --json
grep "from-x" README.md    # NOW should be present — proves pull = fetch + merge converges

# 3b — pull after a force-push (interaction with MWP-8)
cd /tmp/mwp-qa-1a
muse reset HEAD~1 --json   # local-only rewrite, still on main — DO NOT run --hard without asking gabriel first
# (coordinate with gabriel before any reset/force-push against a shared staging repo — see Safety notes)
```

---

### Claim 4 — Repeatable with no stale clones, no phantom large uploads, no manual cache-busting

**Fixed by:** all of MWP-1..MWP-8 in combination; MWP-8 specifically removes
the last known manual-cache-bust scenario (repair).

**Already proven:** each sub-ticket proves its own slice is real-time
self-healing (no `UPDATE musehub_fetch_mpack_cache ...` needed).

**Residual gap — this is the important one:**
- **No test runs the full loop N times in sequence** (push, clone, push,
  clone, repair, clone, force-push, clone) and asserts *zero* manual
  intervention across the whole sequence. Each MWP proved its *own* trigger
  in isolation; nothing proves the *composition*.
- "No phantom large uploads" has only been checked at the byte-count level
  for two-branch topologies (MWP-7). A repeated-push loop (claim 4) should
  also confirm upload size stays flat/small across many iterations, not just
  correct on push #2.

**`/tmp` recipe:**
```bash
# 4a — the full repeated loop, one repo, ~10 iterations, watch for growth
cd /tmp && rm -rf mwp-qa-4a && mkdir mwp-qa-4a && cd mwp-qa-4a
muse init --json
muse remote add staging https://staging.musehub.ai/gabriel/mwp-qa-4a
for i in $(seq 1 10); do
  echo "iteration $i" >> log.txt
  muse code add log.txt
  muse commit -m "iter $i" --agent-id claude-code --model-id claude-sonnet-4-6 --sign
  push_out=$(muse push staging main --json)
  echo "$push_out" | jq '{i: '"$i"', objects_sent, blobs_sent, size_bytes}'
  rm -rf /tmp/mwp-qa-4a-clone-$i
  muse clone https://staging.musehub.ai/gabriel/mwp-qa-4a /tmp/mwp-qa-4a-clone-$i --json
  tail -1 /tmp/mwp-qa-4a-clone-$i/log.txt   # must equal "iteration $i" every time, no staleness
done
# expect: objects_sent/blobs_sent stay ~1 per iteration (not growing), every clone's tail matches
```

---

### Claim 5 — Wire regression suite is re-enabled and green

**Fixed by:** MWP-6.

**Already proven:** 55 tests under `pytest -m wire` green (47 in-process + 8
slow subprocess E2E); MWP6_00 gate enforces no re-introduction of flux-skips.

**Residual gap:**
- This claim is the most rigorously proven of the five — but it should be
  re-run **right now**, post-MWP-8-merge-to-main, to confirm nothing in the
  final merge sequence broke it. This is cheap and should be step 0 of the
  spot-check, not step 5.

**Recipe:**
```bash
cd ~/ecosystem/musehub
python3 -m pytest -m wire -q --tb=short
cd ~/ecosystem/muse
python3 -m pytest -m wire -q --tb=short
```

---

## Cross-cutting risks not captured by any single MWP's test suite

These are the seams between fixes — the reason a "deeper than surface"
review matters more than re-reading each ticket's own tests:

1. **MWP-3/4 (job ordering/idempotency) × MWP-8 (cache invalidation
   enqueues its own prebuild job)** — `_enqueue_repair_prebuild` calls the
   same `enqueue_job` that MWP-3 hardened for push-triggered prebuilds. Has
   the repair-triggered enqueue path been checked against MWP-3's dedup-by-
   `mpack_key` logic? A repair followed immediately by a push could enqueue
   two prebuild jobs — does MWP-3's fix correctly coalesce them, or does one
   get silently dropped (regressing RC-3 in a new trigger)?
2. **MWP-1/2 (generation authority) × MWP-7 (sibling-ref dedup)** — MWP-7's
   dedup subtracts a sibling tip's full manifest. If that sibling's own
   generation was ever backfilled by MWP-1's on-demand repair, does the
   manifest lookup still resolve correctly, or could a repaired-generation
   commit be mis-scoped out of a dedup calculation?
3. **MWP-5 (client retry) × MWP-4 (prebuild ordering)** — MWP-5's bounded
   retry budget (120s default) was tuned against MWP-4's fixed ordering
   guarantee. If a repo is large enough that index+prebuild together exceed
   120s, does the client fail with a clear, actionable error, or a confusing
   timeout?
4. **Real R2 vs local MinIO** — several sub-tickets' "best-effort R2 delete"
   paths (MWP-8's `invalidate_fetch_mpack_cache`) were only tested against a
   mocked backend. Staging uses real Cloudflare R2. Confirm the delete call
   shape matches R2's actual API (auth headers, error codes on 404-already-
   gone) rather than the mock's simplified contract.
5. **Concurrent agents on the same repo** — aaronrene will not be the only
   person pushing to shared repos. No MWP ticket tested two *different
   client machines* (not just two CLI invocations from one machine) pushing
   to the same branch concurrently. This is the actual onboarding scenario.

---

## Safety notes for this QA pass

- **All QA repos are scratch repos under `gabriel/mwp-qa-*`** on staging —
  never reuse `gabriel/muse` or `gabriel/musehub` themselves for destructive
  operations (force-push, reset --hard, repair-*).
- Per workspace hard rules: no `--force` flag, no `reset --hard`, no
  `checkout --ours/--theirs` without gabriel's explicit go-ahead in the
  conversation, even inside a scratch repo — ask first if a recipe step
  needs one.
- Every `/tmp` directory created here is disposable; clean up with a plain
  `rm -rf` when done (not a muse operation, so no special permission needed).
- If any staging repair endpoint needs to be exercised (Claim 1's repair×
  clone interaction), coordinate the exact repo/commit with gabriel before
  running it — repair mutates content-addressed rows in place.

---

## Sign-off checklist before aaronrene onboards

- [ ] Claim 1 recipes (1a, 1b) run clean against staging
- [ ] Claim 2 recipes (2a, 2b, 2c) run clean against staging
- [ ] Claim 3 recipes (3a; 3b coordinated with gabriel) run clean
- [ ] Claim 4 recipe (4a, 10-iteration loop) shows flat upload size, zero
      staleness
- [ ] Claim 5 — `pytest -m wire` green in both repos, run fresh today
- [ ] Cross-cutting risks 1–5 above each explicitly checked or explicitly
      deferred with a written reason
- [ ] All scratch repos (`gabriel/mwp-qa-*`) cleaned up / deleted from staging
- [ ] Findings written back into this file; any new bugs filed as their own
      musehub/muse tickets (not silently patched)
- [ ] gabriel gives explicit go-ahead to send aaronrene the staging URL