# Release pruning deletes the newest nightly because `sort -V` doesn't know PEP 440 precedence

## Background

### The proximate trigger

While deploying the first-ever nightly build (`muse-0.2.0.dev1.tar.gz`)
during the pivot to the `nightly` channel (`docs/versioning.md`),
`deploy/publish_muse_release.sh`'s cleanup step deleted the tarball from
`s3://musehub-releases/` **seconds after uploading it**:

```
[1/5] Building muse 0.2.0.dev1 ...
[2/5] Uploading to s3://musehub-releases/
[3/5] Pushing to staging ...
[4/5] Cleaning up old releases (keeping 3 newest)
  Deleting s3://musehub-releases/muse-0.2.0.dev1.tar.gz   ← the file just uploaded
[5/5] Verifying https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz
  ✅ Live at https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz
```

The file only "survived" because the verification step reads from the
staging instance's local `/data/releases/` copy (populated by the
earlier SSM push step), not from S3. The **durable S3 backup copy was
gone** — silent data loss that would only surface the next time someone
needed to restore from S3, or the next `publish` run repeated the same
mistake. Caught and manually re-uploaded from the live URL as an
immediate fix; this issue tracks the actual root cause.

### Root cause — verified by direct reproduction, not assumed

Both cleanup blocks (S3, at `deploy/publish_muse_release.sh` lines
104-113, and the server's `/data/releases/`, lines 119-127) pipe
`muse-*.tar.gz` filenames through `sort -V` (GNU "version sort") before
keeping only the newest `KEEP_RELEASES` (currently 3). Reproduced
exactly with the four files that existed on staging at deploy time:

```bash
$ printf "muse-0.2.0rc14.tar.gz\nmuse-0.2.0rc15.tar.gz\nmuse-0.2.0rc16.tar.gz\nmuse-0.2.0.dev1.tar.gz\n" | sort -V
muse-0.2.0.dev1.tar.gz    ← sorted FIRST (== "oldest" to the keep-N logic)
muse-0.2.0rc14.tar.gz
muse-0.2.0rc15.tar.gz
muse-0.2.0rc16.tar.gz
```

`sort -V` is a **generic natural/version sort** (the same tool that
correctly orders `file1, file2, file10`) — it has zero knowledge of PEP
440's reserved-segment precedence (`dev < a < b < rc < final`,
documented in `docs/versioning.md`'s "A real gotcha" section). It
compares `"dev1"` against `"rc14"` as plain text, and `'d'` sorts before
`'r'` — placing the *newest* channel (nightly) before three *older*
release candidates. The keep-newest-3 `awk` logic then deletes line 1,
i.e. exactly the file that should have been kept.

Verified the correct order via the tool that actually understands PEP
440 precedence:

```bash
$ python3 -c "
from packaging.version import Version
names = ['0.2.0.dev1', '0.2.0a1', '0.2.0b1', '0.2.0rc14', '0.2.0rc15', '0.2.0rc16']
for n in sorted(names, key=Version): print(n)
"
0.2.0.dev1
0.2.0a1
0.2.0b1
0.2.0rc14
0.2.0rc15
0.2.0rc16
```

`packaging.version.Version` (already a transitive dependency of
`build`/`twine`, both already required by this exact script) sorts all
six real channel examples in the correct maturity order documented in
`docs/versioning.md`. This is not a hypothetical edge case — it is the
exact tool needed, already present in the toolchain.

### Why this isn't a one-line fix

`sort -V` is invoked in **two places** with slightly different
execution contexts:

1. **S3 cleanup** — runs locally, in the deploy script's own shell,
   where Python + `packaging` are guaranteed available (the script
   already builds an sdist via `python3 -m build`).
2. **Server cleanup** — runs remotely via SSM
   (`AWS-RunShellScript` on the staging EC2 instance's host shell, not
   inside the `musehub` Docker container) — Python/`packaging`
   availability on the bare host is not guaranteed the same way.

The correct fix is to **compute the stale-file list once, locally**
(where Python is guaranteed present) and pass that exact, precomputed
list to both the S3 delete calls and the remote SSM `rm` command —
eliminating the need for the remote host to re-implement version
sorting at all, and eliminating the current duplication of the same
buggy sort logic in two places.

## Design — the shape to build toward

A small, dependency-free-of-bash-quirks Python helper,
`deploy/prune_releases.py`:

```python
def stale_releases(filenames: list[str], keep: int) -> list[str]:
    """Return filenames to delete — all but the *keep* highest-precedence
    releases, ordered oldest-first. Never guesses on an unparseable name:
    raises rather than silently mis-sorting."""
```

- Parses `muse-{version}.tar.gz` → `packaging.version.Version` for the
  sort key — the same canonical PEP 440 precedence `docs/versioning.md`
  documents.
- An unparseable filename (doesn't match the pattern, or the version
  segment isn't valid PEP 440) is a hard error, not a silent skip or a
  fallback to lexical sort — a malformed release name is exactly the
  kind of thing that must stop the pipeline, not quietly mis-order it.
- CLI entry point (`--keep N <filenames...>`, prints stale names one
  per line) so both the local S3 step and the remote SSM step can
  consume it identically via `xargs`/`while read`.

`publish_muse_release.sh` changes:
- S3 cleanup: pipe the `aws s3 ls` filename list through
  `prune_releases.py --keep "$KEEP_RELEASES"` instead of
  `sort -V | awk ...`.
- Server cleanup: list remote files via one SSM command
  (`ls /data/releases/muse-*.tar.gz`), compute the stale set **locally**
  with the same `prune_releases.py` call, then send a second SSM command
  with an explicit `rm` of the exact stale filenames — the remote shell
  never sorts anything itself.

## Goal — definition of done

1. Deploying a nightly build never deletes it (or any other
   currently-newest release) from S3 or the server.
2. Pruning correctly follows PEP 440 channel precedence
   (`dev < a < b < rc < final`) regardless of which channels are mixed
   in the release history at prune time.
3. An unparseable/malformed release filename halts pruning with a clear
   error rather than silently mis-ordering everything else.
4. `sort -V`'s buggy precedence logic exists in exactly zero places in
   this script after the fix (both cleanup blocks use the same,
   correct, single source of truth).
5. Every deliverable is TDD'd: red test first, then green.

## Phases

### Phase 1 — `prune_releases.py`, in isolation

- [ ] `PRUNE_01` — `stale_releases(["muse-0.2.0rc14.tar.gz", "muse-0.2.0rc15.tar.gz", "muse-0.2.0rc16.tar.gz", "muse-0.2.0.dev1.tar.gz"], keep=3)` returns `["muse-0.2.0rc14.tar.gz"]` — the actual reproduction case from this incident, pinned as a regression test.
- [ ] `PRUNE_02` — Mixed channels (`dev1`, `a1`, `b1`, `rc1`, `rc2`) with `keep=2` returns everything except the two highest-precedence (`rc1`, `rc2`) — full channel-order coverage, not just the dev-vs-rc pair.
- [ ] `PRUNE_03` — `keep >= len(filenames)` returns `[]` — never deletes when there's nothing to prune.
- [ ] `PRUNE_04` — An unparseable filename (e.g. `muse-not-a-version.tar.gz`) raises rather than silently sorting it somewhere arbitrary.
- [ ] `PRUNE_05` — Stable releases (`0.2.0`, no pre-release suffix) sort after all pre-release channels, per PEP 440 — a mix of `dev1`/`rc1`/stable keeps the stable one.

**Exit gate:** `stale_releases` is correct in isolation for every channel
combination in `docs/versioning.md`'s Build Channels table, with the
exact incident reproduction as a named regression test.

### Phase 2 — Wire into `publish_muse_release.sh`

- [ ] `PRUNE_06` — S3 cleanup calls `prune_releases.py` instead of
      `sort -V | awk`. Test (shell-level, via a fixture directory of
      fake tarball names and a mocked `aws s3 ls`/`aws s3 rm`, or a
      focused manual dry-run against a throwaway S3 prefix): the exact
      `dev1`-survives-`rc14`-deleted case no longer reproduces.
- [ ] `PRUNE_07` — Server cleanup computes the stale set locally and
      sends the remote `rm` with explicit filenames — no sort logic
      executes on the remote host at all. Verified by reading the exact
      SSM command string sent (no `sort` invocation present).

**Exit gate:** re-running the exact reproduction command from the
Background section against the real script (dry-run or a throwaway
bucket) confirms `muse-0.2.0.dev1.tar.gz` is correctly identified as the
one to *keep*, and the three `rc1x` files as the ones to prune.

### Phase 3 — Live re-verification

- [ ] `PRUNE_08` — Run the real `publish_muse_release.sh` for the next
      scheduled nightly/release bump, confirm via `aws s3 ls
      s3://musehub-releases/` that the newest build survives pruning
      and the correct oldest ones are removed.

**Exit gate:** a real deploy cycle completes with correct pruning,
observed live, not just unit-tested.

## Acceptance criteria (whole-issue gate)

- `sort -V` no longer appears anywhere in `publish_muse_release.sh`.
- Pruning is correct for every channel combination in
  `docs/versioning.md`'s Build Channels table.
- The exact incident (nightly deleted from S3 immediately after
  publish) cannot recur — pinned by a named regression test.
- Full TDD coverage: every phase's deliverables have a red-then-green
  test.
- Verified live on a real deploy cycle, not just unit-tested.

## Out of scope (explicit, for future issues)

- Any change to the **server-side** `/data/releases/` retention policy
  itself (e.g. keeping more than 3, or keeping one-per-channel) — this
  issue is about correctness of the existing keep-N-newest policy, not
  changing the policy.
- `muse/core/semver.py` implementing precedence comparison — the
  versioning doc already notes this isn't currently implemented and
  isn't a live bug there; this issue's fix lives entirely in the deploy
  script, using `packaging.version.Version` (already available), not in
  muse's own SemVer module.