Release pruning deletes the newest nightly build (sort -V has no PEP 440 precedence)

Release pruning deletes the newest nightly because sort -V doesn't know PEP 440 precedence

Background

The proximate trigger

While deploying the first-ever nightly build (muse-0.2.0.dev1.tar.gz) during the pivot to the nightly channel (docs/versioning.md), deploy/publish_muse_release.sh's cleanup step deleted the tarball from s3://musehub-releases/ seconds after uploading it:

[1/5] Building muse 0.2.0.dev1 ...
[2/5] Uploading to s3://musehub-releases/
[3/5] Pushing to staging ...
[4/5] Cleaning up old releases (keeping 3 newest)
  Deleting s3://musehub-releases/muse-0.2.0.dev1.tar.gz   ← the file just uploaded
[5/5] Verifying https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz
  ✅ Live at https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz

The file only "survived" because the verification step reads from the staging instance's local /data/releases/ copy (populated by the earlier SSM push step), not from S3. The durable S3 backup copy was gone — silent data loss that would only surface the next time someone needed to restore from S3, or the next publish run repeated the same mistake. Caught and manually re-uploaded from the live URL as an immediate fix; this issue tracks the actual root cause.

Root cause — verified by direct reproduction, not assumed

Both cleanup blocks (S3, at deploy/publish_muse_release.sh lines 104-113, and the server's /data/releases/, lines 119-127) pipe muse-*.tar.gz filenames through sort -V (GNU "version sort") before keeping only the newest KEEP_RELEASES (currently 3). Reproduced exactly with the four files that existed on staging at deploy time:

$ printf "muse-0.2.0rc14.tar.gz\nmuse-0.2.0rc15.tar.gz\nmuse-0.2.0rc16.tar.gz\nmuse-0.2.0.dev1.tar.gz\n" | sort -V
muse-0.2.0.dev1.tar.gz    ← sorted FIRST (== "oldest" to the keep-N logic)
muse-0.2.0rc14.tar.gz
muse-0.2.0rc15.tar.gz
muse-0.2.0rc16.tar.gz

sort -V is a generic natural/version sort (the same tool that correctly orders file1, file2, file10) — it has zero knowledge of PEP 440's reserved-segment precedence (dev < a < b < rc < final, documented in docs/versioning.md's "A real gotcha" section). It compares "dev1" against "rc14" as plain text, and 'd' sorts before 'r' — placing the newest channel (nightly) before three older release candidates. The keep-newest-3 awk logic then deletes line 1, i.e. exactly the file that should have been kept.

Verified the correct order via the tool that actually understands PEP 440 precedence:

$ python3 -c "
from packaging.version import Version
names = ['0.2.0.dev1', '0.2.0a1', '0.2.0b1', '0.2.0rc14', '0.2.0rc15', '0.2.0rc16']
for n in sorted(names, key=Version): print(n)
"
0.2.0.dev1
0.2.0a1
0.2.0b1
0.2.0rc14
0.2.0rc15
0.2.0rc16

packaging.version.Version (already a transitive dependency of build/twine, both already required by this exact script) sorts all six real channel examples in the correct maturity order documented in docs/versioning.md. This is not a hypothetical edge case — it is the exact tool needed, already present in the toolchain.

Why this isn't a one-line fix

sort -V is invoked in two places with slightly different execution contexts:

1. S3 cleanup — runs locally, in the deploy script's own shell, where Python + packaging are guaranteed available (the script already builds an sdist via python3 -m build).
2. Server cleanup — runs remotely via SSM (AWS-RunShellScript on the staging EC2 instance's host shell, not inside the musehub Docker container) — Python/packaging availability on the bare host is not guaranteed the same way.

The correct fix is to compute the stale-file list once, locally (where Python is guaranteed present) and pass that exact, precomputed list to both the S3 delete calls and the remote SSM rm command — eliminating the need for the remote host to re-implement version sorting at all, and eliminating the current duplication of the same buggy sort logic in two places.

Design — the shape to build toward

A small, dependency-free-of-bash-quirks Python helper, deploy/prune_releases.py:

def stale_releases(filenames: list[str], keep: int) -> list[str]:
    """Return filenames to delete — all but the *keep* highest-precedence
    releases, ordered oldest-first. Never guesses on an unparseable name:
    raises rather than silently mis-sorting."""

• Parses muse-{version}.tar.gz → packaging.version.Version for the sort key — the same canonical PEP 440 precedence docs/versioning.md documents.
• An unparseable filename (doesn't match the pattern, or the version segment isn't valid PEP 440) is a hard error, not a silent skip or a fallback to lexical sort — a malformed release name is exactly the kind of thing that must stop the pipeline, not quietly mis-order it.
• CLI entry point (--keep N <filenames...>, prints stale names one per line) so both the local S3 step and the remote SSM step can consume it identically via xargs/while read.

publish_muse_release.sh changes:

• S3 cleanup: pipe the aws s3 ls filename list through prune_releases.py --keep "$KEEP_RELEASES" instead of sort -V | awk ....
• Server cleanup: list remote files via one SSM command (ls /data/releases/muse-*.tar.gz), compute the stale set locally with the same prune_releases.py call, then send a second SSM command with an explicit rm of the exact stale filenames — the remote shell never sorts anything itself.

Goal — definition of done

1. Deploying a nightly build never deletes it (or any other currently-newest release) from S3 or the server.
2. Pruning correctly follows PEP 440 channel precedence (dev < a < b < rc < final) regardless of which channels are mixed in the release history at prune time.
3. An unparseable/malformed release filename halts pruning with a clear error rather than silently mis-ordering everything else.
4. sort -V's buggy precedence logic exists in exactly zero places in this script after the fix (both cleanup blocks use the same, correct, single source of truth).
5. Every deliverable is TDD'd: red test first, then green.

Phases

Phase 1 — prune_releases.py, in isolation

• PRUNE_01 — stale_releases(["muse-0.2.0rc14.tar.gz", "muse-0.2.0rc15.tar.gz", "muse-0.2.0rc16.tar.gz", "muse-0.2.0.dev1.tar.gz"], keep=3) returns ["muse-0.2.0rc14.tar.gz"] — the actual reproduction case from this incident, pinned as a regression test.
• PRUNE_02 — Mixed channels (dev1, a1, b1, rc1, rc2) with keep=2 returns everything except the two highest-precedence (rc1, rc2) — full channel-order coverage, not just the dev-vs-rc pair.
• PRUNE_03 — keep >= len(filenames) returns [] — never deletes when there's nothing to prune.
• PRUNE_04 — An unparseable filename (e.g. muse-not-a-version.tar.gz) raises rather than silently sorting it somewhere arbitrary.
• PRUNE_05 — Stable releases (0.2.0, no pre-release suffix) sort after all pre-release channels, per PEP 440 — a mix of dev1/rc1/stable keeps the stable one.

Exit gate: stale_releases is correct in isolation for every channel combination in docs/versioning.md's Build Channels table, with the exact incident reproduction as a named regression test.

Phase 2 — Wire into publish_muse_release.sh

• PRUNE_06 — S3 cleanup calls prune_releases.py instead of sort -V | awk. Test (shell-level, via a fixture directory of fake tarball names and a mocked aws s3 ls/aws s3 rm, or a focused manual dry-run against a throwaway S3 prefix): the exact dev1-survives-rc14-deleted case no longer reproduces.
• PRUNE_07 — Server cleanup computes the stale set locally and sends the remote rm with explicit filenames — no sort logic executes on the remote host at all. Verified by reading the exact SSM command string sent (no sort invocation present).

Exit gate: re-running the exact reproduction command from the Background section against the real script (dry-run or a throwaway bucket) confirms muse-0.2.0.dev1.tar.gz is correctly identified as the one to keep, and the three rc1x files as the ones to prune.

Phase 3 — Live re-verification

• PRUNE_08 — Run the real publish_muse_release.sh for the next scheduled nightly/release bump, confirm via aws s3 ls s3://musehub-releases/ that the newest build survives pruning and the correct oldest ones are removed.

Exit gate: a real deploy cycle completes with correct pruning, observed live, not just unit-tested.

Acceptance criteria (whole-issue gate)

• sort -V no longer appears anywhere in publish_muse_release.sh.
• Pruning is correct for every channel combination in docs/versioning.md's Build Channels table.
• The exact incident (nightly deleted from S3 immediately after publish) cannot recur — pinned by a named regression test.
• Full TDD coverage: every phase's deliverables have a red-then-green test.
• Verified live on a real deploy cycle, not just unit-tested.

Out of scope (explicit, for future issues)

• Any change to the server-side /data/releases/ retention policy itself (e.g. keeping more than 3, or keeping one-per-channel) — this issue is about correctness of the existing keep-N-newest policy, not changing the policy.
• muse/core/semver.py implementing precedence comparison — the versioning doc already notes this isn't currently implemented and isn't a live bug there; this issue's fix lives entirely in the deploy script, using packaging.version.Version (already available), not in muse's own SemVer module.