Release pruning deletes the newest nightly because sort -V doesn't know PEP 440 precedence

Release pruning deletes the newest nightly because sort -V doesn't know PEP 440 precedence

Status

✅ Phase 1 and Phase 2 landed on dev (2026-07-07), via branch fix/release-prune-pep440-sort → dev, fast-forward, no conflicts. prune_releases.py exists, is wired into both cleanup blocks of publish_muse_release.sh, and all 17 tests across test_prune_releases.py and test_publish_muse_release_prune.py are green on dev.

⏸️ Phase 3 (live re-verification, PRUNE_10) is blocked, not started — it requires running the real publish_muse_release.sh against a live deploy, which is explicitly on hold pending gabriel's go-ahead (manual QA of nightly.3 first; more tickets may land before this nightly goes out publicly). Will resume once deploy is unblocked.

Background

The proximate trigger

While deploying the first-ever nightly build (muse-0.2.0.dev1.tar.gz) during the pivot to the nightly channel (docs/versioning.md), deploy/publish_muse_release.sh's cleanup step deleted the tarball from s3://musehub-releases/ seconds after uploading it:

[1/5] Building muse 0.2.0.dev1 ...
[2/5] Uploading to s3://musehub-releases/
[3/5] Pushing to staging ...
[4/5] Cleaning up old releases (keeping 3 newest)
  Deleting s3://musehub-releases/muse-0.2.0.dev1.tar.gz   ← the file just uploaded
[5/5] Verifying https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz
  ✅ Live at https://staging.musehub.ai/releases/muse-0.2.0.dev1.tar.gz

The file only "survived" because the verification step reads from the staging instance's local /data/releases/ copy (populated by the earlier SSM push step), not from S3. The durable S3 backup copy was gone — silent data loss that would only surface the next time someone needed to restore from S3, or the next publish run repeated the same mistake. Caught and manually re-uploaded from the live URL as an immediate fix; this issue tracks the actual root cause.

Correction (2026-07-07) — the original narrative below misread its own reproduction

The reproduction two sections down already showed 0.2.0.dev1 sorting first/lowest under packaging.version.Version, identically to sort -V. The original write-up misread that output as confirming a fix; it doesn't. PEP 440 defines dev as the lowest-precedence pre-release stage for a given base version (dev < a < b < rc < final) — so 0.2.0.dev1 < 0.2.0rc16 is correct, not a bug, given they share the base version 0.2.0. Caught during TDD Phase 1: the first test written from this plan's own stated exit gate (dev1 survives, rc14 pruned) failed against a correct packaging.version.Version-based implementation — the implementation was right and the plan's expectation was wrong.

The actual root cause isn't the sort algorithm — it's that the channel pivot reused the base version 0.2.0 for both the old rc-labeled builds and the new dev-labeled ones. packaging.version.Version correctly replaces sort -V's text-comparison bug (comparing "dev1" vs "rc14" lexically), but no correct precedence sort can make a dev build of the same version outrank an rc build of that version — because it shouldn't. The durable fix is versioning discipline, not smarter sorting: a channel pivot must bump the base version (docs/versioning.md now has this rule). stale_releases() itself needed no behavior change from the original design — it was already correctly implementing PEP 440 precedence; only this plan's narrative and its Phase 1 test expectations needed correcting.

Root cause — verified by direct reproduction, not assumed

Both cleanup blocks (S3, at deploy/publish_muse_release.sh lines 104-113, and the server's /data/releases/, lines 119-127) pipe muse-*.tar.gz filenames through sort -V (GNU "version sort") before keeping only the newest KEEP_RELEASES (currently 3). Reproduced exactly with the four files that existed on staging at deploy time:

$ printf "muse-0.2.0rc14.tar.gz\nmuse-0.2.0rc15.tar.gz\nmuse-0.2.0rc16.tar.gz\nmuse-0.2.0.dev1.tar.gz\n" | sort -V
muse-0.2.0.dev1.tar.gz    ← sorted FIRST (== "oldest" to the keep-N logic)
muse-0.2.0rc14.tar.gz
muse-0.2.0rc15.tar.gz
muse-0.2.0rc16.tar.gz

sort -V is a generic natural/version sort with zero knowledge of PEP 440's reserved-segment precedence (dev < a < b < rc < final, documented in docs/versioning.md's "A real gotcha" section) — it compares "dev1" against "rc14" as plain text. In this exact case 'd' < 'r' happens to land sort -V on the same ordering PEP 440 itself would produce (dev is correctly lowest-precedence for a shared base version, see the Correction above) — so sort -V's failure mode here isn't "wrong order for these four files," it's "right order by lexical accident, wrong in general." Verified a genuine divergence with a different pair: sort -V ranks muse-0.2.0.tar.gz (final) before muse-0.2.0rc1.tar.gz (pre-release) —

$ printf "muse-0.2.0.tar.gz\nmuse-0.2.0rc1.tar.gz\n" | sort -V
muse-0.2.0.tar.gz
muse-0.2.0rc1.tar.gz

— while PEP 440 correctly ranks the release candidate below the final (rc1 < final, confirmed: Version("0.2.0rc1") < Version("0.2.0") → True). sort -V would treat the final release as older than its own release candidate and prune it first. packaging.version.Version is still the right fix — it's just not the fix for this specific incident's dev1-vs-rc ordering, which was correct all along. The keep-newest-3 logic deleted dev1 in the incident — which, given the base-version collision with the rc builds, is what a correct PEP 440 sort does too. The real problem is the version collision itself, not this particular sort's output.

Verified the correct order via the tool that actually understands PEP 440 precedence:

$ python3 -c "
from packaging.version import Version
names = ['0.2.0.dev1', '0.2.0a1', '0.2.0b1', '0.2.0rc14', '0.2.0rc15', '0.2.0rc16']
for n in sorted(names, key=Version): print(n)
"
0.2.0.dev1
0.2.0a1
0.2.0b1
0.2.0rc14
0.2.0rc15
0.2.0rc16

packaging.version.Version (already a transitive dependency of build/twine, both already required by this exact script) sorts all six real channel examples in the correct maturity order documented in docs/versioning.md. This is not a hypothetical edge case — it is the exact tool needed, already present in the toolchain.

Why this isn't a one-line fix

sort -V is invoked in two places with slightly different execution contexts:

1. S3 cleanup — runs locally, in the deploy script's own shell, where Python + packaging are guaranteed available (the script already builds an sdist via python3 -m build).
2. Server cleanup — runs remotely via SSM (AWS-RunShellScript on the staging EC2 instance's host shell, not inside the musehub Docker container) — Python/packaging availability on the bare host is not guaranteed the same way.

The correct fix is to compute the stale-file list once, locally (where Python is guaranteed present) and pass that exact, precomputed list to both the S3 delete calls and the remote SSM rm command — eliminating the need for the remote host to re-implement version sorting at all, and eliminating the current duplication of the same buggy sort logic in two places.

Design — the shape to build toward

A small, dependency-free-of-bash-quirks Python helper, deploy/prune_releases.py:

def stale_releases(filenames: list[str], keep: int) -> list[str]:
    """Return filenames to delete — all but the *keep* highest-precedence
    releases, ordered oldest-first. Never guesses on an unparseable name:
    raises rather than silently mis-sorting."""

• Parses muse-{version}.tar.gz → packaging.version.Version for the sort key — the same canonical PEP 440 precedence docs/versioning.md documents.
• An unparseable filename (doesn't match the pattern, or the version segment isn't valid PEP 440) is a hard error, not a silent skip or a fallback to lexical sort — a malformed release name is exactly the kind of thing that must stop the pipeline, not quietly mis-order it.
• CLI entry point (--keep N <filenames...>, prints stale names one per line) so both the local S3 step and the remote SSM step can consume it identically via xargs/while read.

publish_muse_release.sh changes:

• S3 cleanup: pipe the aws s3 ls filename list through prune_releases.py --keep "$KEEP_RELEASES" instead of sort -V | awk ....
• Server cleanup: list remote files via one SSM command (ls /data/releases/muse-*.tar.gz), compute the stale set locally with the same prune_releases.py call, then send a second SSM command with an explicit rm of the exact stale filenames — the remote shell never sorts anything itself.

Goal — definition of done

1. Pruning correctly follows PEP 440 channel precedence (dev < a < b < rc < final) regardless of which channels are mixed in the release history at prune time — given a base version is never reused across a channel pivot (see the Correction above and docs/versioning.md's new rule). This code fix guarantees correct sorting; it cannot and should not guarantee "the most recently built file is always kept" when two builds deliberately share a base version with different precedence — that's a numbering-discipline invariant, not a sorting one.
2. An unparseable/malformed release filename halts pruning with a clear error rather than silently mis-ordering everything else.
3. sort -V's lexical-comparison logic exists in exactly zero places in this script after the fix (both cleanup blocks use the same, correct, single source of truth: packaging.version.Version).
4. Every deliverable is TDD'd: red test first, then green.

Phases

Phase 1 — prune_releases.py, in isolation

• PRUNE_01 — stale_releases(["muse-0.2.0rc14.tar.gz", "muse-0.2.0rc15.tar.gz", "muse-0.2.0rc16.tar.gz", "muse-0.2.0.dev1.tar.gz"], keep=3) returns ["muse-0.2.0.dev1.tar.gz"] — correct PEP 440 precedence for the incident's actual four filenames (dev1 is lowest-precedence given the shared base version 0.2.0; see the Correction above — this is NOT the same as "dev1 survives," which was the plan's original, incorrect expectation).
• PRUNE_01b — Same four rc1x files plus muse-0.2.1.dev1.tar.gz (bumped base version, matching the new versioning.md rule) with keep=3 returns ["muse-0.2.0rc14.tar.gz"] — confirms the actual fix for the incident (bump the version on a pivot) produces the desired outcome once correctly numbered.
• PRUNE_02 — Mixed channels (dev1, a1, b1, rc1, rc2) with keep=2 returns everything except the two highest-precedence (rc1, rc2) — full channel-order coverage, not just the dev-vs-rc pair.
• PRUNE_03 — keep >= len(filenames) returns [] — never deletes when there's nothing to prune.
• PRUNE_04 — An unparseable filename (e.g. muse-not-a-version.tar.gz) raises rather than silently sorting it somewhere arbitrary.
• PRUNE_05 — Stable releases (0.2.0, no pre-release suffix) sort after all pre-release channels, per PEP 440 — a mix of dev1/rc1/stable keeps the stable one.

Exit gate: stale_releases is correct in isolation for every channel combination in docs/versioning.md's Build Channels table, with the exact incident reproduction as a named regression test.

Phase 2 — Wire into publish_muse_release.sh

• PRUNE_06 — S3 cleanup calls prune_releases.py instead of sort -V | awk. Test (shell-level, via a fixture directory of fake tarball names and a mocked aws s3 ls/aws s3 rm, or a focused manual dry-run against a throwaway S3 prefix): pruning matches stale_releases()'s output exactly for a realistic mixed set of filenames, including a case with a properly-bumped nightly (PRUNE_01b's shape) surviving pruning as expected.
• PRUNE_07 — Server cleanup computes the stale set locally and sends the remote rm with explicit filenames — no sort logic executes on the remote host at all. Verified by reading the exact SSM command string sent (no sort invocation present).

Exit gate: re-running the S3 cleanup against a throwaway bucket containing a properly-numbered pivot (bumped base version per the new docs/versioning.md rule) confirms the newest, highest-precedence build survives pruning and the correct oldest ones are removed — not a reproduction of the original (mis-specified) incident expectation.

Phase 3 — Live re-verification

• PRUNE_10 — Run the real publish_muse_release.sh for the next scheduled nightly/release bump, confirm via aws s3 ls s3://musehub-releases/ that the newest build survives pruning and the correct oldest ones are removed. (Renumbered from the original PRUNE_08 to avoid colliding with two additional unit tests added to test_prune_releases.py during Phase 1 — PRUNE_08/PRUNE_09 now cover empty-input and oldest-first ordering, unrelated to this live-verification deliverable.)

Exit gate: a real deploy cycle completes with correct pruning, observed live, not just unit-tested.

Acceptance criteria (whole-issue gate)

• sort -V no longer appears anywhere in publish_muse_release.sh.
• Pruning is correct for every channel combination in docs/versioning.md's Build Channels table, verified by packaging.version.Version, not lexical comparison.
• A future channel pivot that follows docs/versioning.md's new base-version-bump rule cannot lose its newest build to pruning — pinned by PRUNE_01b. (This literal incident's exact filenames, with the base version left unbumped, correctly prune dev1 per PEP 440 — see the Correction section — so that specific scenario is not itself the regression target; the numbering discipline is.)
• Full TDD coverage: every phase's deliverables have a red-then-green test.
• Verified live on a real deploy cycle, not just unit-tested.

Out of scope (explicit, for future issues)

• Any change to the server-side /data/releases/ retention policy itself (e.g. keeping more than 3, or keeping one-per-channel) — this issue is about correctness of the existing keep-N-newest policy, not changing the policy.
• muse/core/semver.py implementing precedence comparison — the versioning doc already notes this isn't currently implemented and isn't a live bug there; this issue's fix lives entirely in the deploy script, using packaging.version.Version (already available), not in muse's own SemVer module.